Yes you can. But recommend to wear gloves when gardening. Because there are some reasons. There can be number of harmful bacteria, fungi and microorganisms in the soil. Infections are more likely to occur. Wear gloves to prevent your skin from thorns, poisonous weeds and so on.