Ideally, you should not use a hair towel to dry your body. Hair towels are made using a specific fabric, keeping in mind its intent. It is made light and in a particular size so that it fits well with the purpose of drying hair.