Yes, there is a difference between using a terrycloth body towel and a regular hair towel. The former can roughen the cuticle of your hair, leading to frizz and breakage, while the latter absorbs better without damaging your strands.