Always hang your towel up after use to give it a chance to dry out. This will make it less of a magnet for bacteria. Over a radiator or a towel heater is ideal.
And change for a fresh towel at least once a week.