Detergent build-up can cause your freshly washed towels to smell musty. This happens when the soap residue builds up in the cotton fibers, which prevents the material from absorbing water; this prevents the towel from drying properly, which leads to a musty odor. Over time, contaminants will also start to develop on the fabric, which worsens the smell.