Yes, damaged ducts allow unfiltered air to entire system and harm the indoor quality of air. Also, they can be a reason for increasing pollutants.