In fact, there's so little science that your guess is as good as mine as to how widespread the health damage might be. We don't even know what's in most fragrances. Perfumes are sometimes made from hundreds of chemicals. They may smell great, seduce and arouse, but nice doesn't necessarily mean safe. We do know that odors can affect the brain, to which the nose is intricately connected.
Fragrances appear to affect chemical and electrical configurations in the brain.
They can certainly trigger memories. Whenever I smell pumpkin pie, I immediately recall that Thanksgiving Day long ago when I got seriously bonked on the head with an uncharitably-thrown football. That's the kind of thinking behind the business trend to use fragrance to trigger desired forms of behavior. A floral scent is said to make people work faster. Lemon supposedly improves productivity.