It is a man's world, after all. And in our male-oriented culture, a man showing weakness is committing a terrible sin. Boys don't cry. These are all messages given to us males from our earliest days. And a man wearing a dress is breaking these rules.
Clothing is the ultimate sign of who is submissive and who is powerful in our culture.
It marks us, and what our status in the culture is. How we react to people often depends on how they are dressed, and we react very differently to women than we do to men. In my opinion, the main thrust of the women's equality movement was and is about bringing women up to the level of men. Women struggled to be allowed to share the privilege that males have. And we have come a long way in that respect.