I've been wearing panties for six years, ever since my wife put a pair on me. We see nothing wrong with this, but I have been ridiculed by others. My feeling is that people just are not comfortable enough with themselves to do the extraordinary. I can make light of pretty much anything.
People laugh at the image of men in panties because it's funny.
And sure, in a perfect world, perfectly evolved humans might fail to see anything intrinsically amusing in a man taking on the trappings of womanhood. We do not live in such a world, for which I am more than slightly grateful, as I can't imagine anything more boring. I like gender differences. I'm glad it's still possible to find the occasional convention to flout or taboo to violate. When Iggy Pop songs can sell Royal Caribbean cruises, it's obviously getting more and more difficult to be a bit of a rebel without trying really, really hard.