This is such a derogatory term because Transvestism actually has nothing to do with clothing, and everything to do with gender identity. He usually creates an entire persona and character to perform as an uber-woman. A Drag Queen usually has no urge to dress in women's clothes in private. They don't feel a need for it in their 'everyday lives'. That's the easy way to say it.
Many people can give that explanation and never offend anyone.
I, however, apparently can't. Let's talk about sex and gender for a minute. She would know. Kate was born Al Bornstein and tried being a man for as long as she could. He was married 3 times as a man, but he didn't feel right.