The complication was society saying, oh, this other person is gendered female, that's bad or that's good. That makes you a lesbian, it makes him one. In both of their attempts to fit in, they appropriated language that didn't really fit what was going on. The film suggests that passing both threatens and confirms those societal codes. The sheer act of passing itself is erotic, regardless of the end result or where you come from.
Androgynous people are so erotic because they flicker, they're in motion.
The crossing of boundaries feels good at the societal level, like you can have chaos at night, but order and ritual must be restored by day. Brandon was like a party. They rape her to reassert the order. These are the mechanics of hatred. Brandon saw himself under terms very different from what you or I would understand.