While males are extremely rare in Gerudo society, the people weren't opposed to men quite at all before Ganondorf betrayed them. When he turned into Calamity Ganon, the Gerudo elected to disallow all men from entering Gerudo Town, ever, and establish the city as a haven for Hyrule's females.