Faces for males and females are quite a bit alike, enough so that people who use them can use the same mind pathways and developed techniques to tell male and female faces apart. For this reason, people who use the face seldom say that the members of one sex are significantly harder to differentiate than the members of the other sex. When one must abandon the face to differentiate people, one thing discovered right away is that males and females look very different from each other. Hair styles are different. Men can have facial hair.
Clothing is different.
Mannerisms are different. And even voices are different. The temptation is often there to skimp on one system, and concentrate one's efforts on the other, hoping by doing so to come up with a system that at least half way works. Since the real groundwork for this is done in childhood, decisions of this nature are not made as part of a logical, thought out procedure. In a struggle to survive, a face blind child deals with all this simply the best way he can.