Because face blindness makes recognizing people more difficult, one would expect such groups to be smaller. But another problem arises as well, that has an impact on the tribe one builds. Faces for males and females are quite a bit alike, enough so that people who use them can use the same mind pathways and developed techniques to tell male and female faces apart. For this reason, people who use the face seldom say that the members of one sex are significantly harder to differentiate than the members of the other sex. When one must abandon the face to differentiate people, one thing discovered right away is that males and females look very different from each other.
Hair styles are different.
Men can have facial hair. Clothing is different. Mannerisms are different. And even voices are different. The temptation is often there to skimp on one system, and concentrate one's efforts on the other, hoping by doing so to come up with a system that at least half way works.