Complex sentence: Both sisters were accused of being men or hermaphrodites, and therefore sometimes called the "Press Brothers".
Split into simpler sentences: Both sisters were accused of actually being men. Detractors called them the "Press Brothers".