Don't get freaked when you notice that there are lots of male flowers blossoming, the female flowers will spring up later. The male flowers will be needed to pollinate the female flowers. So no you don't need to remove the male flowers at all.