Both men and women have estrogen, and in men, normal amounts of estrogen work in concert with testosterone to produce sperm and bolster libido. However, excess estrogen in men can lead to negative side effects, including breast growth, erectile dysfunction, infertility, and reduced sex drive. Men who suspect that they have high estrogen levels should consult with a doctor.