It can have an impact on the fertility of men. Varicoceles are known to cause primary infertility in 35 to 44 percent of men and secondary infertility in 45 to 81 percent on men. Talk to your doctor about seeing a reproductive specialist if you and your partner have trouble conceiving a child