There is a spectrum of psychological and physical factors that cause infertility. Some of the top causes are overactive or underactive thyroid gland, prostate problems, recreational drugs, depression, relationship problems, stress, anxiety, etc.