Lifestyle choices can lower sperm numbers. Smoking, drinking alcohol, and taking certain medications can lower sperm numbers. Other causes of low sperm numbers include long-term sickness (such as kidney failure), childhood infections (such as mumps), and chromosome or hormone problems (such as low testosterone).