All central nervous system responses are impaired by stress, including those of the brain and endocrine glands - the very same glands involved in Stages I and II of a man's sexual process. Prolonged recreational drug use lowers libido and stresses adrenal glands, creating lower energy levels. Stress also affects sperm production and motility. Body stressors, like heavy metal or radiation exposure, damage sperm as well. Lower your stress levels with herbs like ginseng and regular exercise.
Exercise is critically important for a man's sexuality.
Find something physical you enjoy. Poor nutrition - High fat meals suppress testosterone, depress libido, and make erection and ejaculation more difficult. See my special sexuality boosting diet. Environmental estrogens - Environmental estrogens enter the body through pesticides and hormonally enhanced meat and dairy foods. Eat organic meats, produce and dairy products as much as possible to reduce the load of environmental estrogens on your system.