Drinking heavily can lower your testosterone levels by interfering with hormones that stimulate testosterone production. Your body may also produce less testosterone because alcohol abuse can shrink your testicles, where testosterone is made.