High sugar intake, especially from sugar-sweetened beverages, has been linked to lower testosterone levels. Consuming sugar can cause testosterone levels to drop for several hours.