Yes, a poor diet — like the Western diet with high sugar, fat, and processed food consumption — can contribute to hormonal imbalances including testosterone levels.Yes, a poor diet — like the Western diet with high sugar, fat, and processed food consumption — can contribute to hormonal imbalances including testosterone levels.