Some specific nuts can affect testosterone levels negatively. These nuts include peanuts, pistachios, and many others. Peanuts have high beta-sitosterol levels that decrease testosterone and dihydrotestosterone production. Whereas the pistachios have phytosterol that results in a severe dip in testosterone levels.