A poor diet full of processed foods and little food diversity will definitely impact your sperm count. Studies have shown that men who eat more fruits and vegetables, whole grains, and less meat, dairy, and sugar have higher sperm counts. Eating foods that are high in fat, especially saturated fat, may also be linked to lower sperm counts. Avoiding these foods may help improve your sperm count.