This may be due to the fact that varicoceles cause progressive damage to the testicle and a progressive decrease in the quality and quantity of sperm produced. The testicles are located in the scrotum, which effectively regulates their temperature. They are maintained at a temperature slightly below body temperature. In cold weather a man's testicles move close into his body as the cremasteric muscles, the muscles in the scrotum wall, tighten. In warm weather the cremasteric muscles relax and lengthen allowing the testicles to hang away from a man's body and cool down.
Varicoceles are a group of dilated veins filled with blood, which surround the testicles.
This blood is at body temperature and if the testes are near these veins they will be kept at a higher temperature than is beneficial for them. Even if a man has a varicocele only on one side, the blood warms the whole scrotum and both testicles can be negatively affected. In general, larger testicles make more sperm than smaller testicles. Often however, you see men who have a large one-sided varicocele that has damaged the testis on one side making it smaller. The small testis makes significantly less sperm than the normal one.