Common Causes Infection: The testicle and epididymis, the part of the testicle that stores sperm, can sometimes become infected, causing pain and swelling that starts quickly and gets worse.reading more