If you have only a mild varicocele that is not causing symptoms like pain or swelling and is not affecting your fertility, your doctor may opt to leave it as it is. However, if you have a severe varicocele, leaving it untreated may result in increased pain, swelling, low sperm count and infertility as well as a drop in testosterone levels.