The testicles produce sperm all the time and it takes 24 to 36 hours for a man to refill sperm.