The right time for a swim cap replacement would be when you see a hole or a tear in your cap, or it no longer provides comfort and firmness on your head. A tear in your swim cap would definitely widen each time you put it on. Holes are red flags so upon sight of one, it's time for you to change that swim cap for a newer and better one!