Most of your repeat sets should be at distances of 100 yards or less. If you have trouble maintaining the quality of 100-yard repeats, do more 75s, 50s and 25s. Or on a longer swim, rather than hold steady pace, throw in a sprint 25 every third or fourth lap. Shorten the length of your average workout set. In general, cut total distance before speed, if something has to give.
Intensity has the more powerful effect on fitness.
Kevin Polansky works as hard at mountain biking as he does at swimming, heading for the Rocky Mountain trails around his Loveland, CO, home as often as he can. Power is another physical capacity that can still be improved upon after age 40. Any time you find swimmers of age 40 and above who are still competitive with the times they did 20 years ago, you can bet regular strength training is one of the ways they gain their edge. It's an advantage they likely didn't have in college since very few swimmers back then lifted weights. At that time, most coaches believed weight training slowed swimmers down with extra bulk.