If you do vary them, do it in a logical and understandable way. And avoid being 'loud'. Look at each page and note what the design and use of colour draws your eye to - it should be the place where the text begins. Anything that detracts from that should be toned down. It is a common mistake to let design detract from readability.
Designers are artists and can get carried away with the beauty of their creation.
But, in a publication, the purpose of the design is to enhance and bring out the meaning and purpose of the text. The design must be the servant of the text and not steal the show. For more details on the impact of design on readability, see our report .