The grout which is in damaged condition and stubborn dirt should be removed. The areas which are in good condition should not be removed. If you want to change the color of the grout, you will need to remove all the grout to put a new grout.