Remove old stuff from cluster config