Delete the old configs that we don't need anymore