When bootstrapping, remove old config