Move config files to a location that won't overwrite everything.