Treat new configuration files properly on upgrade