Override Karaf's config.properties, set
felix.fileinstall.enableConfigSave = true