Correctly check if the config exists before removing