Gracefully cleanup tmp config files when non-cli options are specified.