Check that the config exists when starting