Exit configuration of an app once first successful config is loaded