Create new config file on every run ... until we support multiple directories