runner: Exit with code 2 when we are prevented from overwriting the config file