Now removes the running instances when we remove a configuration