I don't think we should exit here. Otherwise we might always check while a (failing) run is happening, and we wouldn't get notified?