Also bail out early from scheduling

This is pretty much the same as
d2e03993c977b8fdeced0e36a52a437f7c66ffad.