Don't lie about stop_on_success

This absolutely needs to be honored too, though.

See #4365.