Simplify worker stop logic