Make ActivityWorker shutdown consistent with WorkflowWorker shutdown