Fix bug where work queue would start early