Job execution startup race condition

I have a hunch this is causing slow app reloading.

/cc @zendesk/samson @grosser