Only release the process reference when the Worker finishes stopping.

We might be able to be more aggressive in the future.