Debugger: Fix #10764.

- Destroy the background worker before going after all the managers. Otherwise,
there might potentially still be jobs in flight that might touch them, leading
to crashes.