Make 100% sure we don't deadlock on exit

The test runner is deadlocking periodically after the changes in #176, let's make sure that can never ever happen.

## TODO:

- [ ] Make sure tests pass on CI