Attempt to lower the number of resources consumed while running tests

I keep having too many resource issues while running tests, so bit the bullet and try to cleanup a lot. We no longer consume 2k+ threads, but there is still a lot more work to do.

After this PR, we peak around 240 threads (used to be 2-3k)
Maybe just shutdown JVM by exit is better option here?