Attempt to lower the number of resources consumed while running tests

I keep having too many resource issues while running tests, so bit the bullet and try to cleanup a lot. We no longer consume 2k+ threads, but there is still a lot more work to do.

After this PR, we peak around 240 threads (used to be 2-3k)
Can't do for testing reasons.  With the tests running cleanup more often, there is a higher chance to see this.  If we kill the JVM then we loose all test reporting.