Shut down all executors before exiting to reduce spew

Test Plan: Tested broken test cases, verified less spew