Some minor cleanups related to bench API

I found that some of the Bench tests leave benchmarks behind which can have an impact on other tests. We should shutdown all the benchmarks once tests are done. This PR adds some infra that detects this.