Clean up cached config in test class tear down.

This takes over from https://github.com/pantsbuild/pants/pull/858