In this case "large" means around 300 Test Classes with around 2300 tests in it, see: https://cleancoder.ci.cloudbees.com/job/fitnesse/. I would love to see a larger test suite run with the new runner to get even more feedback. :)

Some more facts about my analysis: I ran the JUnit test in IntelliJ and added a simple test at the end that performed a short wait() allowing me to create a heap dump before the process ended. With this I ran the suite three times:

1. "normal" - fresh compile & run after checkout from github.
2. "context-withoutpool" - running all tests with the HierarchicalContextRunner, create TestClass instances when needed.
3. "context-withpool" - running all tests with the HierarchicalContextRunner, keeping TestClass instances in the TestClassPool for reuse.

My inspections:
- Runtime:

The runtime of "normal" and "context-withpool" are identical, "context-withoutpool" lacks for 4 seconds.

- Memory:

Memory consumption of "normal" is about 18 MB and increases about 5 MB for the test run with the "context-withoutpool" and another 5 MB for the test run with the "context-withpool". The first 5 MB are clearly the extra amount of Runners for the HierarchicalContextRunner, as it requires two instances per test class, the HierarchicalContextRunner itself and one ContextHierarchyLevelClassRunner per context (1 per class in this case).

The additional 5 MB for the pool are also easy to explain. My tests ran against the add-on, which does not modify the JUnit core. Therefore, the TestClass instances are simply doubled, and require a lot more memory than it is necessary. I will modify my add-on so that it is independent from the JUnit-Core altogether and rerun the tests to see if this reduces the memory consumptions.