Cache compiler tests and print a summary