Aww, there is no fallback in the test runner. This actually disabled all tests on our auto-tester.
See #808 for a fixup.