Test that the checker is really running

Fixes test freezes with unusable checkers.