Ported TestCanceled's rerunner change from kepler-nightly branch.