Ported TestCanceled's rerunner changes from kepler-nightly branch.