Improve rerun test

for troubleshooting TNL-692
Merge pull request #5724 from edx/zoldak/improve-rerun-test

Improve rerun test