Adding some tracing around the test retry logic

Hopefully we can trace @lukebakken's heisenfails with this.