Make shutdown retries explicit in benchmarks.

It seems to be stuck on Travis-CI but I can't duplicate it locally.