Attempt to recover as best as possible if the runner fails

This isn't likely to happen, but it does occasionally, so we need to
handle that case.