Collect jobs before retry all. Fixes #686.