Fix race condition in retry_all, fixes #1851