Tests mail fail with an exception, we need to retry those too.