Repository: conan-io/conan

Issue:
Retry upload for error '500' if the user is requesting it (concurrent uploading)
To help us debug your issue please explain:

- [ x] I've read the [CONTRIBUTING guide](https://github.com/conan-io/conan/blob/develop/.github/CONTRIBUTING.md).
- [ x] I've specified the Conan version, operating system version and any tool that can be relevant.
- [ x] I've explained the steps to reproduce the error or the motivation/use case of the question/suggestion.

I try to upload a package up to our Artifactory server. We are building for multiple compiler on multiple slaves. Sometimes it happens, that they upload nearly at the same time slot.
I thought, that with "--force --retry 5 --retry-wait 10" it would try again to upload again, when this is happening. Unluckily it stops with the first exception happening.

[hil_drv_cifx] $ cmd.exe /C "conan upload "*" --all --force --retry 5 --retry-wait 10 -r 9d380644-252b-44ff-b4af-feded185890d --confirm && exit %%ERRORLEVEL%%"

ERROR: 
Error uploading file: conanmanifest.txt, 'b'{\r\n  "errors" : [ {\r\n    "status" : 500,\r\n    "message" : "Failed to persist file with sha1: 3093033974f36098596e33acf477de5a9e8ddd87"\r\n  } ]\r\n}''

I am using Conan 1.13.1 on multiple Windows & Linux Jenkins Slaves (all python3.6). We are using the Artifactory plugin for Jenkins (newest version, just some days old)



Discussion:
It looks like a concurrency issue in Artifactory, those clients must be creating/deleting files and something goes wrong in Artifactory trying to regenerate indexes. Probably it is safe for Artifactory to fail, and maybe we in Conan should handle this `error 500` and keep retrying as you have specified it in the command line (I have to check the source code to know if we are doing something different with the `500` and the retries).

Thanks for reporting
Yes, having a look at the source code, retries are only performed for exceptions `NotFoundException` and `ConanException`, which in turn are only raised during the upload process for errors `401` and `403` or an exception thrown from `requests.put`. If an error `500` is returned, then it is handled in `RestV*Methods::_upload_files` (outside the retries). 

IMO we should retry always if the user is requesting it, it would be different for automatic retries.
It sounds good, since we have some errors when uploading packages to Bintray.
I tried to bypass the problem by writing a script in Jenkins (see https://github.com/jfrog/jenkins-artifactory-plugin/issues/133), but got another problem inside the Artifactory plugin.


Resolving patch:
diff --git a/conans/client/rest/uploader_downloader.py b/conans/client/rest/uploader_downloader.py
--- a/conans/client/rest/uploader_downloader.py
+++ b/conans/client/rest/uploader_downloader.py
@@ -63,6 +63,9 @@ def _upload_file(self, url, data,  headers, auth):
                 if auth.token is None:
                     raise AuthenticationException(response.content)
                 raise ForbiddenException(response.content)
+
+            response.raise_for_status()  # Raise HTTPError for bad http response status
+
         except ConanException:
             raise
         except Exception as exc:


Test patch:
diff --git a/conans/test/functional/remote/retry_test.py b/conans/test/functional/remote/retry_test.py
new file mode 100644
--- /dev/null
+++ b/conans/test/functional/remote/retry_test.py
@@ -0,0 +1,104 @@
+# coding=utf-8
+
+import os
+import unittest
+from collections import namedtuple, Counter
+
+import six
+
+from conans.client.rest.uploader_downloader import Uploader
+from conans.errors import AuthenticationException, ForbiddenException
+from conans.test.utils.test_files import temp_folder
+from conans.test.utils.tools import TestBufferConanOutput
+from conans.util.files import save
+from requests.exceptions import HTTPError
+
+
+class response_type:
+    def __init__(self, status_code, content):
+        self.status_code = status_code
+        self.content = content
+
+    def raise_for_status(self):
+        """Raises stored :class:`HTTPError`, if one occurred."""
+
+        http_error_msg = ''
+        if 400 <= self.status_code < 500:
+            http_error_msg = u'%s Client Error: %s' % (self.status_code, self.content)
+
+        elif 500 <= self.status_code < 600:
+            http_error_msg = u'%s Server Error: %s' % (self.status_code, self.content)
+
+        if http_error_msg:
+            raise HTTPError(http_error_msg, response=self)
+
+
+class _RequesterMock:
+    def __init__(self, status_code, content):
+        self.response = response_type(status_code, content)
+
+    def put(self, *args, **kwargs):
+        return self.response
+
+
+class RetryDownloadTests(unittest.TestCase):
+
+    def setUp(self):
+        self.filename = os.path.join(temp_folder(), "anyfile")
+        save(self.filename, "anything")
+
+    def test_error_401(self):
+        output = TestBufferConanOutput()
+        uploader = Uploader(requester=_RequesterMock(401, "content"), output=output, verify=False)
+        with six.assertRaisesRegex(self, AuthenticationException, "content"):
+            uploader.upload(url="fake", abs_path=self.filename, retry=3)
+        output_lines = str(output).splitlines()
+        counter = Counter(output_lines)
+        self.assertEqual(counter["ERROR: content"], 2)
+        self.assertEqual(counter["Waiting 0 seconds to retry..."], 2)
+
+    def test_error_403_forbidden(self):
+        output = TestBufferConanOutput()
+        uploader = Uploader(requester=_RequesterMock(403, "content"), output=output, verify=False)
+        with six.assertRaisesRegex(self, ForbiddenException, "content"):
+            auth = namedtuple("auth", "token")
+            uploader.upload(url="fake", abs_path=self.filename, retry=3, auth=auth("token"))
+        output_lines = str(output).splitlines()
+        counter = Counter(output_lines)
+        self.assertEqual(counter["ERROR: content"], 2)
+        self.assertEqual(counter["Waiting 0 seconds to retry..."], 2)
+
+    def test_error_403_authentication(self):
+        output = TestBufferConanOutput()
+        uploader = Uploader(requester=_RequesterMock(403, "content"), output=output, verify=False)
+        with six.assertRaisesRegex(self, AuthenticationException, "content"):
+            auth = namedtuple("auth", "token")
+            uploader.upload(url="fake", abs_path=self.filename, retry=3, auth=auth(None))
+        output_lines = str(output).splitlines()
+        counter = Counter(output_lines)
+        self.assertEqual(counter["ERROR: content"], 2)
+        self.assertEqual(counter["Waiting 0 seconds to retry..."], 2)
+
+    def test_error_requests(self):
+        class _RequesterMock:
+            def put(self, *args, **kwargs):
+                raise Exception("any exception")
+
+        output = TestBufferConanOutput()
+        uploader = Uploader(requester=_RequesterMock(), output=output, verify=False)
+        with six.assertRaisesRegex(self, Exception, "any exception"):
+            uploader.upload(url="fake", abs_path=self.filename, retry=3)
+        output_lines = str(output).splitlines()
+        counter = Counter(output_lines)
+        self.assertEqual(counter["ERROR: any exception"], 2)
+        self.assertEqual(counter["Waiting 0 seconds to retry..."], 2)
+
+    def test_error_500(self):
+        output = TestBufferConanOutput()
+        uploader = Uploader(requester=_RequesterMock(500, "content"), output=output, verify=False)
+        with six.assertRaisesRegex(self, Exception, "500 Server Error: content"):
+            uploader.upload(url="fake", abs_path=self.filename, retry=3)
+        output_lines = str(output).splitlines()
+        counter = Counter(output_lines)
+        self.assertEqual(counter["ERROR: 500 Server Error: content"], 2)
+        self.assertEqual(counter["Waiting 0 seconds to retry..."], 2)
diff --git a/conans/test/utils/tools.py b/conans/test/utils/tools.py
--- a/conans/test/utils/tools.py
+++ b/conans/test/utils/tools.py
@@ -23,6 +23,7 @@
 from six import StringIO
 from six.moves.urllib.parse import quote, urlsplit, urlunsplit
 from webtest.app import TestApp
+from requests.exceptions import HTTPError
 
 from conans import tools, load
 from conans.client.cache.cache import ClientCache
@@ -109,6 +110,18 @@ def headers(self):
     def ok(self):
         return self.test_response.status_code == 200
 
+    def raise_for_status(self):
+        """Raises stored :class:`HTTPError`, if one occurred."""
+        http_error_msg = ''
+        if 400 <= self.status_code < 500:
+            http_error_msg = u'%s Client Error: %s' % (self.status_code, self.content)
+
+        elif 500 <= self.status_code < 600:
+            http_error_msg = u'%s Server Error: %s' % (self.status_code, self.content)
+
+        if http_error_msg:
+            raise HTTPError(http_error_msg, response=self)
+
     @property
     def content(self):
         return self.test_response.body