Repository: pulp/pulpcore

Issue:
Repair API should be tolerant of any failure during the repair process
**Version**
main and older

**Describe the bug**
We have made repair api tolerant to a 404 error but that is not the only error that someone can receive. 

If remote is associated with a proxy and that proxy is limiting connections or misconfigured someway, Then TimeoutException would be a common thing to occur as well. 
If Firewall is not properly configured that may even result in "Connection Reset".

No matter what the failure is, The Repair API, should not halt but continue processing rest of the units that it has identified as corrupted or missing.

**To Reproduce**
Steps to reproduce the behavior:

**Expected behavior**
A clear and concise description of what you expected to happen.

**Additional context**
https://bugzilla.redhat.com/show_bug.cgi?id=2173692#c13
https://github.com/pulp/pulpcore/issues/3611 cope with 404



Discussion:
I would request you to look at 6.13.3 Failed QA BZ's traceback which is here https://bugzilla.redhat.com/show_bug.cgi?id=2218980#c2 and we need to consider "DigestValidationError" error as well in the fix of upstream issue request https://github.com/pulp/pulpcore/issues/4111, as this checksum verification is another issue CU are facing.

Resolving patch:
diff --git a/pulpcore/app/management/commands/handle-artifact-checksums.py b/pulpcore/app/management/commands/handle-artifact-checksums.py
--- a/pulpcore/app/management/commands/handle-artifact-checksums.py
+++ b/pulpcore/app/management/commands/handle-artifact-checksums.py
@@ -2,8 +2,6 @@
 
 from gettext import gettext as _
 
-from aiohttp.client_exceptions import ClientResponseError
-
 from django.conf import settings
 from django.core.management import BaseCommand, CommandError
 from django.db.models import Q, Sum
@@ -126,7 +124,7 @@ def _download_artifact(self, artifact, checksum, file_path):
                         downloader = remote.get_downloader(ra)
                         try:
                             dl_result = downloader.fetch()
-                        except ClientResponseError as e:
+                        except Exception as e:
                             self.stdout.write(
                                 _("Redownload failed from '{}': {}.").format(ra.url, str(e))
                             )
@@ -139,8 +137,9 @@ def _download_artifact(self, artifact, checksum, file_path):
                                 setattr(artifact, checksum, dl_result.artifact_attributes[checksum])
                                 restored = True
                                 break
-                self.stdout.write(_("Deleting unreparable file {}".format(file_path)))
-                artifact.file.delete(save=False)
+                if not restored:
+                    self.stdout.write(_("Deleting unrepairable file {}".format(file_path)))
+                    artifact.file.delete(save=False)
             else:
                 break
         return restored
diff --git a/pulpcore/app/tasks/repository.py b/pulpcore/app/tasks/repository.py
--- a/pulpcore/app/tasks/repository.py
+++ b/pulpcore/app/tasks/repository.py
@@ -4,7 +4,6 @@
 import asyncio
 import hashlib
 
-from aiohttp.client_exceptions import ClientResponseError
 from asgiref.sync import sync_to_async
 from django.core.files.storage import default_storage
 from django.db import transaction
@@ -71,7 +70,7 @@ async def _repair_ca(content_artifact, repaired=None):
             "Artifact {} is unrepairable - no remote source".format(content_artifact.artifact)
         )
         log.warning(
-            "Deleting file for the unreparable artifact {}".format(content_artifact.artifact)
+            "Deleting file for the unrepairable artifact {}".format(content_artifact.artifact)
         )
         await sync_to_async(content_artifact.artifact.file.delete)(save=False)
         return False
@@ -81,7 +80,7 @@ async def _repair_ca(content_artifact, repaired=None):
         downloader = detail_remote.get_downloader(remote_artifact)
         try:
             dl_result = await downloader.run()
-        except ClientResponseError as e:
+        except Exception as e:
             log.warn(_("Redownload failed from '{}': {}.").format(remote_artifact.url, str(e)))
         else:
             if dl_result.artifact_attributes["sha256"] == content_artifact.artifact.sha256:
@@ -94,7 +93,7 @@ async def _repair_ca(content_artifact, repaired=None):
                 if repaired is not None:
                     await repaired.aincrement()
                 return True
-    log.warning("Deleting file for the unreparable artifact {}".format(content_artifact.artifact))
+    log.warning("Deleting file for the unrepairable artifact {}".format(content_artifact.artifact))
     await sync_to_async(content_artifact.artifact.file.delete)(save=False)
     return False
 


Test patch: