Repository: pulp/pulpcore

Issue:
Content summary shows incorrect numbers when previous version is deleted
Author: daviddavis (daviddavis)


Redmine Issue: 8798, https://pulp.plan.io/issues/8798

---

Not sure if this is a bug or intentional but I noticed while working on #8793.

If you create two repo versions (version 1 and 2) and add a content item to each, and then delete version 1, the "added" count will show a count of 1 even though compared to version 0, version 2 has 2 items added:

```
  "content_summary": {                                                                                                                                                                        
    "added": {                                                                                                                                                                                
      "file.file": {                                                                                                                                                                          
        "count": 1,                                                                                                                                                                           
        "href": "/pulp/api/v3/content/file/files/?repository_version_added=/pulp/api/v3/repositories/file/file/f53f39d7-341d-4c4c-8835-1fcddb359791/versions/2/"                              
      }                                                                                                                                                                                       
    },                                                                                                                                                                                        
    "removed": {},                                                                                                                                                                            
    "present": {                                                                                                                                                                              
      "file.file": {                                                                                                                                                                          
        "count": 2,                                                                                                                                                                           
        "href": "/pulp/api/v3/content/file/files/?repository_version=/pulp/api/v3/repositories/file/file/f53f39d7-341d-4c4c-8835-1fcddb359791/versions/2/"                                    
      }                                                                                                                                                                                       
    }                                                                                                                                                                                         
```

Also, when you query the content through the href provided by the content summary, you get back 2 units which doesn't seem to match the count provided:

```
$ http :/pulp/api/v3/content/file/files/?repository_version_added=/pulp/api/v3/repositories/file/file/f53f39d7-341d-4c4c-8835-1fcddb359791/versions/2/                                          
{                                                                                                                                                                                             
    "count": 2,                                                                                                                                                                               
    "next": null,                                                                                                                                                                             
    "previous": null,                                                                                                                                                                         
    "results": [...]
}
```

## Steps to reproduce

```
pulp file repository create --name test1 --remote file
pulp file repository sync --name test1
c1=$(pulp file content list | jq -r ".[0] | with_entries(select([.key] | inside([\"sha256\", \"relative_path\"])))")
c2=$(pulp file content list | jq -r ".[1] | with_entries(select([.key] | inside([\"sha256\", \"relative_path\"])))")

pulp file repository create --name test2
pulp file repository content modify --repository test2 --add-content "[$c1]"
pulp file repository content modify --repository test2 --add-content "[$c2]"

pulp file repository version destroy --repository test2 --version 1
pulp file repository version show --repository test2 --version 2

1. then query the content added
http :/pulp/api/v3/content/file/files/?repository_version_added=/pulp/api/v3/repositories/file/file/f53f39d7-341d-4c4c-8835-1fcddb359791/versions/2/
```





Discussion:
From: @gerrod3 (gerrod)
Date: 2022-01-10T10:54:28Z

---

Applied in changeset commit:pulpcore|d767beb104fa95da4ee6521711cf6953eabda7f2.

Resolving patch:
diff --git a/pulpcore/app/models/repository.py b/pulpcore/app/models/repository.py
--- a/pulpcore/app/models/repository.py
+++ b/pulpcore/app/models/repository.py
@@ -11,8 +11,10 @@
 from dynaconf import settings
 from django.core.validators import MinValueValidator
 from django.db import models, transaction
+from django.db.models import Q
 from django.urls import reverse
 from django_lifecycle import AFTER_UPDATE, BEFORE_DELETE, hook
+from rest_framework.exceptions import APIException
 
 from pulpcore.app.util import batch_qs, get_view_name_for_model
 from pulpcore.constants import ALL_KNOWN_CONTENT_CHECKSUMS
@@ -67,7 +69,7 @@ class Meta:
         verbose_name_plural = "repositories"
 
     def on_new_version(self, version):
-        """Called when new repository versions are created.
+        """Called after a new repository version has been created.
 
         Subclasses are expected to override this to do useful things.
 
@@ -200,10 +202,25 @@ def artifacts_for_version(version):
         return Artifact.objects.filter(content__pk__in=version.content)
 
     @hook(AFTER_UPDATE, when="retain_repo_versions", has_changed=True)
+    def _cleanup_old_versions_hook(self):
+        # Do not attempt to clean up anything, while there is a transaction involving repo versions
+        # still in flight.
+        transaction.on_commit(self.cleanup_old_versions)
+
     def cleanup_old_versions(self):
         """Cleanup old repository versions based on retain_repo_versions."""
+        # I am still curious how, but it was reported that this state can happen in day to day
+        # operations but its easy to reproduce manually in the pulpcore shell:
+        # https://github.com/pulp/pulpcore/issues/2268
+        if self.versions.filter(complete=False).exists():
+            raise RuntimeError(
+                _("Attempt to cleanup old versions, while a new version is in flight.")
+            )
         if self.retain_repo_versions:
-            for version in self.versions.order_by("-number")[self.retain_repo_versions :]:
+            # Consider only completed versions for cleanup
+            for version in self.versions.complete().order_by("-number")[
+                self.retain_repo_versions :
+            ]:
                 _logger.info(
                     _("Deleting repository version {} due to version retention limit.").format(
                         version
@@ -496,7 +513,7 @@ class RepositoryVersionQuerySet(models.QuerySet):
     """A queryset that provides repository version filtering methods."""
 
     def complete(self):
-        return self.exclude(complete=False)
+        return self.filter(complete=True)
 
     def with_content(self, content):
         """
@@ -878,6 +895,9 @@ def _squash(self, repo_relations, next_version):
         repo_relations.filter(version_added=self).update(version_added=next_version)
         repo_relations.filter(version_removed=self).update(version_removed=next_version)
 
+        # Update next version's counts as they have been modified
+        next_version._compute_counts()
+
     def delete(self, **kwargs):
         """
         Deletes a RepositoryVersion
@@ -889,22 +909,34 @@ def delete(self, **kwargs):
         Deletion of a complete RepositoryVersion should be done in a RQ Job.
         """
         if self.complete:
+            if self.repository.versions.complete().count() <= 1:
+                raise APIException(_("Attempt to delete the last remaining version."))
             if settings.CACHE_ENABLED:
                 base_paths = self.distribution_set.values_list("base_path", flat=True)
                 if base_paths:
                     Cache().delete(base_key=base_paths)
 
-            repo_relations = RepositoryContent.objects.filter(repository=self.repository)
-            try:
-                next_version = self.next()
-                self._squash(repo_relations, next_version)
-
-            except RepositoryVersion.DoesNotExist:
-                # version is the latest version so simply update repo contents
-                # and delete the version
-                repo_relations.filter(version_added=self).delete()
-                repo_relations.filter(version_removed=self).update(version_removed=None)
-            super().delete(**kwargs)
+            # Handle the manipulation of the repository version content and its final deletion in
+            # the same transaction.
+            with transaction.atomic():
+                repo_relations = RepositoryContent.objects.filter(
+                    repository=self.repository
+                ).select_for_update()
+                try:
+                    next_version = self.next()
+                    self._squash(repo_relations, next_version)
+
+                except RepositoryVersion.DoesNotExist:
+                    # version is the latest version so simply update repo contents
+                    # and delete the version
+                    repo_relations.filter(version_added=self).delete()
+                    repo_relations.filter(version_removed=self).update(version_removed=None)
+
+                if repo_relations.filter(Q(version_added=self) | Q(version_removed=self)).exists():
+                    raise RuntimeError(
+                        _("Some repo relations of this version were not translated.")
+                    )
+                super().delete(**kwargs)
 
         else:
             with transaction.atomic():
@@ -949,6 +981,10 @@ def __enter__(self):
         Returns:
             RepositoryVersion: self
         """
+        if self.complete:
+            raise RuntimeError(
+                _("This Repository version is complete. It cannot be modified further.")
+            )
         return self
 
     def __exit__(self, exc_type, exc_value, traceback):
@@ -975,15 +1011,16 @@ def __exit__(self, exc_type, exc_value, traceback):
                     unsupported_types = content_types_seen - content_types_supported
                     if unsupported_types:
                         raise ValueError(
-                            "Saw unsupported content types {}".format(unsupported_types)
+                            _("Saw unsupported content types {}").format(unsupported_types)
                         )
 
                     self.complete = True
                     self.repository.next_version = self.number + 1
-                    self.repository.save()
-                    self.save()
+                    with transaction.atomic():
+                        self.repository.save()
+                        self.save()
+                        self._compute_counts()
                     self.repository.cleanup_old_versions()
-                    self._compute_counts()
                     repository.on_new_version(self)
             except Exception:
                 self.delete()


Test patch:
diff --git a/pulpcore/tests/functional/api/using_plugin/test_repo_versions.py b/pulpcore/tests/functional/api/using_plugin/test_repo_versions.py
--- a/pulpcore/tests/functional/api/using_plugin/test_repo_versions.py
+++ b/pulpcore/tests/functional/api/using_plugin/test_repo_versions.py
@@ -357,12 +357,16 @@ def test_delete_last_version(self):
 
     def test_delete_middle_version(self):
         """Delete a middle version."""
-        index = randint(1, len(self.repo_version_hrefs) - 2)
+        index = randint(1, len(self.repo_version_hrefs) - 3)
         delete_version(self.repo, self.repo_version_hrefs[index])
 
         with self.assertRaises(HTTPError):
             get_content(self.repo, self.repo_version_hrefs[index])
 
+        # Check added count is updated properly
+        added = get_added_content_summary(self.repo, self.repo_version_hrefs[index + 1])
+        self.assertEqual(added["file.file"], 2)
+
         for repo_version_href in self.repo_version_hrefs[index + 1 :]:
             artifact_paths = get_artifact_paths(self.repo, repo_version_href)
             self.assertIn(self.content[index]["artifact"], artifact_paths)