Repository: fedora-infra/bodhi

Issue:
bodhi not approving rawhide updates to stable
Bodhi seems to not be approving rawhide updates to stable. 

The logs have: 

```
2020-06-14 17:43:01,831: ERROR/ForkPoolWorker-1] There was an error approving testing updates.
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/koji/__init__.py", line 2785, in _callMethod
    return self._sendCall(handler, headers, request)
  File "/usr/lib/python3.8/site-packages/koji/__init__.py", line 2703, in _sendCall
    return self._sendOneCall(handler, headers, request)
  File "/usr/lib/python3.8/site-packages/koji/__init__.py", line 2749, in _sendOneCall
    ret = self._read_xmlrpc_response(r)
  File "/usr/lib/python3.8/site-packages/koji/__init__.py", line 2761, in _read_xmlrpc_response
    result = u.close()
  File "/usr/lib64/python3.8/xmlrpc/client.py", line 655, in close
    raise Fault(**self._stack[0])
xmlrpc.client.Fault: <Fault 1000: "Invalid tagInfo: 'f33-build-side-24506-signing-pending'">

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/bodhi/server/tasks/approve_testing.py", line 46, in main
    approve_update(update, db)
  File "/usr/lib/python3.8/site-packages/bodhi/server/tasks/approve_testing.py", line 137, in approve_update
    koji.deleteTag(pending_signing_tag)
  File "/usr/lib/python3.8/site-packages/koji/__init__.py", line 2263, in __call__
    return self.__func(self.__name, args, opts)
  File "/usr/lib/python3.8/site-packages/koji/__init__.py", line 2804, in _callMethod
    raise err
koji.GenericError: Invalid tagInfo: 'f33-build-side-24506-signing-pending'
[2020-06-14 17:43:01,858: INFO/ForkPoolWorker-1] Task approve_testing[198076d8-2b59-4424-afc3-d291a090bad8] succeeded in 19.52551427000435s: None
```

And even tho it has 'Task...succeeded' it doesn't actually approve any. 


Discussion:
Some thoughts about this when I briefly looked into it:
````
  pingou | so the builds in https://bodhi.fedoraproject.org/updates/FEDORA-2020-11078fbad3 made it to F33                                                
  pingou | but the update wasn't marked as stable because koji said that f33-build-side-24510-signing-pending is invalid                                 
  pingou | I wonder if this error ^ isn't what blocks other updates from being created                                                                   
ignatenkobrain | pingou: I've actually had some packages being pushed through few hours ago                                                                    
ignatenkobrain | so it should not be related to my issue with btrfs-progs                                                                                      
  pingou | ignatenkobrain: it could be that at the next run of monitor-gating, a new update is created which obsolete the old one that is then ignored   
  pingou | allowing other updates to be processed in the mean while                                                                                      
  pingou | this also explains why I keep receiving emails from bodhi about my updates :(   
````

What I don't understand is why f33-build-side-24510-signing-pending no longer exists:
https://koji.fedoraproject.org/koji/search?match=glob&type=tag&terms=f33-build-side-24510-signing-pending
while f33-build-side-24510 still does:
https://koji.fedoraproject.org/koji/search?match=glob&type=tag&terms=f33-build-side-24510
(at the time of this writing)
So it looks like koji is deleting tags when the last build in the tag was untagged. That makes bodhi unhappy when it is trying to delete the tag since this tag does not exists anymore.

I think we should let koji manage the deletion of tags and side tags (https://pagure.io/fedora-infra/ansible/pull-request/137). I ll open a PR to remove the tag deletion code we have in bodhi.
So in fact it looks like bodhi is deleting the signing-pending and testing-pending tags 
```
 [cverna@localhost]  [fix_editing_updates] $ koji list-history --tag=f33-build-side-24581-signing-pending
Wed Jun 17 11:23:22 2020 new tag: f33-build-side-24581-signing-pending by bodhi
Wed Jun 17 11:23:22 2020 inheritance line f33-build-side-24581-signing-pending->f33-build-side-24581 added by bodhi
Wed Jun 17 11:23:23 2020 dummy-test-package-crested-0-670.fc33 tagged into f33-build-side-24581-signing-pending by bodhi
Wed Jun 17 11:23:23 2020 dummy-test-package-rubino-0-670.fc33 tagged into f33-build-side-24581-signing-pending by bodhi
Wed Jun 17 11:23:26 2020 dummy-test-package-crested-0-670.fc33 untagged from f33-build-side-24581-signing-pending by autopen
Wed Jun 17 11:23:28 2020 dummy-test-package-rubino-0-670.fc33 untagged from f33-build-side-24581-signing-pending by autopen
Wed Jun 17 12:11:34 2020 tag deleted: f33-build-side-24581-signing-pending by bodhi
Wed Jun 17 12:11:34 2020 inheritance line f33-build-side-24581-signing-pending->f33-build-side-24581 removed by bodhi

```
OK I got to the bottom of it, the problem comes from https://pagure.io/koji/issue/2321, Bodhi is not allowed to delete side-tag anymore which makes the approve_testing tasks crash. But before crashing Bodhi manages to delete the signing-pending and testing-pending tags. So the next time the task runs we try to push the update to stable again but when bodhi tries to delete these 2 tags they don't exist anymore.

I think the solution is adding a db commit after updating the status of the update so that if there is a failure in koji deleting tag, the update will not be stuck in testing for ever.

I also think that we should take advantage of koji to delete the sidetags (https://pagure.io/fedora-infra/ansible/pull-request/137)
Hotfixed production with https://github.com/cverna/bodhi/commit/c8866578cd3f46e06bac3d842864e02706b79aaf

Resolving patch:
diff --git a/bodhi/server/buildsys.py b/bodhi/server/buildsys.py
--- a/bodhi/server/buildsys.py
+++ b/bodhi/server/buildsys.py
@@ -139,7 +139,6 @@ def clear(cls):
         cls.__rpms__ = []
         cls.__tags__ = []
         cls.__side_tags__ = list(cls._side_tag_data)
-        cls.__removed_side_tags__ = []
 
     def multiCall(self):
         """Emulate Koji's multiCall."""
@@ -541,23 +540,6 @@ def deleteTag(self, tagid: typing.Union[str, int]):
         else:
             del self.__tags__[tagid]
 
-    def removeSideTag(self, side_tag):
-        """Emulate side-tag and build target deletion."""
-        if isinstance(side_tag, int):
-            what = 'id'
-        elif isinstance(side_tag, str):
-            what = 'name'
-        else:
-            raise TypeError(f'sidetag: {side_tag!r}')
-
-        matching_tags = [t for t in self.__side_tags__ if t[what] == side_tag]
-
-        if not matching_tags:
-            raise koji.GenericError(f"Not a sidetag: {side_tag}")
-
-        self.__side_tags__.remove(matching_tags[0])
-        self.__removed_side_tags__.append(matching_tags[0])
-
     def getRPMHeaders(self, rpmID: str,
                       headers: typing.Any) -> typing.Union[typing.Mapping[str, str], None]:
         """
diff --git a/bodhi/server/tasks/handle_side_and_related_tags.py b/bodhi/server/tasks/handle_side_and_related_tags.py
--- a/bodhi/server/tasks/handle_side_and_related_tags.py
+++ b/bodhi/server/tasks/handle_side_and_related_tags.py
@@ -59,7 +59,6 @@ def main(builds: typing.List[str],
             # side tag into the release pending_signing and candidate tag.
             # We can remove the side tag.
             tags.append(candidate_tag)
-            koji.removeSideTag(from_tag)
 
         koji.multicall = True
         for b in builds:


Test patch:
diff --git a/bodhi/server/tasks/approve_testing.py b/bodhi/server/tasks/approve_testing.py
--- a/bodhi/server/tasks/approve_testing.py
+++ b/bodhi/server/tasks/approve_testing.py
@@ -124,6 +124,7 @@ def approve_update(update: Update, db: Session):
             update.date_stable = update.date_pushed = func.current_timestamp()
             update.comment(db, "This update has been submitted for stable by bodhi",
                            author=u'bodhi')
+            db.commit()
             # Multi build update
             if update.from_tag:
                 # Merging the side tag should happen here
@@ -136,8 +137,7 @@ def approve_update(update: Update, db: Session):
                 koji = buildsys.get_session()
                 koji.deleteTag(pending_signing_tag)
                 koji.deleteTag(testing_tag)
-                # Removes the tag and the build target from koji.
-                koji.removeSideTag(update.from_tag)
+
             else:
                 # Single build update
                 update.remove_tag(update.release.pending_testing_tag)
diff --git a/bodhi/tests/server/tasks/test_handle_side_and_related_tags.py b/bodhi/tests/server/tasks/test_handle_side_and_related_tags.py
--- a/bodhi/tests/server/tasks/test_handle_side_and_related_tags.py
+++ b/bodhi/tests/server/tasks/test_handle_side_and_related_tags.py
@@ -37,7 +37,6 @@ def test_side_tag_composed_by_bodhi(self):
         koji = buildsys.get_session()
         assert ('f17-updates-signing-pending', 'bodhi-2.0-1.fc17') in koji.__added__
         assert ('f17-updates-candidate', 'bodhi-2.0-1.fc17') in koji.__added__
-        assert {'id': 1234, 'name': 'f17-build-side-1234'} in koji.__removed_side_tags__
 
     def test_side_tag_not_composed_by_bodhi(self):
         u = self.db.query(models.Update).first()
@@ -56,6 +55,7 @@ def test_side_tag_not_composed_by_bodhi(self):
     def test_side_tag_raise_exception(self, caplog):
         update = self.db.query(models.Update).first()
         builds = [b.nvr for b in update.builds]
+        update.release.pending_signing_tag = None
         handle_srtags_main(builds, update.release.pending_signing_tag, None, None,
                            update.release.candidate_tag)
         assert "There was an error handling side-tags updates" in caplog.messages