Repository: fedora-infra/bodhi

Issue:
Auto obsoletion not working in rawhide
Looking at the list of updates in Testing state for Rawhide, it seems there are some packages which are not obsoleted automatically. All those packages were failing gating tests and never reached the Stable state.
https://bodhi.fedoraproject.org/updates/?search=&status=testing&releases=F32


Discussion:
What to do with updates which fail CI? Should they be unpushed instantly at failure, since they cannot reach stable?
I think we indented it that way when implementing the change (ie not have the update set to obsolete) but I am not sure why anymore :stuck_out_tongue: 

@pypingou do you remember ?
If we obsolete a build -1 for -2 and -2 gets gated while -1 passed its tests, nothing would go out, in the current mode at least -1 goes out.

If -1 fails gating and -2 pass, then -1 will stay in limbo and -2 will go to stable.

However, I agree that we need a mechanism to allow withdrawing updates that failed gating.
Unless I'm mistaken, the current workflow when adding a new update is:

1. Create package in db
2. Add builds
3. Create update with Update.new()
   1. Add update in db
   2. Set update.request to Testing if the update is not from sidetag (I don't understand where updates from sidetags are pushed to testing)
   3. Set test_gating_status to waiting
4. Set Koji tags
5. Obsolete older builds

So it seems to me that updates are pushed to Testing before CI gating.
https://github.com/fedora-infra/bodhi/blob/1be3f91c9da1b7ba1f7556999091511a674ad317/bodhi/server/models.py#L2279-L2286
In my opinion, the Update should stay in Pending and we must not set koji tags and obsolete builds  until we receive CI results, then we decide if it can be pushed to Testing and obsolete older builds or it should be unpushed instantly.
Instead I can't find anything in the code that gates the update based on its test_gating_status. Only the push to Stable is gated.
On Sat, Feb 08, 2020 at 12:05:33AM -0800, Mattia Verga wrote:
>    Unless I'm mistaken, the current workflow when adding a new update is:
> 
>     1. Create package in db
>     2. Add builds
>     3. Create update with Update.new()
>          1. Add update in db
>          2. Set update.request to Testing if the update is not from sidetag
>             (I don't understand where updates from sidetags are pushed to
>             testing)
>          3. Set test_gating_status to waiting
>     4. Set Koji tags
>     5. Obsolete older builds
> 
>    So it seems to me that updates are pushed to Testing before CI gating.
>    https://github.com/fedora-infra/bodhi/blob/1be3f91c9da1b7ba1f7556999091511a674ad317/bodhi/server/models.py#L2279-L2286
>    In my opinion, the Update should stay in Pending until we receive CI
>    results, then we decide if it can be pushed to Testing and obsolete older
>    builds or it should be unpushed instantly.
>    Instead I can't find anything in the code that gates the update based on
>    its test_gating_status.

Couple of items missing from the description here:

a) we only automatically create updates for (single) build in rawhide, 
b) there is no testing repositoring in rawhide

So: pending == build is waiting to be signed
testing == CI is running
stable == available in the buildroot (not necessarily in the mirrors)

The code that moves updates based on its gating status is a celery bead which
replaces a cron job.

For stable releases, we announce (or should announce) that the update is ready
for testing when it's submitted to testing and again when it's submitted to
stable.
There the code that will leverage the gating status should part of the compose
(iirc :s)


Resolving patch:
diff --git a/bodhi/server/consumers/automatic_updates.py b/bodhi/server/consumers/automatic_updates.py
--- a/bodhi/server/consumers/automatic_updates.py
+++ b/bodhi/server/consumers/automatic_updates.py
@@ -193,6 +193,12 @@ def __call__(self, message: fedora_messaging.api.Message) -> None:
             log.debug("Flushing changes to the database.")
             dbsession.flush()
 
+            # Obsolete older updates which may be stuck in testing due to failed gating
+            try:
+                update.obsolete_older_updates(dbsession)
+            except Exception as e:
+                log.error(f'Problem obsoleting older updates: {e}')
+
         # This must be run after dbsession is closed so changes are committed to db
         alias = update.alias
         work_on_bugs_task.delay(alias, closing_bugs)


Test patch:
diff --git a/bodhi/tests/server/consumers/test_automatic_updates.py b/bodhi/tests/server/consumers/test_automatic_updates.py
--- a/bodhi/tests/server/consumers/test_automatic_updates.py
+++ b/bodhi/tests/server/consumers/test_automatic_updates.py
@@ -28,7 +28,7 @@
 from bodhi.server.config import config
 from bodhi.server.consumers.automatic_updates import AutomaticUpdateHandler
 from bodhi.server.models import (
-    Build, Release, Update, UpdateRequest, UpdateStatus, UpdateType, User
+    Build, Release, TestGatingStatus, Update, UpdateRequest, UpdateStatus, UpdateType, User
 )
 from bodhi.tests.server import base
 
@@ -210,6 +210,81 @@ def test_existing_pending_update(self, caplog):
         assert (f"Build, active update for {self.sample_nvr} exists already, skipping."
                 in caplog.messages)
 
+    def test_obsolete_testing_update(self, caplog):
+        """Assert that older builds stuck in Testing get obsoleted."""
+        caplog.set_level(logging.DEBUG)
+
+        self.handler(self.sample_message)
+        update = self.db.query(Update).filter(
+            Update.builds.any(Build.nvr == self.sample_nvr)
+        ).first()
+        assert update is not None
+        # Simulate update status after failed gating
+        update.status = UpdateStatus.testing
+        update.test_gating_status = TestGatingStatus.failed
+        self.db.flush()
+        # Clear pending messages
+        self.db.info['messages'] = []
+
+        caplog.clear()
+
+        # Create an update with a newer build
+        msg = deepcopy(self.sample_message)
+        msg.body['version'] = '1.3.5'
+        msg.body['build_id'] = 442563
+        self.handler(msg)
+        nvr = self.sample_nvr.replace('1.3.4', '1.3.5')
+        old_update = self.db.query(Update).filter(
+            Update.builds.any(Build.nvr == self.sample_nvr)
+        ).first()
+        new_update = self.db.query(Update).filter(
+            Update.builds.any(Build.nvr == nvr)
+        ).first()
+        assert new_update is not None
+        assert old_update is not None
+        assert new_update.status == UpdateStatus.pending
+        assert old_update.status == UpdateStatus.obsolete
+
+    def test_problem_obsoleting_older_update(self, caplog):
+        """Assert that an error while obsoleting doesn't block a new update being created."""
+        caplog.set_level(logging.DEBUG)
+
+        self.handler(self.sample_message)
+        update = self.db.query(Update).filter(
+            Update.builds.any(Build.nvr == self.sample_nvr)
+        ).first()
+        assert update is not None
+        # Simulate update status after failed gating
+        update.status = UpdateStatus.testing
+        update.test_gating_status = TestGatingStatus.failed
+        self.db.flush()
+        # Clear pending messages
+        self.db.info['messages'] = []
+
+        caplog.clear()
+
+        # Create an update with a newer build
+        msg = deepcopy(self.sample_message)
+        msg.body['version'] = '1.3.5'
+        msg.body['build_id'] = 442563
+        with mock.patch('bodhi.server.models.Update.obsolete_older_updates',
+                        side_effect=Exception('Something gone wrong')):
+            self.handler(msg)
+        assert 'Problem obsoleting older updates: Something gone wrong' in caplog.messages
+
+        # The new update should have been created and the old one should be stuck in testing
+        nvr = self.sample_nvr.replace('1.3.4', '1.3.5')
+        old_update = self.db.query(Update).filter(
+            Update.builds.any(Build.nvr == self.sample_nvr)
+        ).first()
+        new_update = self.db.query(Update).filter(
+            Update.builds.any(Build.nvr == nvr)
+        ).first()
+        assert new_update is not None
+        assert old_update is not None
+        assert new_update.status == UpdateStatus.pending
+        assert old_update.status == UpdateStatus.testing
+
     # The following tests cover lesser-travelled code paths.
 
     @mock.patch('bodhi.server.consumers.automatic_updates.transactional_session_maker')