Repository: fedora-infra/bodhi

Issue:
batched regression: updates are not present in any repo if karma is supplied quick enough
Before batched: If karma was supplied quick enough, the package went stable immediately.
With batched: If karma is supplied quick enough, the package goes to batched, and waits for 2 weeks before going stable. During this time, it's not available in any repository. Instead, it should be pushed to updates-testing repo for the duration of the batched tag.

Example:
https://bodhi.fedoraproject.org/updates/FEDORA-2017-6db5c0829b


Discussion:
This bug is surprisingly tricky to fix. Bodhi's state machine code is in [real bad shape](https://github.com/fedora-infra/bodhi/issues/1723), and is essentially spread all over the codebase. That's how this bug happened in the first place, as its exceedingly easy to make a mistake or overlook something when altering the state machine since it is not clearly expressed in one place as it should be.

I've been considering three different ideas for fixing this bug:

1. I could make ```bodhi-push``` and the masher also select ```pending:batched``` updates when mashing testing updates. This seems like the natural thing to do, but I think it might actually be the trickiest solution because so many if statements in the code need to be touched to say something like "or update.status is pending and update.request is batched".
2. It occurred to me that the ```state:pending``` should always imply ```request:testing```, and that I could adjust the code that looks to mash ```request:testing``` mash ```state:pending``` instead, which would pick up these updates too. However, this solution suffers from a similar problem to the first solution - a whole lot of places in the code would need to be touched and so it's not a simple fix. (The more places I have to touch to fix this, the higher chance I have of introducing new bugs ☺).
3. I could make it so pending updates don't get request:batched with karma, but stay at request:testing. Then after the mash runs, I could have the masher check the mashed updates to see if any of them are eligible to become batched and go ahead and do it. This seems simpler than #1 and #2, except for non-autokarma updates - we need to make sure that developers can't manually mark their updates to be batched when they are state:pending or those will end up in this same situation. That last caveat makes this potential solution a little less simple than it would otherwise be.

I would *really* like to take the time to refactor Bodhi to use a proper state machine implementation so bugs like this happen less frequently, and so that fixing bugs like this is much easier when they do happen. I'm currently debating whether now is the right time to go ahead and just do that refactor or not. I think it would take a long time to do that, and of course, also introduces the risk of new bugs.
I talked with @jeremycline today and he preferred the refactoring approach, but agreed that due to the potential severity of the bug that solution 3 above is a reasonable approach to solve it.

I also talked with @puiterwijk who also concurred that solution 3 would work.

Thus, I am going for solution 3 for now, until I finally refactor all this crazy code one day.

Resolving patch:
diff --git a/bodhi/server/models.py b/bodhi/server/models.py
--- a/bodhi/server/models.py
+++ b/bodhi/server/models.py
@@ -1785,6 +1785,12 @@ def set_request(self, db, action, username):
                         else:
                             action = UpdateRequest.testing
 
+        if action is UpdateRequest.batched and self.status is not UpdateStatus.testing:
+            # We don't want to allow updates to go to batched if they haven't been mashed into the
+            # testing repository yet.
+            raise BodhiException('This update is not in the testing repository yet. It cannot be '
+                                 'requested for batching until it is in testing.')
+
         # Add the appropriate 'pending' koji tag to this update, so tools like
         # AutoQA can mash repositories of them for testing.
         if action is UpdateRequest.testing:
@@ -2379,7 +2385,8 @@ def check_karma_thresholds(self, db, agent):
                     log.info("Automatically marking %s as stable" % self.title)
                     self.set_request(db, UpdateRequest.stable, agent)
                 else:
-                    if self.request not in (UpdateRequest.batched, UpdateRequest.stable):
+                    if self.request not in (UpdateRequest.batched, UpdateRequest.stable) and \
+                            self.status is not UpdateStatus.pending:
                         log.info("Automatically adding %s to batch of updates that will be pushed "
                                  "to stable at a later date" % self.title)
                         self.set_request(db, UpdateRequest.batched, agent)


Test patch:
diff --git a/bodhi/tests/server/services/test_updates.py b/bodhi/tests/server/services/test_updates.py
--- a/bodhi/tests/server/services/test_updates.py
+++ b/bodhi/tests/server/services/test_updates.py
@@ -1,5 +1,8 @@
 # -*- coding: utf-8 -*-
-
+# Copyright 2011-2017 Red Hat, Inc. and others.
+#
+# This file is part of Bodhi.
+#
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public License
 # as published by the Free Software Foundation; either version 2
@@ -2335,8 +2338,7 @@ def test_edit_locked_update(self, publish, *args):
     @mock.patch(**mock_valid_requirements)
     @mock.patch('bodhi.server.notifications.publish')
     def test_pending_update_on_stable_karma_reached_autopush_enabled(self, publish, *args):
-        """ Ensure that pending update directly requests for stable if
-        it hits stable karma before reaching testing state """
+        """Ensure that a pending update stays in testing if it hits stable karma while pending."""
         nvr = u'bodhi-2.0.0-2.fc17'
         args = self.get_update(nvr)
         args['autokarma'] = True
@@ -2355,7 +2357,36 @@ def test_pending_update_on_stable_karma_reached_autopush_enabled(self, publish,
         up = self.db.query(Update).filter_by(title=nvr).one()
 
         self.assertEquals(up.karma, 2)
-        self.assertEquals(up.request, UpdateRequest.batched)
+        self.assertEquals(up.request, UpdateRequest.testing)
+        self.assertEquals(up.status, UpdateStatus.pending)
+
+    @mock.patch(**mock_valid_requirements)
+    @mock.patch('bodhi.server.notifications.publish')
+    def test_pending_urgent_update_on_stable_karma_reached_autopush_enabled(self, publish, *args):
+        """
+        Ensure that a pending urgent update directly requests for stable if
+        it hits stable karma before reaching testing state.
+        """
+        nvr = u'bodhi-2.0.0-2.fc17'
+        args = self.get_update(nvr)
+        args['autokarma'] = True
+        args['stable_karma'] = 2
+        args['unstable_karma'] = -2
+        args['severity'] = 'urgent'
+        self.app.post_json('/updates/', args)
+
+        up = self.db.query(Update).filter_by(title=nvr).one()
+        up.status = UpdateStatus.pending
+        self.db.commit()
+
+        up.comment(self.db, u'WFM', author=u'dustymabe', karma=1)
+        up = self.db.query(Update).filter_by(title=nvr).one()
+
+        up.comment(self.db, u'LGTM', author=u'bowlofeggs', karma=1)
+        up = self.db.query(Update).filter_by(title=nvr).one()
+
+        self.assertEquals(up.karma, 2)
+        self.assertEquals(up.request, UpdateRequest.stable)
         self.assertEquals(up.status, UpdateStatus.pending)
 
     @mock.patch(**mock_valid_requirements)
@@ -4067,6 +4098,7 @@ def test_manually_push_to_stable_based_on_karma(self, publish, *args):
         upd = Update.get(nvr, self.db)
         upd.status = UpdateStatus.testing
         upd.request = UpdateRequest.batched
+        upd.pushed = True
         upd.date_testing = datetime.now() - timedelta(days=1)
         self.db.commit()
 
@@ -4080,6 +4112,7 @@ def test_manually_push_to_stable_based_on_karma(self, publish, *args):
         self.assertEquals(upd.status, UpdateStatus.testing)
         self.assertEquals(upd.request, UpdateRequest.batched)
         self.assertEquals(upd.autokarma, False)
+        self.assertEquals(upd.pushed, True)
 
         text = unicode(config.get('testing_approval_msg_based_on_karma'))
         upd.comment(self.db, text, author=u'bodhi')
@@ -4113,6 +4146,7 @@ def test_manually_push_to_batched_based_on_karma(self, publish, *args):
         # Marks it as testing
         upd = Update.get(nvr, self.db)
         upd.status = UpdateStatus.testing
+        upd.pushed = True
         upd.request = None
         upd.date_testing = datetime.now() - timedelta(days=1)
         self.db.commit()
@@ -4318,6 +4352,9 @@ def test_batched_update(self, publish, *args):
         args = self.get_update('bodhi-2.0.0-3.fc17')
         resp = self.app.post_json('/updates/', args)
         up = self.db.query(Update).filter_by(title=resp.json['title']).one()
+        up.status = UpdateStatus.testing
+        up.request = None
+        up.pushed = True
         up.test_gating_status = TestGatingStatus.passed
         up.comment(self.db, u"foo1", 1, u'foo1')
         up.comment(self.db, u"foo2", 1, u'foo2')
diff --git a/bodhi/tests/server/test_models.py b/bodhi/tests/server/test_models.py
--- a/bodhi/tests/server/test_models.py
+++ b/bodhi/tests/server/test_models.py
@@ -1753,6 +1753,42 @@ def test_unicode_bug_title(self):
             link, (u"<a target='_blank' href='https://bugzilla.redhat.com/show_bug.cgi?id=1'>#1</a>"
                    u" foo\xe9bar"))
 
+    def test_set_request_pending_batched(self):
+        """
+        Ensure that we can't submit an update to batched if it is pending, even if it has met the
+        minimum testing requirements.
+        """
+        req = DummyRequest(user=DummyUser())
+        req.errors = cornice.Errors()
+        req.koji = buildsys.get_session()
+        self.assertEqual(self.obj.status, UpdateStatus.pending)
+        self.obj.stable_karma = 1
+        self.obj.comment(self.db, 'works', karma=1, author='bowlofeggs')
+
+        with self.assertRaises(BodhiException) as exc:
+            self.obj.set_request(self.db, UpdateRequest.batched, req.user.name)
+
+        self.assertEqual(self.obj.request, UpdateRequest.testing)
+        self.assertEqual(self.obj.status, UpdateStatus.pending)
+        self.assertEqual(
+            str(exc.exception),
+            ('This update is not in the testing repository yet. It cannot be requested for '
+             'batching until it is in testing.'))
+
+    def test_set_request_pending_stable(self):
+        """Ensure that we can submit an update to stable if it is pending and has enough karma."""
+        req = DummyRequest(user=DummyUser())
+        req.errors = cornice.Errors()
+        req.koji = buildsys.get_session()
+        self.assertEqual(self.obj.status, UpdateStatus.pending)
+        self.obj.stable_karma = 1
+        self.obj.comment(self.db, 'works', karma=1, author='bowlofeggs')
+
+        self.obj.set_request(self.db, UpdateRequest.stable, req.user.name)
+
+        self.assertEqual(self.obj.request, UpdateRequest.stable)
+        self.assertEqual(self.obj.status, UpdateStatus.pending)
+
     def test_set_request_untested_stable(self):
         """
         Ensure that we can't submit an update for stable if it hasn't met the