Repository: fedora-infra/bodhi

Issue:
traceback when determining tag actions
We hit this traceback in a mash today.

```
[2015-08-28 17:41:52][    fedmsg    INFO] Updating comps                                                            
[2015-08-28 17:41:52][     bodhi    INFO] Running ['git', 'pull']
[2015-08-28 17:41:52][     bodhi    INFO] Running ['make']
[2015-08-28 17:41:52][    fedmsg    INFO] Mashing f21-updates-testing
[2015-08-28 17:41:52][    fedmsg    INFO] Running post-request actions on updates
[2015-08-28 17:41:52][     bodhi    INFO] Running ['mash', '-o', '/mnt/koji/mash/updates/f21-updates-testing-150828.
[2015-08-28 17:41:52][    fedmsg    INFO] Generating testing digest for F21
[2015-08-28 17:42:20][    fedmsg   ERROR] Cannot find relevant tag for python-affinity-0.1.0-5.fc23: ['f23', 'f23-up
[2015-08-28 17:42:20][    fedmsg   ERROR] Exception in MasherThread(f23-updates-testing)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/bodhi/masher.py", line 298, in work
self.determine_tag_actions()
File "/usr/lib/python2.7/site-packages/bodhi/masher.py", line 513, in determine_tag_actions
raise Exception
Exception
```

So, we raise that exception there.  Perhaps instead we could try to gracefully recover by just ejecting whichever updates fail at that point.

The sticking point there is that `determine_tag_actions` currently happens _after_ we try to update the security bugs.  So, maybe we want to reverse the order there if we can.. and do the tag actions first, and then the security bugs.  Possible?



Discussion:

Resolving patch:
diff --git a/bodhi/consumers/masher.py b/bodhi/consumers/masher.py
--- a/bodhi/consumers/masher.py
+++ b/bodhi/consumers/masher.py
@@ -292,13 +292,9 @@ def work(self):
             if self.request is UpdateRequest.stable:
                 self.perform_gating()
 
-            self.update_security_bugs()
+            self.determine_and_perform_tag_actions()
 
-            if not self.state.get('tagged', False):
-                self.determine_tag_actions()
-                self.perform_tag_actions()
-                self.state['tagged'] = True
-                self.save_state()
+            self.update_security_bugs()
 
             self.expire_buildroot_overrides()
             self.remove_pending_tags()
@@ -420,6 +416,7 @@ def perform_gating(self):
     def eject_from_mash(self, update, reason):
         update.locked = False
         text = '%s ejected from the push because %r' % (update.title, reason)
+        log.warn(text)
         update.comment(text, author=u'bodhi')
         update.request = None
         if update in self.state['updates']:
@@ -492,7 +489,12 @@ def update_security_bugs(self):
                 for bug in update.bugs:
                     bug.update_details()
 
-    def determine_tag_actions(self):
+    @checkpoint
+    def determine_and_perform_tag_actions(self):
+        self._determine_tag_actions()
+        self._perform_tag_actions()
+
+    def _determine_tag_actions(self):
         tag_types, tag_rels = Release.get_tags()
         for update in sorted_updates(self.updates):
             if update.status is UpdateStatus.testing:
@@ -508,9 +510,10 @@ def determine_tag_actions(self):
                         from_tag = tag
                         break
                 else:
-                    self.log.error('Cannot find relevant tag for %s: %s' % (
-                                   build.nvr, tags))
-                    raise Exception
+                    reason = 'Cannot find relevant tag for %s: %s' % (
+                        build.nvr, tags)
+                    self.eject_from_mash(update, reason)
+                    break
 
                 if self.skip_mash:
                     self.add_tags.append((update.requested_tag, build.nvr))
@@ -518,7 +521,7 @@ def determine_tag_actions(self):
                     self.move_tags.append((from_tag, update.requested_tag,
                                            build.nvr))
 
-    def perform_tag_actions(self):
+    def _perform_tag_actions(self):
         self.koji.multicall = True
         for action in self.add_tags:
             tag, build = action


Test patch: