Repository: fedora-infra/bodhi

Issue:
bodhi does not untag packages from koji tags properly
"redi" complained on fedora-devel that he receives taskotron failure emails 3 weeks after that update was pushed stable:
https://bodhi.fedoraproject.org/updates/FEDORA-2015-13454

Ouch! In taskotron we look at koji tags to determine which builds to test. It seems that none of those builds from that update were untagged from updates-testing and updates-testing-pending properly:

```
$ koji list-tag-history --build rcssserver3d-0.6.10-2.fc23 | grep "still active"
Thu Jul 23 09:43:43 2015: rcssserver3d-0.6.10-2.fc23 tagged into f23-boost by dtardon [still active]
Fri Aug 14 16:24:14 2015: rcssserver3d-0.6.10-2.fc23 tagged into f23-updates-testing by bodhi [still active]
Fri Aug 21 15:10:04 2015: rcssserver3d-0.6.10-2.fc23 tagged into f23-updates-testing-pending by bodhi [still active]
Tue Aug 25 18:03:53 2015: rcssserver3d-0.6.10-2.fc23 tagged into f23 by bodhi [still active]
```

I believe the two middle lines should not be there. Bodhi did not untag that build from u-t and u-t-p when it was pushed stable. The same for other builds from that update:

```
$ koji list-tag-history --build slic3r-1.2.9-3.fc23 | grep "still active"
Thu Jul 30 00:35:22 2015: slic3r-1.2.9-3.fc23 tagged into f23-boost by ausil [still active]
Fri Aug 14 16:25:30 2015: slic3r-1.2.9-3.fc23 tagged into f23-updates-testing by bodhi [still active]
Fri Aug 21 15:09:46 2015: slic3r-1.2.9-3.fc23 tagged into f23-updates-testing-pending by bodhi [still active]
Tue Aug 25 18:03:49 2015: slic3r-1.2.9-3.fc23 tagged into f23 by bodhi [still active]
```

```
$ koji list-tag-history --build rb_libtorrent-1.0.5-2.fc23 | grep "still active"
Thu Jul 30 00:16:26 2015: rb_libtorrent-1.0.5-2.fc23 tagged into f23-boost by ausil [still active]
Fri Aug 14 16:25:11 2015: rb_libtorrent-1.0.5-2.fc23 tagged into f23-updates-testing by bodhi [still active]
Fri Aug 21 15:10:05 2015: rb_libtorrent-1.0.5-2.fc23 tagged into f23-updates-testing-pending by bodhi [still active]
Tue Aug 25 18:03:50 2015: rb_libtorrent-1.0.5-2.fc23 tagged into f23 by bodhi [still active]
```

But that's not all. I looked into a different update:
https://bodhi.fedoraproject.org/updates/FEDORA-2015-15280
and it seems to have the same problem:

```
$ koji list-tag-history --build rpm-4.13.0-0.rc1.3.fc23  | grep "still active"
Mon Sep  7 18:48:00 2015: rpm-4.13.0-0.rc1.3.fc23 tagged into f23-updates-testing by bodhi [still active]
Tue Sep  8 03:21:27 2015: rpm-4.13.0-0.rc1.3.fc23 tagged into f23 by bodhi [still active]
```

Again, it was not untagged from u-t.

Probably since the Bodhi2 deployment, all builds pushed to stable were not untagged correctly? Or maybe only F23 builds?



Discussion:
That's always how Bodhi has handled branched releases. It adds the f23 tag instead of moving it from f23-updates-testing. But yes, it should have untagged it from -pending.

Hmm, in that case that whole problem should be caused by the extra -pending tag. But just to be sure, I'll check how we process packages which have updates-testing tag, but are actually pushed to stable.

What is the reason for keeping the updates-testing tag even for things pushed to stable? You don't do that for stable releases.

It was a long time ago, but I believe this behavior is from the No Frozen Rawhide initiative. https://fedoraproject.org/wiki/No_Frozen_Rawhide_Proposal

Since a seperate cron job performs the f23 mash during this branched phase, if bodhi did a `koji.moveBuild`, then there is a window where things could potentially disappear from the repos before they made it to 'stable'.

I'm cleaning up the tags for that specific update now, and also working on a cron job to handle the updates-testing case as well.

Hmm, how exactly does the window appear? If `moveBuild` first adds a new tag and then removes the old tag, it should not happen. Unless there are _several_ mashers (one for f23 stable, one for f23 updates, one for f23 updates-testing) which don't run from the same repo specifications, but pull them individually at different times. Or am I completely off the mark?

> I'm cleaning up the tags for that specific update now, and also working on a cron job to handle the updates-testing case as well.

Do you mean that the packages will be untagged from updates-testing eventually, using this cron job? That would be definitely helpful, even though atomicity when changing koji tags would be even better, if possible.

> Hmm, how exactly does the window appear? If moveBuild first adds a new tag and then removes the old tag, it should not happen. Unless there are several mashers (one for f23 stable, one for f23 updates, one for f23 updates-testing) which don't run from the same repo specifications, but pull them individually at different times. Or am I completely off the mark?

Say bodhi does a f23 + f23-updates-testing push. The stuff going from updates-testing -> "stable" gets tagged with f23 and bodhi leaves the updates-testing tag. Later on, a nightly cron job seperate from bodhi mashes the F23 branched repos. If bodhi removed the updates-testing tag, then during that push the "stable" stuff would disappear from the updates-testing repo, and there would be a window of time where it would be unavailable until the nightly branched cron job runs.

> Do you mean that the packages will be untagged from updates-testing eventually, using this cron job? That would be definitely helpful, even though atomicity when changing koji tags would be even better, if possible.

Yeah, I'm thinking the cron job should probably wait a day or so before removing the updates-testing tag.

Thanks for info. I checked Taskotron and we're relying just on `-pending` tags (and I see you fixed that for the update reported above), so we should be fine. But it's definitely helpful to learn that there's intended delay when removing `updates-testing` koji tag from packages. I'll try to keep that in mind if we ever start relying on that in our tests.


Resolving patch:
diff --git a/bodhi/scripts/untag_branched.py b/bodhi/scripts/untag_branched.py
new file mode 100644
--- /dev/null
+++ b/bodhi/scripts/untag_branched.py
@@ -0,0 +1,82 @@
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
+"""
+A script used to remove the pending and testing tags from updates in a branched
+release. Since a seperate task mashes the branched stable repos, this will leave
+those stable updates with the testing tags for 1 day before untagging.
+
+https://github.com/fedora-infra/bodhi/issues/576
+"""
+
+import os
+import sys
+import logging
+import transaction
+
+from datetime import datetime, timedelta
+
+from pyramid.paster import get_appsettings, setup_logging
+from sqlalchemy import engine_from_config
+
+from ..models import DBSession, Release, ReleaseState, Update, UpdateStatus
+
+from bodhi import buildsys
+
+
+def usage(argv):
+    cmd = os.path.basename(argv[0])
+    print('usage: %s <config_uri>\n'
+          '(example: "%s development.ini")' % (cmd, cmd))
+    sys.exit(1)
+
+
+def main(argv=sys.argv):
+    if len(argv) != 2:
+        usage(argv)
+
+    config_uri = argv[1]
+
+    setup_logging(config_uri)
+    log = logging.getLogger(__name__)
+
+    settings = get_appsettings(config_uri)
+    engine = engine_from_config(settings, 'sqlalchemy.')
+    DBSession.configure(bind=engine)
+    koji = buildsys.get_session()
+    one_day = timedelta(days=1)
+    now = datetime.utcnow()
+
+    with transaction.manager:
+        db = DBSession()
+        for release in db.query(Release).filter_by(
+                state=ReleaseState.pending).all():
+            log.info(release.name)
+            for update in db.query(Update).filter_by(
+                    release=release, status=UpdateStatus.stable).all():
+                assert update.date_stable, update.title
+                if now - update.date_stable > one_day:
+                    for build in update.builds:
+                        tags = build.get_tags()
+                        stable_tag = release.dist_tag
+                        testing_tag = release.testing_tag
+                        pending_testing_tag = Release.pending_testing_tag
+                        if stable_tag not in tags:
+                            log.error('%s not tagged as stable %s' % (build.nvr, tags))
+                            continue
+                        if testing_tag in tags:
+                            log.info('Removing %s from %s' % (testing_tag, build.nvr))
+                            koji.untagBuild(testing_tag, build.nvr)
+                        if pending_testing_tag in tags:
+                            log.info('Removing %s from %s' % (pending_testing_tag, build.nvr))
+                            koji.untagBuild(pending_testing_tag, build.nvr)
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -112,6 +112,7 @@
       bodhi = bodhi.cli:cli
       bodhi-push = bodhi.push:push
       bodhi-expire-overrides = bodhi.scripts.expire_overrides:main
+      bodhi-untag-branched = bodhi.scripts.untag_branched:main
       [moksha.consumer]
       masher = bodhi.consumers.masher:Masher
       updates = bodhi.consumers.updates:UpdatesHandler


Test patch: