Repository: fedora-infra/bodhi

Issue:
Autopush seems to be broken
Usually when an update reaches the stable karma threshold, there will be a comment from the bodhi user stating that the update has reached this status and that it will be pushed to stable. For example, this update from bodhi-2.0-212 exibited that behavior:

https://bodhi.fedoraproject.org/updates/FEDORA-2016-ffc86f2b30

Since deploying 2.2.0 to production, it has been noted that updates no longer seem to get that comment from Bodhi when they reach stable karma. For example, see this update in stg:

https://bodhi.stg.fedoraproject.org/updates/FEDORA-2016-4d0198119a

My comment should have caused bodhi to submit it for stable automatically, but it did not. Here is one more example of a non-critical path update that should be autopushed:

https://bodhi.stg.fedoraproject.org/updates/FEDORA-2016-93c58fa1b3



Discussion:
@bowlofeggs Not sure why. We have test for this.

What about this one? https://bodhi.fedoraproject.org/updates/quantum-espresso-5.4.0-10.fc25
This seems to be autopushed. 

@trishnaguha I think that update may have happened before we finished updating production to 2.2.0. The outage started at 19:00:00 UTC that day, and that update comment came it at ~19:25, but it took us longer than that to get everything done. I don't know the exact time we started/finished, but I could probably find out in the logs if we want to know for sure. I could be wrong.

Furthermore, there is some suspicion that the way we resumed a failed push might have put some strange data into the database that might be causing this issue for some updates. It's possible also that on some updates, the client submitted a form with the old version of bodhi running in the browser that was in some way incompatible with the new server. We really don't know what is causing this, but there are definitely updates in production that have some issue here. I was even able to reproduce the problem in stg by adding a comment that should have triggered the stability.

This may be related to https://github.com/fedora-infra/bodhi/issues/949 in some way, but that's just a guess.

@puiterwijk has come to the rescue again! It turns out that there was a commit a while ago that added an autokarma column to the Update model. A later commit added a schema migration, but the schema migration did not include a data migration to fill out this column for existing commits. When the code that detected autokarma was changed to use this column, all existing Updates were skipped in the autokarma logic since they had their autokarma columns set to None.

I will fix this by writing a new migration that sets all Updates with autokarma set to NULL to True.


Resolving patch:
diff --git a/alembic/versions/4df1fcd59050_.py b/alembic/versions/4df1fcd59050_.py
--- a/alembic/versions/4df1fcd59050_.py
+++ b/alembic/versions/4df1fcd59050_.py
@@ -1,7 +1,7 @@
 """Add the new pending_signing_tag column to the releases table.
 
 Revision ID: 4df1fcd59050
-Revises: 37f38ddc4c8d
+Revises: 5110dfc1a01a
 Create Date: 2016-09-16 18:51:19.514301
 
 """
@@ -11,7 +11,7 @@
 
 # revision identifiers, used by Alembic.
 revision = '4df1fcd59050'
-down_revision = '37f38ddc4c8d'
+down_revision = '5110dfc1a01a'
 
 
 def upgrade():
diff --git a/alembic/versions/5110dfc1a01a_set_autokarma_to_true_for_updates_.py b/alembic/versions/5110dfc1a01a_set_autokarma_to_true_for_updates_.py
new file mode 100644
--- /dev/null
+++ b/alembic/versions/5110dfc1a01a_set_autokarma_to_true_for_updates_.py
@@ -0,0 +1,32 @@
+"""Set NULL autokarma to True for Updates that have non-NULL stable karma thresholds.
+
+Revision ID: 5110dfc1a01a
+Revises: 37f38ddc4c8d
+Create Date: 2016-09-24 02:53:42.025785
+
+"""
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision = '5110dfc1a01a'
+down_revision = '37f38ddc4c8d'
+
+
+def upgrade():
+    """Set NULL autokarma to True for Updates that have non-NULL stable karma thresholds."""
+    # Build a fake mini version of the updates table so we can form an UPDATE statement.
+    updates = sa.sql.table('updates', sa.sql.column('autokarma', sa.Boolean),
+                           sa.sql.column('stable_karma', sa.Integer))
+    # Set autokarma to True if there is a stable threshold set and autokarma is None.
+    op.execute(
+        updates.update().where(
+            updates.c.stable_karma != None).where(updates.c.autokarma == None).values(
+                {'autokarma': True}))
+
+
+def downgrade():
+    """There isn't a way to downgrade this migration."""
+    pass


Test patch: