Repository: fedora-infra/bodhi

Issue:
Users are receiving repeat e-mails every 6 hours
@kparal reported than [an Epiphany update](https://bodhi.fedoraproject.org/updates/FEDORA-2016-f4b5897686) has been e-mailing him every 6 hours with the push to stable message, like this:

```
This update has reached 11 days in testing and can be pushed to stable now if the maintainer wishes
```

He received one at 2016-10-04T08:00, 2016-10-04T14:00, and 2016-10-04T20:00, for example.

We should ensure that these messages only get sent once.



Discussion:
The interesting thing is that I received those emails (about 5 of them I think), but no bodhi comment was added. I'd expect a bodhi comment to be added (as is usually is in this case) and therefore receiving an email.

Also, it seems that every commenter of this update received such emails, according to the response in that update.

That is interesting to note. Perhaps it indicates that there was some error and a transaction was rolled back, which could lead to Bodhi not knowing it had already sent the e-mails. I'm not familiar with that area of the code yet, so that's just a wild guess.

I think the reminder comments and/or emails haven't been working for a couple of weeks. I realised recently that some of my updates (e.g. https://bodhi.fedoraproject.org/updates/FEDORA-2016-3a5fa75880) haven't had the reminder comment added, nor have I received the emails.

For another data point: I gave karma to [an update](https://bodhi.fedoraproject.org/updates/FEDORA-2016-4da6039b17) yesterday at 15:17, and starting at 20:00 I started receiving the notifications (one every 6 hours). (I doesn't really bother me, I'm good at ignoring e-mails.)

I believe I know the cause of both getting too many e-mails, and getting none. The [code that generates these e-mails](https://github.com/fedora-infra/bodhi/blob/2.2.4/bodhi/server/scripts/approve_testing.py#L52-L73) looks like this:

```
    with transaction.manager:
        testing = db.query(Update).filter_by(status=UpdateStatus.testing,
                                             request=None)
        for update in testing:
            # If this release does not have any testing requirements, skip it
            if not update.release.mandatory_days_in_testing:
                print('%s doesn\'t have mandatory days in testing' % update.release.name)
                continue
            # If this has already met testing requirements, skip it
            if update.met_testing_requirements:
                continue
            if update.meets_testing_requirements:
                print('%s now meets testing requirements' % update.title)
                text = config.get('testing_approval_msg') % update.days_in_testing
                update.comment(db, text, author='bodhi')

            # Approval message when testing based on karma threshold
            if update.stable_karma not in (0, None) and update.karma >= update.stable_karma \
                    and not update.autokarma:
                print('%s now reaches stable karma threshold' % update.title)
                text = config.get('testing_approval_msg_based_on_karma')
                update.comment(db, text, author='bodhi')
```

I think there are two contributing factors at play here. The first is that the approve testing executable is making a single transaction for all updates, meaning that if anything within them is an error all changes will be rolled back. The second is that 2.2.0 introduced a new setting, called `testing_approval_msg_based_on_karma` and that setting is not present in production. Therefore, what I think is happening is that:
1. Every 6 hours this job runs,
2. A few e-mails get sent out as various updates meet the requirements,
3. Until an update with enough karma is reached in the query that wants to use that setting,
4. Since the setting doesn't exist, this causes a traceback which causes the transaction to roll back, which explains why we don't see the comments on the updates from Bodhi despite the e-mails having already gone out (you can't roll back sent e-mails ☺).

I think we should fix this two ways:
1. Most obviously, we need to add that setting to production. This will require a Freeze Break Request, since Fedora is currently in a beta freeze. I'll file one today.
2. I believe we should be committing our transaction as we go through the for loop, so that we at least save the changes we make to the database as we go.

While I was in there, I noticed the call to [`met_testing_requirements()`](https://github.com/fedora-infra/bodhi/blob/develop/bodhi/server/models/models.py#L1791-L1812), which looks like this:

```
        min_num_days = self.release.mandatory_days_in_testing
        if min_num_days:
            if not self.meets_testing_requirements:
                return False
        else:
            return True
        for comment in self.comments:
            if comment.user.name == 'bodhi' and \
               comment.text.startswith('This update has reached') and \
               'days in testing and can be pushed to stable now if the ' \
               'maintainer wishes' in comment.text:
                return True
        return False
```

I don't like that this code is using this startwith/endswith stuff, because which message gets put on the comment is actually a setting (see production/staging/development.ini files). Therefore, the code as it is is depending on the setting being set a particular way. I've filed https://github.com/fedora-infra/bodhi/issues/1014 about that.

I will use this issue to track fixing the transaction handling, so problems that happen during the run don't roll back any changes that were made (sending confusing ghost e-mails).

I will use [Richard Fearn's infrastructure ticket](https://pagure.io/fedora-infrastructure/issue/5515) to track fixing the missing setting in production.

The production configuration changes are in place on production now. I will keep this ticket open to track fixing the transactions to be more granular. Thanks for the report and the patience everyone!

Upon further reflection, I believe that fixing the production settings will solve only part of the issue. There is still a problem that can only be fixed by editing the code. `met_testing_requirements()` as cited above only checks for the `testing_approval_msg` having been added to the update, which happens when the update has been in testing for the required amount of time. Unfortunately, that means that updates that hit the karma threshold before the required number of days are going to get the "This update has reached the stable karma threshold and can be pushed to stable now if the maintainer wishes." message every six hours until the required number of days is reached, or until the maintainer pushes the update to stable.

One way we can prevent that from happening is if we fix `met_testing_requirements()` to check for either message instead of only checking for the one about length of time. I still think we can come up with a better check than looking for a comment with particular text in it, but for now we might want a quick fix, such as making `met_testing_requirements()` look like this:

```
            if comment.user.name == 'bodhi' and \
               comment.text.startswith('This update has reached') and \
               'can be pushed to stable now if the ' \
               'maintainer wishes' in comment.text:
                return True
```

The only difference is that I removed "days in testing and " from the second half of the check. It's still not a good way to do it, but I've got another open ticket to fix the way this works and right now a quick fix might be what we want.

I decided to file https://github.com/fedora-infra/bodhi/issues/1016 to track fixing the transaction, so that this ticket can focus solely on fixing the past-comment-detection.

Thanks for looking into this so quickly! I've started getting the reminder emails again now. I did spot another issue and I'll file another issue about that.


Resolving patch:
diff --git a/bodhi/server/models/models.py b/bodhi/server/models/models.py
--- a/bodhi/server/models/models.py
+++ b/bodhi/server/models/models.py
@@ -1783,16 +1783,28 @@ def meets_testing_requirements(self):
                     return False
             num_days = int(config.get('critpath.stable_after_days_without_negative_karma'))
             return self.days_in_testing >= num_days
+
         num_days = self.release.mandatory_days_in_testing
         if not num_days:
             return True
+
+        # non-autokarma updates have met the testing requirements if they've reached the karma
+        # threshold.
+        if not self.autokarma and self.stable_karma not in (0, None)\
+                and self.karma >= self.stable_karma:
+            return True
+
+        # Any update that reaches num_days has met the testing requirements.
         return self.days_in_testing >= num_days
 
     @property
     def met_testing_requirements(self):
         """
         Return whether or not this update has already met the testing
-        requirements.
+        requirements and bodhi has commented on the update that the
+        requirements have been met. This is used to determine whether bodhi
+        should add the comment about the Update's eligibility to be pushed,
+        as we only want Bodhi to add the comment once.
 
         If this release does not have a mandatory testing requirement, then
         simply return True.
@@ -1806,7 +1818,7 @@ def met_testing_requirements(self):
         for comment in self.comments:
             if comment.user.name == 'bodhi' and \
                comment.text.startswith('This update has reached') and \
-               'days in testing and can be pushed to stable now if the ' \
+               'and can be pushed to stable now if the ' \
                'maintainer wishes' in comment.text:
                 return True
         return False


Test patch:
diff --git a/bodhi/server/scripts/approve_testing.py b/bodhi/server/scripts/approve_testing.py
--- a/bodhi/server/scripts/approve_testing.py
+++ b/bodhi/server/scripts/approve_testing.py
@@ -41,13 +41,7 @@ def main(argv=sys.argv):
     if len(argv) != 2:
         usage(argv)
 
-    config_uri = argv[1]
-
-    settings = get_appsettings(config_uri)
-    engine = engine_from_config(settings, 'sqlalchemy.')
-    Session = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))
-    Session.configure(bind=engine)
-    db = Session()
+    db = _get_db_session(argv[1])
 
     with transaction.manager:
         testing = db.query(Update).filter_by(status=UpdateStatus.testing,
@@ -57,10 +51,17 @@ def main(argv=sys.argv):
             if not update.release.mandatory_days_in_testing:
                 print('%s doesn\'t have mandatory days in testing' % update.release.name)
                 continue
+
             # If this has already met testing requirements, skip it
             if update.met_testing_requirements:
                 continue
-            if update.meets_testing_requirements:
+
+            # If autokarma updates have reached the testing threshold, say something! Keep in mind
+            # that we don't care about karma here, because autokarma updates get their request set
+            # to stable by the Update.comment() workflow when they hit the required threshold. Thus,
+            # this function only needs to consider the time requirements because these updates have
+            # not reached the karma threshold.
+            if update.autokarma and update.meets_testing_requirements:
                 print('%s now meets testing requirements' % update.title)
                 text = config.get('testing_approval_msg') % update.release.mandatory_days_in_testing
                 update.comment(db, text, author='bodhi')
@@ -71,3 +72,20 @@ def main(argv=sys.argv):
                 print('%s now reaches stable karma threshold' % update.title)
                 text = config.get('testing_approval_msg_based_on_karma')
                 update.comment(db, text, author='bodhi')
+
+
+def _get_db_session(config_uri):
+    """
+    Construct and return a database session using settings from the given config_uri.
+
+    :param config_uri: A path to a config file to use to get the db settings.
+    :type  config_uri: basestring
+    :return:           A database session
+    """
+    # There are many blocks of code like this in the codebase. We should consolidate them into a
+    # single utility function as described in https://github.com/fedora-infra/bodhi/issues/1028
+    settings = get_appsettings(config_uri)
+    engine = engine_from_config(settings, 'sqlalchemy.')
+    Session = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))
+    Session.configure(bind=engine)
+    return Session()
diff --git a/bodhi/tests/server/base.py b/bodhi/tests/server/base.py
new file mode 100644
--- /dev/null
+++ b/bodhi/tests/server/base.py
@@ -0,0 +1,59 @@
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
+"""
+This module contains a useful base test class that helps with common testing needs when testing
+bodhi.server modules.
+"""
+
+import unittest
+
+from sqlalchemy import create_engine
+from sqlalchemy import event
+from sqlalchemy.orm import scoped_session, sessionmaker
+from zope.sqlalchemy import ZopeTransactionExtension
+
+from bodhi.server import log
+from bodhi.tests.server import populate
+from bodhi.server.models import (
+    Base,
+)
+
+DB_PATH = 'sqlite://'
+DB_NAME = None
+
+
+class BaseTestCase(unittest.TestCase):
+    def setUp(self):
+        engine = create_engine(DB_PATH)
+        # We want a special session that lasts longer than a transaction
+        Session = scoped_session(
+            sessionmaker(bind=engine, extension=ZopeTransactionExtension(keep_session=True)))
+        log.debug('Creating all models for %s' % engine)
+        Base.metadata.bind = engine
+        Base.metadata.create_all(engine)
+        self.db = Session()
+        populate(self.db)
+
+        # Track sql statements in every test
+        self.sql_statements = []
+
+        def track(conn, cursor, statement, param, ctx, many):
+            self.sql_statements.append(statement)
+
+        event.listen(engine, "before_cursor_execute", track)
+
+    def tearDown(self):
+        log.debug('Removing session')
+        self.db.close()
+        del self.sql_statements
diff --git a/bodhi/tests/server/functional/base.py b/bodhi/tests/server/functional/base.py
--- a/bodhi/tests/server/functional/base.py
+++ b/bodhi/tests/server/functional/base.py
@@ -13,23 +13,14 @@
 # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
 
 import os
-import unittest
 
 from webtest import TestApp
-from sqlalchemy import create_engine
-from sqlalchemy import event
-from sqlalchemy.orm import scoped_session, sessionmaker
-from zope.sqlalchemy import ZopeTransactionExtension
 
-from bodhi.server import main, log
-from bodhi.tests.server import populate
-from bodhi.server.models import (
-    Base,
-)
+from bodhi.server import main
+from bodhi.tests.server.base import BaseTestCase, DB_NAME, DB_PATH
+
 
 FAITOUT = 'http://209.132.184.152/faitout/'
-DB_PATH = 'sqlite://'
-DB_NAME = None
 # The BUILD_ID environment variable is set by Jenkins and allows us to detect if
 # we are running the tests in jenkins or not
 # https://wiki.jenkins-ci.org/display/JENKINS/Building+a+software+project#Buildingasoftwareproject-below
@@ -45,7 +36,7 @@
         pass
 
 
-class BaseWSGICase(unittest.TestCase):
+class BaseWSGICase(BaseTestCase):
     app_settings = {
         'sqlalchemy.url': DB_PATH,
         'mako.directories': 'bodhi:server/templates',
@@ -84,26 +75,11 @@ class BaseWSGICase(unittest.TestCase):
     }
 
     def setUp(self):
-        engine = create_engine(DB_PATH)
-        Session = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))
-        Session.configure(bind=engine)
-        log.debug('Creating all models for %s' % engine)
-        Base.metadata.bind = engine
-        Base.metadata.create_all(engine)
-        self.db = Session()
-        populate(self.db)
+        super(BaseWSGICase, self).setUp()
         self.app = TestApp(main({}, testing=u'guest', session=self.db, **self.app_settings))
 
-        # Track sql statements in every test
-        self.sql_statements = []
-        def track(conn, cursor, statement, param, ctx, many):
-            self.sql_statements.append(statement)
-
-        event.listen(engine, "before_cursor_execute", track)
-
     def tearDown(self):
-        log.debug('Removing session')
-        self.db.close()
+        super(BaseWSGICase, self).tearDown()
         if DB_NAME:
             try:
                 import requests
diff --git a/bodhi/tests/server/functional/test_updates.py b/bodhi/tests/server/functional/test_updates.py
--- a/bodhi/tests/server/functional/test_updates.py
+++ b/bodhi/tests/server/functional/test_updates.py
@@ -3025,6 +3025,7 @@ def test_manually_push_to_stable_based_on_karma(self, publish, *args):
         upd = Update.get(nvr, self.db)
         upd.status = UpdateStatus.testing
         upd.request = None
+        upd.date_testing = datetime.now() - timedelta(days=1)
         self.db.flush()
 
         # Checks karma threshold is reached
diff --git a/bodhi/tests/server/models/__init__.py b/bodhi/tests/server/models/__init__.py
--- a/bodhi/tests/server/models/__init__.py
+++ b/bodhi/tests/server/models/__init__.py
@@ -32,9 +32,10 @@ class ModelTest(object):
     klass = None
     attrs = {}
 
-    def setup(self):
+    def setUp(self):
         engine = create_engine('sqlite://')
-        Session = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))
+        Session = scoped_session(
+            sessionmaker(extension=ZopeTransactionExtension(keep_session=True)))
         Session.configure(bind=engine)
         self.db = Session()
         Base.metadata.create_all(engine)
diff --git a/bodhi/tests/server/models/test_models.py b/bodhi/tests/server/models/test_models.py
--- a/bodhi/tests/server/models/test_models.py
+++ b/bodhi/tests/server/models/test_models.py
@@ -514,29 +514,153 @@ def test_set_request_stable_after_week_in_testing(self, publish):
             topic='update.request.stable', msg=mock.ANY)
 
     @mock.patch('bodhi.server.notifications.publish')
-    def test_set_meets_then_met_requirements(self, publish):
-        req = DummyRequest()
-        req.errors = cornice.Errors()
-        req.koji = buildsys.get_session()
-        req.user = model.User(name='bob')
-
+    def test_met_testing_requirements_at_7_days_after_bodhi_comment(self, publish):
+        """
+        Ensure a correct True return value from Update.met_testing_requirements() after an update
+        has been in testing for 7 days and after bodhi has commented about it.
+        """
         self.obj.status = UpdateStatus.testing
-        self.obj.request = None
+        # Pretend it's been in testing for a week
+        self.obj.comment(
+            self.db, u'This update has been pushed to testing.', author=u'bodhi')
+        self.obj.date_testing = self.obj.comments[-1].timestamp - timedelta(days=7)
+        eq_(self.obj.days_in_testing, 7)
+        # The update should be eligible to receive the testing_approval_msg now.
+        eq_(self.obj.meets_testing_requirements, True)
+        # Add the testing_approval_message
+        text = config.get('testing_approval_msg') % self.obj.days_in_testing
+        self.obj.comment(self.db, text, author=u'bodhi')
+
+        # met_testing_requirement() should return True since Bodhi has commented on the Update to
+        # say that it can now be pushed to stable.
+        eq_(self.obj.met_testing_requirements, True)
 
+    @mock.patch('bodhi.server.notifications.publish')
+    def test_met_testing_requirements_at_7_days_before_bodhi_comment(self, publish):
+        """
+        Ensure a correct False return value from Update.met_testing_requirements() after an update
+        has been in testing for 7 days but before bodhi has commented about it.
+        """
+        self.obj.status = UpdateStatus.testing
         # Pretend it's been in testing for a week
         self.obj.comment(
             self.db, u'This update has been pushed to testing.', author=u'bodhi')
         self.obj.date_testing = self.obj.comments[-1].timestamp - timedelta(days=7)
         eq_(self.obj.days_in_testing, 7)
+        # The update should be eligible to receive the testing_approval_msg now.
         eq_(self.obj.meets_testing_requirements, True)
+
+        # Since bodhi hasn't added the testing_approval_message yet, this should be False.
         eq_(self.obj.met_testing_requirements, False)
 
-        text = config.get('testing_approval_msg') % self.obj.days_in_testing
-        self.obj.comment(self.db, text, author=u'bodhi')
+    def test_meets_testing_requirements_with_non_autokarma_update_below_stable_karma(self):
+        """
+        Assert that meets_testing_requirements() correctly returns True for non-autokarma updates
+        that haven't reached the days in testing but have reached the stable_karma threshold.
+        """
+        self.obj.autokarma = False
+        self.obj.status = UpdateStatus.testing
+        self.obj.stable_karma = 1
+
+        # meets_testing_requirement() should return False since the karma threshold has not been
+        # reached (note that this Update does not have any karma).
+        eq_(self.obj.meets_testing_requirements, False)
+
+    def test_meets_testing_requirements_with_non_autokarma_update_reaching_stable_karma(self):
+        """
+        Assert that meets_testing_requirements() correctly returns True for non-autokarma updates
+        that haven't reached the days in testing but have reached the stable_karma threshold.
+        """
+        self.obj.autokarma = False
+        self.obj.status = UpdateStatus.testing
+        self.obj.stable_karma = 1
+        # Now let's add some karma to get it to the required threshold
+        self.obj.comment(self.db, u'testing', author='hunter2', anonymous=False, karma=1)
 
+        # meets_testing_requirement() should return True since the karma threshold has been reached
         eq_(self.obj.meets_testing_requirements, True)
+
+    def test_meets_testing_requirements_with_non_autokarma_update_with_stable_karma_0(self):
+        """
+        Assert that meets_testing_requirements() correctly returns False for non-autokarma updates
+        that haven't reached the days in testing but have stable_karma set to 0 (indicating that the
+        update must stay in testing for the full amount of time).
+        """
+        self.obj.autokarma = False
+        self.obj.status = UpdateStatus.testing
+        self.obj.stable_karma = 0
+        # Now let's add some karma to get it above stable_karma, which should not be counted as
+        # meeting the requirements.
+        self.obj.comment(self.db, u'testing', author='hunter2', anonymous=False, karma=1)
+
+        # meets_testing_requirement() should return False since the stable_karma threshold is 0.
+        eq_(self.obj.meets_testing_requirements, False)
+
+    def test_meets_testing_requirements_with_non_autokarma_update_with_stable_karma_None(self):
+        """
+        Assert that meets_testing_requirements() correctly returns False for non-autokarma updates
+        that haven't reached the days in testing but have stable_karma set to None (indicating that
+        the update must stay in testing for the full amount of time).
+        """
+        self.obj.autokarma = False
+        self.obj.status = UpdateStatus.testing
+        self.obj.stable_karma = None
+        # Now let's add some karma to get it above stable_karma, which should not be counted as
+        # meeting the requirements.
+        self.obj.comment(self.db, u'testing', author='hunter2', anonymous=False, karma=1)
+
+        # meets_testing_requirement() should return False since the stable_karma threshold is None.
+        eq_(self.obj.meets_testing_requirements, False)
+
+    @mock.patch('bodhi.server.notifications.publish')
+    def test_met_testing_requirements_with_karma_after_bodhi_comment(self, publish):
+        """
+        Ensure a correct True return value from Update.met_testing_requirements() after a
+        non-autokarma update has reached the karma requirement and after bodhi has commented about
+        it.
+        """
+        self.obj.autokarma = False
+        self.obj.status = UpdateStatus.testing
+        # Pretend it's been in testing for a day
+        self.obj.comment(
+            self.db, u'This update has been pushed to testing.', author=u'bodhi')
+        self.obj.date_testing = self.obj.comments[-1].timestamp - timedelta(days=1)
+        eq_(self.obj.days_in_testing, 1)
+        # Now let's add some karma to get it to the required threshold
+        self.obj.comment(self.db, u'testing', author='hunter1', anonymous=False, karma=1)
+        self.obj.comment(self.db, u'testing', author='hunter2', anonymous=False, karma=1)
+        self.obj.comment(self.db, u'testing', author='hunter3', anonymous=False, karma=1)
+        # Add the testing_approval_message
+        text = config.get('testing_approval_msg_based_on_karma')
+        self.obj.comment(self.db, text, author=u'bodhi')
+
+        # met_testing_requirement() should return True since Bodhi has commented on the Update to
+        # say that it can now be pushed to stable.
         eq_(self.obj.met_testing_requirements, True)
 
+    @mock.patch('bodhi.server.notifications.publish')
+    def test_met_testing_requirements_with_karma_before_bodhi_comment(self, publish):
+        """
+        Ensure a correct False return value from Update.met_testing_requirements() after a
+        non-autokarma update has reached the karma requirement but before bodhi has commented about
+        it.
+        """
+        self.obj.autokarma = False
+        self.obj.status = UpdateStatus.testing
+        # Pretend it's been in testing for a day
+        self.obj.comment(
+            self.db, u'This update has been pushed to testing.', author=u'bodhi')
+        self.obj.date_testing = self.obj.comments[-1].timestamp - timedelta(days=1)
+        eq_(self.obj.days_in_testing, 1)
+        # Now let's add some karma to get it to the required threshold
+        self.obj.comment(self.db, u'testing', author='hunter1', anonymous=False, karma=1)
+        self.obj.comment(self.db, u'testing', author='hunter2', anonymous=False, karma=1)
+        self.obj.comment(self.db, u'testing', author='hunter3', anonymous=False, karma=1)
+
+        # met_testing_requirement() should return False since Bodhi has not yet commented on the
+        # Update to say that it can now be pushed to stable.
+        eq_(self.obj.met_testing_requirements, False)
+
     @mock.patch('bodhi.server.notifications.publish')
     def test_set_request_obsolete(self, publish):
         req = DummyRequest(user=DummyUser())
diff --git a/bodhi/tests/server/scripts/__init__.py b/bodhi/tests/server/scripts/__init__.py
new file mode 100644
diff --git a/bodhi/tests/server/scripts/test_approve_testing.py b/bodhi/tests/server/scripts/test_approve_testing.py
new file mode 100644
--- /dev/null
+++ b/bodhi/tests/server/scripts/test_approve_testing.py
@@ -0,0 +1,237 @@
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
+"""
+This module contains tests for the bodhi.server.scripts.approve_testing module.
+"""
+from datetime import datetime, timedelta
+
+from mock import patch
+import transaction
+
+from bodhi.server.config import config
+from bodhi.server.models import models
+from bodhi.server.scripts import approve_testing
+from bodhi.tests.server.base import BaseTestCase
+
+
+class TestMain(BaseTestCase):
+    """
+    This class contains tests for the main() function.
+    """
+    def test_autokarma_update_meeting_time_requirements_gets_one_comment(self):
+        """
+        Ensure that an update that meets the required time in testing gets only one comment from
+        Bodhi to that effect, even on subsequent runs of main().
+        """
+        update = self.db.query(models.Update).all()[0]
+        update.autokarma = True
+        update.request = None
+        update.stable_karma = 10
+        update.status = models.UpdateStatus.testing
+        update.date_testing = datetime.now() - timedelta(days=7)
+        self.db.flush()
+
+        with patch(
+                'bodhi.server.scripts.approve_testing._get_db_session', return_value=self.db):
+            approve_testing.main(['nosetests', 'some_config.ini'])
+
+            # Now we will run main() again, but this time we expect Bodhi not to add any further
+            # comments.
+            approve_testing.main(['nosetests', 'some_config.ini'])
+
+        bodhi = self.db.query(models.User).filter_by(name=u'bodhi').one()
+        comment_q = self.db.query(models.Comment).filter_by(update_id=update.id, user_id=bodhi.id)
+        self.assertEqual(comment_q.count(), 1)
+        self.assertEqual(
+            comment_q[0].text,
+            config.get('testing_approval_msg') % update.release.mandatory_days_in_testing)
+
+    # Set the release's mandatory days in testing to 0 to set up the condition for this test.
+    @patch.dict(config, [('fedora.mandatory_days_in_testing', 0)])
+    def test_autokarma_update_without_mandatory_days_in_testing(self):
+        """
+        If the Update's release doesn't have a mandatory days in testing, main() should ignore it
+        (and should not comment on the update at all, even if it does reach karma levels.)
+        """
+        update = self.db.query(models.Update).all()[0]
+        update.autokarma = True
+        update.request = None
+        update.status = models.UpdateStatus.testing
+        update.date_testing = datetime.now() - timedelta(days=7)
+        # Let's delete all the comments to make our assertion at the end of this simpler.
+        for c in update.comments:
+            self.db.delete(c)
+        transaction.commit()
+
+        with patch(
+                'bodhi.server.scripts.approve_testing._get_db_session', return_value=self.db):
+            approve_testing.main(['nosetests', 'some_config.ini'])
+
+        # The bodhi user shouldn't exist, since it shouldn't have made any comments
+        self.assertEqual(self.db.query(models.User).filter_by(name=u'bodhi').count(), 0)
+        self.assertEqual(self.db.query(models.Comment).count(), 0)
+
+    def test_autokarma_update_not_meeting_testing_requirments(self):
+        """
+        If an autokarma update has not met the testing requirements, bodhi should not comment on the
+        update.
+        """
+        update = self.db.query(models.Update).all()[0]
+        update.autokarma = True
+        update.request = None
+        update.status = models.UpdateStatus.testing
+        # 6 days isn't enough time to meet the testing requirements.
+        update.date_testing = datetime.now() - timedelta(days=6)
+        # Let's delete all the comments to make our assertion at the end of this simpler.
+        for c in update.comments:
+            self.db.delete(c)
+        transaction.commit()
+
+        with patch(
+                'bodhi.server.scripts.approve_testing._get_db_session', return_value=self.db):
+            approve_testing.main(['nosetests', 'some_config.ini'])
+
+        # The bodhi user shouldn't exist, since it shouldn't have made any comments
+        self.assertEqual(self.db.query(models.User).filter_by(name=u'bodhi').count(), 0)
+        self.assertEqual(self.db.query(models.Comment).count(), 0)
+
+    def test_non_autokarma_update_meeting_karma_requirements_gets_one_comment(self):
+        """
+        Ensure that a non-autokarma update that meets the required karma threshold gets only one
+        comment from Bodhi to that effect, even on subsequent runs of main(). There was an issue[0]
+        where Bodhi wasn't correctly detected that it has already commented on updates, and would
+        repeatedly comment that non-autokarma updates could be pushed. This test ensures that issue
+        stays fixed.
+
+        [0] https://github.com/fedora-infra/bodhi/issues/1009
+        """
+        update = self.db.query(models.Update).all()[0]
+        update.autokarma = False
+        update.request = None
+        update.stable_karma = 1
+        update.status = models.UpdateStatus.testing
+        update.comment(self.db, 'testing', author='hunter2', anonymous=False, karma=1)
+
+        with patch(
+                'bodhi.server.scripts.approve_testing._get_db_session', return_value=self.db):
+            approve_testing.main(['nosetests', 'some_config.ini'])
+
+            # Now we will run main() again, but this time we expect Bodhi not to add any further
+            # comments.
+            approve_testing.main(['nosetests', 'some_config.ini'])
+
+        bodhi = self.db.query(models.User).filter_by(name=u'bodhi').one()
+        comment_q = self.db.query(models.Comment).filter_by(update_id=update.id, user_id=bodhi.id)
+        self.assertEqual(comment_q.count(), 1)
+        self.assertEqual(comment_q[0].text, config.get('testing_approval_msg_based_on_karma'))
+
+    def test_non_autokarma_update_with_stable_karma_0(self):
+        """
+        A non-autokarma update with a stable_karma set to 0 should not get comments from Bodhi.
+        """
+        update = self.db.query(models.Update).all()[0]
+        update.autokarma = False
+        update.request = None
+        update.stable_karma = 0
+        update.status = models.UpdateStatus.testing
+        update.comment(self.db, 'testing', author='hunter2', anonymous=False, karma=1)
+
+        with patch(
+                'bodhi.server.scripts.approve_testing._get_db_session', return_value=self.db):
+            approve_testing.main(['nosetests', 'some_config.ini'])
+
+        # The bodhi user shouldn't exist, since it shouldn't have made any comments
+        self.assertEqual(self.db.query(models.User).filter_by(name=u'bodhi').count(), 0)
+        # There are three comments, but none from the non-existing bodhi user.
+        self.assertEqual(self.db.query(models.Comment).count(), 3)
+        usernames = [
+            c.user.name
+            for c in self.db.query(models.Comment).order_by(models.Comment.timestamp).all()]
+        self.assertEqual(usernames, [u'guest', u'anonymous', u'hunter2'])
+
+    def test_non_autokarma_update_with_stable_karma_None(self):
+        """
+        A non-autokarma update with a stable_karma set to None should not get comments from Bodhi.
+        """
+        update = self.db.query(models.Update).all()[0]
+        update.autokarma = False
+        update.request = None
+        update.stable_karma = None
+        update.status = models.UpdateStatus.testing
+        update.comment(self.db, 'testing', author='hunter2', anonymous=False, karma=1)
+
+        with patch(
+                'bodhi.server.scripts.approve_testing._get_db_session', return_value=self.db):
+            approve_testing.main(['nosetests', 'some_config.ini'])
+
+        # The bodhi user shouldn't exist, since it shouldn't have made any comments
+        self.assertEqual(self.db.query(models.User).filter_by(name=u'bodhi').count(), 0)
+        # There are three comments, but none from the non-existing bodhi user.
+        self.assertEqual(self.db.query(models.Comment).count(), 3)
+        usernames = [
+            c.user.name
+            for c in self.db.query(models.Comment).order_by(models.Comment.timestamp).all()]
+        self.assertEqual(usernames, [u'guest', u'anonymous', u'hunter2'])
+
+    def test_non_autokarma_update_with_unmet_karma_requirement(self):
+        """
+        A non-autokarma update without enough karma should not get comments from Bodhi.
+        """
+        update = self.db.query(models.Update).all()[0]
+        update.autokarma = False
+        update.request = None
+        update.stable_karma = 10
+        update.status = models.UpdateStatus.testing
+        update.comment(self.db, 'testing', author='hunter2', anonymous=False, karma=1)
+
+        with patch(
+                'bodhi.server.scripts.approve_testing._get_db_session', return_value=self.db):
+            approve_testing.main(['nosetests', 'some_config.ini'])
+
+        # The bodhi user shouldn't exist, since it shouldn't have made any comments
+        self.assertEqual(self.db.query(models.User).filter_by(name=u'bodhi').count(), 0)
+        # There are three comments, but none from the non-existing bodhi user.
+        self.assertEqual(self.db.query(models.Comment).count(), 3)
+        usernames = [
+            c.user.name
+            for c in self.db.query(models.Comment).order_by(models.Comment.timestamp).all()]
+        self.assertEqual(usernames, [u'guest', u'anonymous', u'hunter2'])
+
+    # Set the release's mandatory days in testing to 0 to set up the condition for this test.
+    @patch.dict(config, [('fedora.mandatory_days_in_testing', 0)])
+    def test_non_autokarma_update_without_mandatory_days_in_testing(self):
+        """
+        If the Update's release doesn't have a mandatory days in testing, main() should ignore it
+        (and should not comment on the update at all, even if it does reach karma levels.)
+        """
+        update = self.db.query(models.Update).all()[0]
+        update.autokarma = False
+        update.request = None
+        update.stable_karma = 1
+        update.status = models.UpdateStatus.testing
+        update.comment(self.db, 'testing', author='hunter2', anonymous=False, karma=1)
+        transaction.commit()
+
+        with patch(
+                'bodhi.server.scripts.approve_testing._get_db_session', return_value=self.db):
+            approve_testing.main(['nosetests', 'some_config.ini'])
+
+        # The bodhi user shouldn't exist, since it shouldn't have made any comments
+        self.assertEqual(self.db.query(models.User).filter_by(name=u'bodhi').count(), 0)
+        # There are three comments, but none from the non-existing bodhi user.
+        self.assertEqual(self.db.query(models.Comment).count(), 3)
+        usernames = [
+            c.user.name
+            for c in self.db.query(models.Comment).order_by(models.Comment.timestamp).all()]
+        self.assertEqual(usernames, [u'guest', u'anonymous', u'hunter2'])
diff --git a/bodhi/tests/server/test_metadata.py b/bodhi/tests/server/test_metadata.py
--- a/bodhi/tests/server/test_metadata.py
+++ b/bodhi/tests/server/test_metadata.py
@@ -49,7 +49,7 @@ def __init__(self, *args, **kw):
 
     def setUp(self):
         engine = create_engine(DB_PATH)
-        Session = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))
+        Session = scoped_session(sessionmaker(extension=ZopeTransactionExtension(keep_session=True)))
         Session.configure(bind=engine)
         log.debug('Creating all models for %s' % engine)
         Base.metadata.bind = engine