Repository: buildbot/buildbot

Issue:
Git poller state exceeds database field size after some time
When the Git poller covers all branches, it can happen that eventually the `lastRev` state gets too large so that it cannot be stored in the DB anymore (resulting in `Data too long for column 'value_json' at row 1` exceptions). This is because it never forgets branches that used to exist in the past (https://github.com/buildbot/buildbot/blob/master/master/buildbot/changes/gitpoller.py#L239), so its `lastRev` state never shrinks.

I'm wondering whether the `lastRev` state really needs to be updated (and not replaced) by the current set of references. The only reason why update is used (instead of replacing) I can think of is that if a branch is deleted and then restored (with the same HEAD), it will not trigger a change.

I'd like to create a PR which removes the updating, or at least makes it configurable. What is the preferred solution here?


Discussion:
@felixfontein Thanks for the bug report. I think replacing makes sense here. It can be argued that removing a branch will remove all state related to it in the repository and subsequent push is unrelated even if the commit SHA is the same.

Resolving patch:
diff --git a/master/buildbot/changes/gitpoller.py b/master/buildbot/changes/gitpoller.py
--- a/master/buildbot/changes/gitpoller.py
+++ b/master/buildbot/changes/gitpoller.py
@@ -236,7 +236,7 @@ def poll(self):
                 log.err(_why="trying to poll branch {} of {}".format(
                         branch, self.repourl))
 
-        self.lastRev.update(revs)
+        self.lastRev = revs
         yield self.setState('lastRev', self.lastRev)
 
     def _get_commit_comments(self, rev):


Test patch:
diff --git a/master/buildbot/test/unit/changes/test_gitpoller.py b/master/buildbot/test/unit/changes/test_gitpoller.py
--- a/master/buildbot/test/unit/changes/test_gitpoller.py
+++ b/master/buildbot/test/unit/changes/test_gitpoller.py
@@ -636,7 +636,6 @@ def test_poll_multipleBranches_buildPushesWithNoCommits_default(self):
 
         self.assert_all_commands_ran()
         self.assertEqual(self.poller.lastRev, {
-            'master': '4423cdbcbb89c14e50dd5f4152415afd686c5241',
             'release': '4423cdbcbb89c14e50dd5f4152415afd686c5241'
         })
         self.assertEqual(len(self.master.data.updates.changesAdded), 0)
@@ -702,7 +701,6 @@ def comments(rev):
 
         self.assert_all_commands_ran()
         self.assertEqual(self.poller.lastRev, {
-            'master': '4423cdbcbb89c14e50dd5f4152415afd686c5241',
             'release': '4423cdbcbb89c14e50dd5f4152415afd686c5241'
         })
         self.assertEqual(self.master.data.updates.changesAdded, [
@@ -785,7 +783,6 @@ def comments(rev):
 
         self.assert_all_commands_ran()
         self.assertEqual(self.poller.lastRev, {
-            'master': '4423cdbcbb89c14e50dd5f4152415afd686c5241',
             'release': '4423cdbcbb89c14e50dd5f4152415afd686c5241'
         })
         self.assertEqual(self.master.data.updates.changesAdded, [
@@ -866,7 +863,6 @@ def comments(rev):
 
         self.assert_all_commands_ran()
         self.assertEqual(self.poller.lastRev, {
-            'master': '0ba9d553b7217ab4bbad89ad56dc0332c7d57a8c',
             'release': '4423cdbcbb89c14e50dd5f4152415afd686c5241'
         })
         self.assertEqual(self.master.data.updates.changesAdded, [
@@ -1182,10 +1178,7 @@ def __call__(self, branch):
         # The release branch id should remain unchanged,
         # because it was ignored.
         self.assertEqual(self.poller.lastRev, {
-            'refs/heads/master':
-            '4423cdbcbb89c14e50dd5f4152415afd686c5241',
-            'refs/heads/release':
-            'bf0b01df6d00ae8d1ffa0b2e2acbe642a6cd35d5'
+            'refs/heads/master': '4423cdbcbb89c14e50dd5f4152415afd686c5241'
         })
 
         added = self.master.data.updates.changesAdded
@@ -1276,7 +1269,6 @@ def pullFilter(branch):
 
         self.assert_all_commands_ran()
         self.assertEqual(self.poller.lastRev, {
-            'master': 'fa3ae8ed68e664d4db24798611b352e3c6509930',
             'refs/pull/410/head': '9118f4ab71963d23d02d4bdc54876ac8bf05acf2'
         })