Repository: buildbot/buildbot

Issue:
Nightly scheduler trigger empty on first scheduled check
We gave a number of Nightly schedulers that builds specific name branches, when there are new commits since the last build.

A util.ChangeFilter(branch="foo") is used for this check.

The changes are reported from the git server to a changes.PBChangeSource object.

Earlier today we updated from 3.3.0 to 3.7.0, and when a group of Nightly schedulers checked for updates they all triggered, with source stamp

`[{"branch":null,"codebase":"","created_at":1620536400,"patch":null,"project":"","repository":"","revision":null,"ssid":28815}]`

rather than branch "foo", and a specific revision, probably because there had been no new commits reported to the PBChangeSource  port, which there wasn't because these branches are stable production branches, usually only updated a few times a month.

The result of the action was that all the triggered builds failed, especially since there was a bug in some code that it turns out does not tolerate an empty branch name.

My guess is that the ChangeFilter and PBChangeSource  completely ignored (or destroyed) previously pushed branch and commit info, as well as what had been scheduled earlier.

I suspect that the root cause is related to https://github.com/buildbot/buildbot/issues/5750 , about the change source not checking for updated commits on scheduled branches.

I have pushed new empty commits to a couple of branches to check the functionality, and once the commit had been registered the schedulers worked fine. 


Discussion:

Resolving patch:
diff --git a/master/buildbot/schedulers/timed.py b/master/buildbot/schedulers/timed.py
--- a/master/buildbot/schedulers/timed.py
+++ b/master/buildbot/schedulers/timed.py
@@ -28,6 +28,46 @@
 from buildbot.util import croniter
 from buildbot.util.codebase import AbsoluteSourceStampsMixin
 
+# States of objects which have to be observed are registered in the data base table `object_state`.
+# `objectid` in the `object_state` refers to the object from the `object` table.
+# Schedulers use the following state keys:
+# - `last_only_if_changed` - bool, setting of `onlyIfChanged` when the scheduler checked whether to
+# run build the last time. Does not exist if there was no build before.
+# - `last_build` - timestamp, time when the last build was scheduled to run. If `onlyIfChanged` is
+# set to True, only when there are designated changes build will be started. If the build was not
+# started,
+# `last_build` means on what time build was scheduled to run ignoring the fact if it actually ran or
+# not.
+# Value of these state keys affects the decision whether to run a build.
+
+# When deciding whether to run the build or to skip it, several factors and their interactions are
+# evaluated:
+# - the value of `onlyIfChanged` (default is False);
+# - has the state of `onlyIfChanged` changed;
+# - whether this would be first build (True if `last_build` value was not detected). If there
+# already were builds in the past, it indicates that the scheduler is existing;
+# - were there any important changes after the last build.
+
+# If `onlyIfChanged` is not set or its setting changes to False, builds will always run on the time
+# set, ignoring the status of `last_only_if_changed` and `last_build` regardless of what the state
+# is or anything else.
+
+# If `onlyIfChanged` is True, then builds will be run when there are relevant changes.
+
+# If `onlyIfChanged` is True and even when there were no relevant changes, builds will run for the
+# the first time on specified time as well when the following condition holds:
+# - `last_only_if_changed` was set to False on previous build. This ensures that any changes that
+# happened while `onlyIfChanged` was still False are not missed. This may result in a single build
+# done unnecessarily, but it is accepted as a good compromise because it only happens when
+# `onlyIfChanged` is adjusted;
+# - `last_build` does not have a value yet meaning that it is a new scheduler and we should have
+# initial build to set a baseline.
+
+# There is an edge case, when upgrading to v3.5.0 and new object status variable
+# `last_only_if_changed` is introduced. If scheduler exists and had builds before
+# (`last_build` has a value), build should only be started if there are relevant changes.
+# Thus upgrading the version does not start unnecessary builds.
+
 
 class Timed(AbsoluteSourceStampsMixin, base.BaseScheduler):
 
@@ -73,6 +113,7 @@ def __init__(self, name, builderNames, reason='',
         # If True, only important changes will be added to the buildset.
         self.onlyImportant = onlyImportant
         self._reactor = reactor  # patched by tests
+        self.is_first_build = None
 
     @defer.inlineCallbacks
     def activate(self):
@@ -87,6 +128,10 @@ def activate(self):
 
         # get the scheduler's last_build time (note: only done at startup)
         self.lastActuated = yield self.getState('last_build', None)
+        if self.lastActuated is None:
+            self.is_first_build = True
+        else:
+            self.is_first_build = False
 
         # schedule the next build
         yield self.scheduleNextBuild()
@@ -152,10 +197,18 @@ def startBuild(self):
         # occurred since the last invocation. Note that when the scheduler has just been started
         # there may not be any important changes yet and we should start the build for the
         # current state of the code whatever it is.
-        last_only_if_changed = yield self.getState('last_only_if_changed', False)
-        if last_only_if_changed and self.onlyIfChanged and not any(classifications.values()):
+        #
+        # Note that last_only_if_changed will always be set to the value of onlyIfChanged
+        # at the point when startBuild finishes (it is not obvious, that all code paths lead
+        # to this outcome)
+
+        last_only_if_changed = yield self.getState('last_only_if_changed', True)
+
+        if last_only_if_changed and self.onlyIfChanged and not any(classifications.values()) and \
+                not self.is_first_build:
             log.msg(("{} scheduler <{}>: skipping build " +
                      "- No important changes").format(self.__class__.__name__, self.name))
+            self.is_first_build = False
             return
 
         if last_only_if_changed != self.onlyIfChanged:
@@ -176,6 +229,7 @@ def startBuild(self):
             yield self.addBuildsetForSourceStampsWithDefaults(
                 reason=self.reason,
                 sourcestamps=sourcestamps)
+        self.is_first_build = False
 
     def getCodebaseDict(self, codebase):
         if self.createAbsoluteSourceStamps:


Test patch:
diff --git a/master/buildbot/test/fakedb/state.py b/master/buildbot/test/fakedb/state.py
--- a/master/buildbot/test/fakedb/state.py
+++ b/master/buildbot/test/fakedb/state.py
@@ -94,15 +94,17 @@ def atomicCreateState(self, objectid, name, thd_create_callback):
 
     # fake methods
 
-    def set_fake_state(self, object, **kwargs):
+    def set_fake_state(self, object, name, value):
         state_key = (object.name, object.__class__.__name__)
         if state_key in self.objects:
             id = self.objects[state_key]
         else:
             id = self.objects[state_key] = self._newId()
 
-        self.states[id] = dict((k, json.dumps(v))
-                               for k, v in kwargs.items())
+        if id in self.states:
+            self.states[id][name] = json.dumps(value)
+        else:
+            self.states[id] = {name: json.dumps(value)}
         return id
 
     # assertions
diff --git a/master/buildbot/test/unit/changes/test_gitpoller.py b/master/buildbot/test/unit/changes/test_gitpoller.py
--- a/master/buildbot/test/unit/changes/test_gitpoller.py
+++ b/master/buildbot/test/unit/changes/test_gitpoller.py
@@ -1493,8 +1493,7 @@ def test_startService_loadLastRev(self):
         yield self.poller.stopService()
 
         self.master.db.state.set_fake_state(
-            self.poller,
-            lastRev={"master": "fa3ae8ed68e664d4db24798611b352e3c6509930"},
+            self.poller, 'lastRev', {"master": "fa3ae8ed68e664d4db24798611b352e3c6509930"}
         )
 
         yield self.poller.startService()
diff --git a/master/buildbot/test/unit/schedulers/test_timed_Nightly.py b/master/buildbot/test/unit/schedulers/test_timed_Nightly.py
--- a/master/buildbot/test/unit/schedulers/test_timed_Nightly.py
+++ b/master/buildbot/test/unit/schedulers/test_timed_Nightly.py
@@ -246,14 +246,18 @@ def test_iterations_simple_with_branch(self):
         d = sched.deactivate()
         return d
 
-    def do_test_iterations_onlyIfChanged(self, *changes_at, last_only_if_changed=None, **kwargs):
+    def do_test_iterations_onlyIfChanged(self, *changes_at, last_only_if_changed,
+                                         is_new_scheduler=False, **kwargs):
         fII = mock.Mock(name='fII')
         self.makeScheduler(name='test', builderNames=['test'], branch=None,
                            minute=[5, 25, 45], onlyIfChanged=True,
                            fileIsImportant=fII, **kwargs)
 
+        if not is_new_scheduler:
+            self.db.state.set_fake_state(self.sched, 'last_build', -1)
+
         if last_only_if_changed is not None:
-            self.db.state.set_fake_state(self.sched, last_only_if_changed=last_only_if_changed)
+            self.db.state.set_fake_state(self.sched, 'last_only_if_changed', last_only_if_changed)
 
         return self.do_test_iterations_onlyIfChanged_test(fII, *changes_at)
 
@@ -282,7 +286,8 @@ def do_test_iterations_onlyIfChanged_test(self, fII, *changes_at):
 
     @defer.inlineCallbacks
     def test_iterations_onlyIfChanged_no_changes_new_scheduler(self):
-        yield self.do_test_iterations_onlyIfChanged()
+        yield self.do_test_iterations_onlyIfChanged(last_only_if_changed=None,
+                                                    is_new_scheduler=True)
         self.assertEqual(self.addBuildsetCalls, [
             ('addBuildsetForSourceStampsWithDefaults', {
                 'builderNames': None,
@@ -306,6 +311,14 @@ def test_iterations_onlyIfChanged_no_changes_existing_scheduler(self):
 
     @defer.inlineCallbacks
     def test_iterations_onlyIfChanged_no_changes_existing_scheduler_setting_changed(self):
+        # When onlyIfChanged==False, builds are run every time on the time set
+        # (changes or no changes). Changes are being recognized but do not have any effect on
+        # starting builds.
+        # It might happen that onlyIfChanged was False, then change happened, then setting was
+        # changed to onlyIfChanged==True.
+        # Because onlyIfChanged was False possibly important change will be missed.
+        # Therefore the first build should start immediately.
+
         yield self.do_test_iterations_onlyIfChanged(last_only_if_changed=False)
         self.assertEqual(self.addBuildsetCalls, [
             ('addBuildsetForSourceStampsWithDefaults', {
@@ -320,11 +333,24 @@ def test_iterations_onlyIfChanged_no_changes_existing_scheduler_setting_changed(
                                          last_build=1500 + self.localtime_offset)
         yield self.sched.deactivate()
 
+    @defer.inlineCallbacks
+    def test_iterations_onlyIfChanged_no_changes_existing_scheduler_update_to_v3_5_0(self):
+        # v3.4.0 have not had a variable last_only_if_changed yet therefore this case is tested
+        # separately
+        yield self.do_test_iterations_onlyIfChanged(last_only_if_changed=None)
+        self.assertEqual(self.addBuildsetCallTimes, [])
+        self.assertEqual(self.addBuildsetCalls, [])
+        self.db.state.assertStateByClass('test', 'Nightly',
+                                         last_build=1500 + self.localtime_offset)
+        yield self.sched.deactivate()
+
     @defer.inlineCallbacks
     def test_iterations_onlyIfChanged_unimp_changes_calls_for_new_scheduler(self):
         yield self.do_test_iterations_onlyIfChanged(
             (60, mock.Mock(), False),
-            (600, mock.Mock(), False))
+            (600, mock.Mock(), False),
+            last_only_if_changed=None,
+            is_new_scheduler=True)
         self.assertEqual(self.addBuildsetCalls, [
             ('addBuildsetForSourceStampsWithDefaults', {
                 'builderNames': None,
@@ -340,7 +366,7 @@ def test_iterations_onlyIfChanged_unimp_changes_calls_for_new_scheduler(self):
         yield self.sched.deactivate()
 
     @defer.inlineCallbacks
-    def test_iterations_onlyIfChanged_unimp_changes_existing_scheduler(self):
+    def test_iterations_onlyIfChanged_unimp_changes_existing_sched_changed_only_if_changed(self):
         yield self.do_test_iterations_onlyIfChanged(
             (60, mock.Mock(), False),
             (600, mock.Mock(), False),
@@ -360,7 +386,7 @@ def test_iterations_onlyIfChanged_unimp_changes_existing_scheduler(self):
         yield self.sched.deactivate()
 
     @defer.inlineCallbacks
-    def test_iterations_onlyIfChanged_unimp_changes_existing_scheduler_setting_changed(self):
+    def test_iterations_onlyIfChanged_unimp_changes_existing_sched_same_only_if_changed(self):
         yield self.do_test_iterations_onlyIfChanged(
             (60, mock.Mock(), False),
             (600, mock.Mock(), False),
@@ -371,6 +397,31 @@ def test_iterations_onlyIfChanged_unimp_changes_existing_scheduler_setting_chang
                                          last_build=1500 + self.localtime_offset)
         yield self.sched.deactivate()
 
+    @defer.inlineCallbacks
+    def test_iterations_onlyIfChanged_changes_existing_scheduler_update_to_v3_5_0(self):
+        # v3.4.0 have not had a variable last_only_if_changed yet therefore this case is tested
+        # separately
+        yield self.do_test_iterations_onlyIfChanged(
+            (120, self.makeFakeChange(number=1, branch=None), False),
+            (1200, self.makeFakeChange(number=2, branch=None), True),
+            (1201, self.makeFakeChange(number=3, branch=None), False),
+            last_only_if_changed=None,
+            )
+
+        self.assertEqual(self.addBuildsetCallTimes, [1500])
+        self.assertEqual(self.addBuildsetCalls, [(
+            'addBuildsetForChanges', {
+                'waited_for': False,
+                'reason': "The Nightly scheduler named 'test' triggered this build",
+                'external_idstring': None,
+                'changeids': [1, 2, 3],
+                'properties': None, 'builderNames': None
+                }),
+            ])
+        self.db.state.assertStateByClass('test', 'Nightly',
+                                         last_build=1500 + self.localtime_offset)
+        yield self.sched.deactivate()
+
     @defer.inlineCallbacks
     def test_iterations_onlyIfChanged_off_branch_changes(self):
         yield self.do_test_iterations_onlyIfChanged(
@@ -448,7 +499,8 @@ def test_iterations_onlyIfChanged_createAbsoluteSourceStamps_oneChanged_loadOthe
                                       'b': {'repository': "", 'branch': 'master'}},
                            createAbsoluteSourceStamps=True)
 
-        self.db.state.set_fake_state(self.sched, last_only_if_changed=True, lastCodebases={
+        self.db.state.set_fake_state(self.sched, 'last_only_if_changed', True)
+        self.db.state.set_fake_state(self.sched, 'lastCodebases', {
             'b': {
                 'branch': 'master',
                 'repository': 'B',
@@ -488,6 +540,7 @@ def test_iterations_onlyIfChanged_createAbsoluteSourceStamps_bothChanged(self):
                 number=4, codebase='b', revision='1234:abc'), True),
             codebases={'a': {'repository': "", 'branch': 'master'},
                        'b': {'repository': "", 'branch': 'master'}},
+            last_only_if_changed=None,
             createAbsoluteSourceStamps=True)
         self.db.state.assertStateByClass('test', 'Nightly',
                                          last_build=1500 + self.localtime_offset)
diff --git a/master/buildbot/test/unit/util/test_codebase.py b/master/buildbot/test/unit/util/test_codebase.py
--- a/master/buildbot/test/unit/util/test_codebase.py
+++ b/master/buildbot/test/unit/util/test_codebase.py
@@ -62,7 +62,7 @@ def test_getCodebaseDict_not_found(self):
 
     @defer.inlineCallbacks
     def test_getCodebaseDict_existing(self):
-        self.db.state.set_fake_state(self.object, lastCodebases={'a': {
+        self.db.state.set_fake_state(self.object, 'lastCodebases', {'a': {
             'repository': 'A',
             'revision': '1234:abc',
             'branch': 'master',
@@ -83,7 +83,7 @@ def test_recordChange(self):
 
     @defer.inlineCallbacks
     def test_recordChange_older(self):
-        self.db.state.set_fake_state(self.object, lastCodebases={'a': {
+        self.db.state.set_fake_state(self.object, 'lastCodebases', {'a': {
             'repository': 'A',
             'revision': '2345:bcd',
             'branch': 'master',
@@ -97,7 +97,7 @@ def test_recordChange_older(self):
 
     @defer.inlineCallbacks
     def test_recordChange_newer(self):
-        self.db.state.set_fake_state(self.object, lastCodebases={'a': {
+        self.db.state.set_fake_state(self.object, 'lastCodebases', {'a': {
             'repository': 'A',
             'revision': '1234:abc',
             'branch': 'master',
diff --git a/master/buildbot/test/unit/util/test_state.py b/master/buildbot/test/unit/util/test_state.py
--- a/master/buildbot/test/unit/util/test_state.py
+++ b/master/buildbot/test/unit/util/test_state.py
@@ -39,7 +39,7 @@ def setUp(self):
 
     @defer.inlineCallbacks
     def test_getState(self):
-        self.master.db.state.set_fake_state(self.object, fav_color=['red', 'purple'])
+        self.master.db.state.set_fake_state(self.object, 'fav_color', ['red', 'purple'])
         res = yield self.object.getState('fav_color')
 
         self.assertEqual(res, ['red', 'purple'])
@@ -51,7 +51,7 @@ def test_getState_default(self):
         self.assertEqual(res, 'black')
 
     def test_getState_KeyError(self):
-        self.master.db.state.set_fake_state(self.object, fav_color=['red', 'purple'])
+        self.master.db.state.set_fake_state(self.object, 'fav_color', ['red', 'purple'])
         d = self.object.getState('fav_book')
 
         def cb(_):
@@ -72,7 +72,7 @@ def test_setState(self):
 
     @defer.inlineCallbacks
     def test_setState_existing(self):
-        self.master.db.state.set_fake_state(self.object, x=13)
+        self.master.db.state.set_fake_state(self.object, 'x', 13)
         yield self.object.setState('x', 14)
 
         self.master.db.state.assertStateByClass('fake-name', 'FakeObject',
diff --git a/master/buildbot/test/util/scheduler.py b/master/buildbot/test/util/scheduler.py
--- a/master/buildbot/test/util/scheduler.py
+++ b/master/buildbot/test/util/scheduler.py
@@ -87,15 +87,25 @@ def attachScheduler(self, scheduler, objectid, schedulerid,
         db.insert_test_data(rows)
 
         if overrideBuildsetMethods:
-            for method in (
-                    'addBuildsetForSourceStampsWithDefaults',
-                    'addBuildsetForChanges',
-                    'addBuildsetForSourceStamps'):
-                actual = getattr(scheduler, method)
-                fake = getattr(self, f'fake_{method}')
-
-                self.assertArgSpecMatches(actual, fake)
-                setattr(scheduler, method, fake)
+            self.assertArgSpecMatches(
+                scheduler.addBuildsetForSourceStampsWithDefaults,
+                self.fake_addBuildsetForSourceStampsWithDefaults
+            )
+            scheduler.addBuildsetForSourceStampsWithDefaults = \
+                self.fake_addBuildsetForSourceStampsWithDefaults
+
+            self.assertArgSpecMatches(
+                scheduler.addBuildsetForChanges,
+                self.fake_addBuildsetForChanges
+            )
+            scheduler.addBuildsetForChanges = self.fake_addBuildsetForChanges
+
+            self.assertArgSpecMatches(
+                scheduler.addBuildsetForSourceStamps,
+                self.fake_addBuildsetForSourceStamps
+            )
+            scheduler.addBuildsetForSourceStamps = self.fake_addBuildsetForSourceStamps
+
             self.addBuildsetCalls = []
             self._bsidGenerator = iter(range(500, 999))
             self._bridGenerator = iter(range(100, 999))