Repository: buildbot/buildbot

Issue:
HgPoller crashes in mercurial repo with stripped commits
I'm witnessing issues with **mercurial**'s poller after updating my bots from 2.0.1 to 2.3.0. The issues have been introduced starting at version 2.2.0, see the entry _Buildbot can now monitor multiple branches in a Mercurial repository._ in the [changelog](http://docs.buildbot.net/current/relnotes/index.html) and is related to [this streak of commits](https://github.com/buildbot/buildbot/commits?author=slightlyunconventional) by @slightlyunconventional 

Here are the issues I've witnessed:

* BUG1: Without changing my bot's code, I'm witnessing that only a **single poller is alive** whereas I'm creating multiple instances of HgPoller (for multiple branches of the same repo). This is visible in the bot's log, because `buildbot/changes/hgpoller.py` logs every time it polls a repo (see "hgpoller: polling hg repo" string). With version 2.3.0 only a single branch is active. On a production bot I was witnessing that solely branch "default" (whereas pollers were created for branches "default" and "DtsXtextEditor") was being polled. When creating a minimal example, I was witnessing that a non-"default" branch was the working one. I tried understanding which branch was being chosen to no avail. Note that I however witness:

     ```
     buildbotNetUsageData: sending  { ...  'buildbot/changes/hgpoller/HgPoller': 2 }
     ```
    
     when creating two pollers for two branches. So the pollers are sent but are somehow lost afterwards.

* BUG2: By changing my bot's code, to use a single HgPoller for all branches (i.e. using the new feature introduced in 2.2.0), I'm witnessing that **only one branch is being pulled** (I've tried with at most two branches) successfully. The other branch fails as follows (every time the poller runs, not only on an incoming commit):
    
    ```[-] hgpoller: repo poll failed
    Unhandled Error
    File ".../sandbox/lib/python3.6/site-packages/buildbot/changes/hgpoller.py", line 277, in `_processBranchChanges`
    del revNodeList[0]
    builtins.IndexError: list assignment index out of range
   ````

  I've witnessed this behavior on two production bots. I could not reproduce it with a minimal example with fresh mercurial repositories. I believe it has to do with the database's state. This is a **blocking bug** as it forbids the bot to consume incoming commits.
    
For the record, I've also witnessing the following issues when **stripping the mercurial repo**:

```[-] hgpoller: repo poll failed
[-] Unhandled Error
	Traceback (most recent call last):
	--- <exception caught here> ---
	  File "/media/truecrypt1/merlin/korhal/sandbox/lib/python3.6/site-packages/buildbot/changes/hgpoller.py", line 255, in _processChanges
	    yield self._processBranchChanges(rev, branch)
	  File "/media/truecrypt1/merlin/korhal/sandbox/lib/python3.6/site-packages/buildbot/changes/hgpoller.py", line 272, in _processBranchChanges
	    path=self._absWorkdir(), env=os.environ, errortoo=False)
	twisted.internet.utils._UnexpectedErrorOutput: got stderr: b"abort: unknown revision '11'!\n"
```
	
when revision `11` got stripped, and afterwards the following traces (at every polling):

```
[-] Unhandled error in Deferred:
[-] Unhandled Error
	Traceback (most recent call last):
	Failure: twisted.internet.error.ProcessTerminated: A process has ended with a probable error condition: process ended with exit code 255.
```
The fact that this trace was appearing after every polling seem to disagree with `_processChangesFailure` comment:

```
        # eat the failure to continue along the deferred chain - we still want
        # to catch up
```

However these last two traces seem less of an issue that my first two bullets, because if new changesets are created (making the stripped revision number "come back"), polling will work again (with the caveat of my second bullet). It could however be an issue if you strip a large number of commits. I can't say if this behavior was already in 2.0.1.

I'm willing to test patches, please send them my way. For the moment I've stayed on version 2.1.0.


Discussion:
Well that's no good at all.
1. Can you provide that minimal config for the "multiple instances of HgPoller" case?
2. Just before the "del revNodeList[0]" error, do you see a log of the hg command that was executed? It should look something like this: 'hg log -r FOO::BAR --template={rev}:{node}\n'. What are FOO and BAR? When you run that command by hand on the poller's repo, what are the results?
(Obviously, a workaround for this would be to clear your database, but clearly it shouldn't have happened in the first place and it would be great to save a copy of the failing database for debugging.)

As for stripping, I'm not sure if you're talking about stripping the upstream repo (the one buildbot is monitoring), or buildbot's local repo, but (for different reasons) I wouldn't recommend either... :-)
1. We're using an XML layer over buildbot (the build is described in XML and buildbot is instantiated with that) so I can't give you code directly. However, the poller is built like this:

        hgpoller.HgPoller(workdir="/home/john/dest", repourl="/home/john/src",
                          project=name, branch=branch, pollInterval=30, pollAtLaunch=True)

    and the scheduler is an instance of AnyBranchScheduler built like this:

         AnyBranchScheduler(name="Mercurial AnyBranchScheduler", builderNames=bnames, change_filter=util.ChangeFilter(filter_fn=filter_fun),
                            treeStableTimer=30, codebases=CODEBASES)
2. There's no log by default. Here's what I've got manually: `print(" ".join(revListArgs))` yields:

        log -r 213::231 --template={rev}:{node}\n

    213 and 231 belongs to different branches. When executing this command in both repositories, the command succeeds and prints nothing.

Regarding stripping, I stripped both repos. I agree it's not recommended (actually I did it when debugging this issue) but I believe it should not prevent the bot from going on (eating commits until reaching an acceptable state). On a few occasions repos watched by production bots got a commit or two stripped ^^
@smelc: Thanks for the bug report. Indeed this is an important edge case that we would like to address.
@p12tic IMHO there are two bugs reported here, but neither of them is related to strip. :-)

@smelc:

1. I will try to reproduce this problem.

2. I can't figure out how this could happen -- specifically the part where the two commits are on different branches.
As far as I can see (and intended), when running 2.2.0 or later, the legacy database fields (named something like "HgPoller.current_rev") are simply ignored, and new ones ("HgPoller.lastRev") are created. So the bit about "_I could not reproduce it with a minimal example with fresh mercurial repositories. I believe it has to do with the database's state._" is extra confusing. I don't know how to debug this without a testcase.
@p12tic : indeed the bugs are not related to strip

@slightlyunconventional : I see, then it means I must be able to reproduce the issue with fresh repositories. I will try again and report here.
For bug 1: I think the key line in the log is this:
`2019-07-09 16:29:20-0700 [-] adding 1 new change_sources, removing 0`

Providing a unique name for each HgPoller changes this to "2 new change_sources", and everything seems happy -- I see multiple `hgpoller: polling hg repo` messages repeating in the log. (Without an explicit `name` parameter, HgPoller uses `repourl` as the name.)

(FWIW, I don't think I introduced this change; maybe it happened somewhere in the c["change_source"] code, wherever that lies...?)
Here's the config file I was testing with. You'd still need to connect a worker and supply your own branched Hg repo, of course.

[buildbot-hgpoller-test.txt](https://github.com/buildbot/buildbot/files/3375153/buildbot-hgpoller-test.txt)
As for reproducing bug 2, it will likely be a case of adding debug prints in the `_processChanges()` path. I'd like to know the branch and return values around `self._getHead(branch)` and `self._getCurrentRev(branch)` for the first few 2.3.0 polls on your 2.0.1 database.

Maybe there's some subtle issue like unicode vs str mismatch? I dunno, I'm still struggling to imagine a way that revision numbers could be associated with the wrong branches in `lastRev` in the `object_state` database table.
Concerning bug 1, I confirm @slightlyunconventional's workaround above of making HgPoller's `name` argument unique (I used `repourl+"_"+branch`). It allows again to have multiple HgPoller instances for a given `repourl`. So this either must become the default value in `HgPoller`'s constructor (and this uniqueness requirement must be documented!) or the cause of this change must be identified (@tardyp> any clue?)
> Concerning bug 1, I confirm @slightlyunconventional's workaround above of making HgPoller's `name` argument unique (I used `repourl+"_"+branch`). It allows again to have multiple HgPoller instances for a given `repourl`. So this either must become the default value in `HgPoller`'s constructor (and this uniqueness requirement must be documented!) or the cause of this change must be identified (@tardyp> any clue?)

We could easily change the default `name` to be repo+branch inside hgpoller.py, to avoid user config file changes. From what I could see, this would be needed in gitpoller.py too. I didn't audit others.
I cannot reproduce bug 2 on a minimal example without stripping so **my initial comment must be wrong**, I must have stripped at some point in my reproduction procedure. The `builtins.IndexError: list assignment index out of range` happens when, by stripping you assign a changeset identifier to a different branch than its original branch: commit on branch "default", let bot process the commit; strip this commit, update on another branch, commit again. This is a regression since versions < 2.1.0 but it's not that bad I suppose, although it will block bots from processing new commits.

Resolving patch:
diff --git a/master/buildbot/changes/hgpoller.py b/master/buildbot/changes/hgpoller.py
--- a/master/buildbot/changes/hgpoller.py
+++ b/master/buildbot/changes/hgpoller.py
@@ -50,9 +50,6 @@ def __init__(self, repourl, branch=None, branches=None, bookmarks=None,
         if pollinterval != -2:
             pollInterval = pollinterval
 
-        if name is None:
-            name = repourl
-
         self.repourl = repourl
 
         if branch and branches:
@@ -64,6 +61,13 @@ def __init__(self, repourl, branch=None, branches=None, bookmarks=None,
 
         self.bookmarks = bookmarks or []
 
+        if name is None:
+            name = repourl
+            if self.bookmarks:
+                name += "_" + "_".join(self.bookmarks)
+            if self.branches:
+                name += "_" + "_".join(self.branches)
+
         if not self.branches and not self.bookmarks:
             self.branches = ['default']
 
@@ -260,6 +264,16 @@ def _processChanges(self, unused_output):
                 continue
             yield self._processBranchChanges(rev, branch)
 
+    @defer.inlineCallbacks
+    def _getRevNodeList(self, revset):
+        revListArgs = ['log', '-r', revset, r'--template={rev}:{node}\n']
+        results = yield utils.getProcessOutput(self.hgbin, revListArgs,
+                                               path=self._absWorkdir(), env=os.environ, errortoo=False)
+        results = results.decode(self.encoding)
+
+        revNodeList = [rn.split(':', 1) for rn in results.strip().split()]
+        defer.returnValue(revNodeList)
+
     @defer.inlineCallbacks
     def _processBranchChanges(self, new_rev, branch):
         prev_rev = yield self._getCurrentRev(branch)
@@ -272,15 +286,15 @@ def _processBranchChanges(self, new_rev, branch):
             return
 
         # two passes for hg log makes parsing simpler (comments is multi-lines)
-        revset = '{}::{}'.format(prev_rev, new_rev)
-        revListArgs = ['log', '-r', revset, r'--template={rev}:{node}\n']
-        results = yield utils.getProcessOutput(self.hgbin, revListArgs,
-                                               path=self._absWorkdir(), env=os.environ, errortoo=False)
-        results = results.decode(self.encoding)
+        revNodeList = yield self._getRevNodeList('{}::{}'.format(prev_rev, new_rev))
 
-        revNodeList = [rn.split(':', 1) for rn in results.strip().split()]
         # revsets are inclusive. Strip the already-known "current" changeset.
-        del revNodeList[0]
+        if not revNodeList:
+            # empty revNodeList probably means the branch has changed head (strip of force push?)
+            # in that case, we should still produce a change for that new rev (but we can't know how many parents were pushed)
+            revNodeList = yield self._getRevNodeList(new_rev)
+        else:
+            del revNodeList[0]
 
         log.msg('hgpoller: processing %d changes in branch %r: %r in %r'
                 % (len(revNodeList), branch, revNodeList, self._absWorkdir()))


Test patch:
diff --git a/master/buildbot/test/unit/test_changes_hgpoller.py b/master/buildbot/test/unit/test_changes_hgpoller.py
--- a/master/buildbot/test/unit/test_changes_hgpoller.py
+++ b/master/buildbot/test/unit/test_changes_hgpoller.py
@@ -244,6 +244,11 @@ def test_name(self):
             self.remote_repo, name="MyName", workdir='/some/dir')
         self.assertEqual("MyName", other.name)
 
+        # and one with explicit branches...
+        other = hgpoller.HgPoller(
+            self.remote_repo, branches=["b1", "b2"], workdir='/some/dir')
+        self.assertEqual(self.remote_repo + "_b1_b2", other.name)
+
     def test_hgbin_default(self):
         self.assertEqual(self.poller.hgbin, "hg")
 
@@ -332,6 +337,48 @@ def test_poll_regular(self):
         self.assertEqual(change['revision'], '784bd')
         self.assertEqual(change['comments'], 'Comment for rev 5')
 
+    @defer.inlineCallbacks
+    def test_poll_force_push(self):
+        #  There's a previous revision, but not linked with new rev
+        self.expectCommands(
+            gpo.Expect('hg', 'pull', '-b', 'default',
+                       'ssh://example.com/foo/baz')
+            .path('/some/dir'),
+            gpo.Expect(
+                'hg', 'heads', '-r', 'default', '--template={rev}' + os.linesep)
+            .path('/some/dir').stdout(b'5' + LINESEP_BYTES),
+            gpo.Expect('hg', 'log', '-r', '4::5',
+                       '--template={rev}:{node}\\n')
+            .path('/some/dir').stdout(b""),
+            gpo.Expect('hg', 'log', '-r', '5',
+                       '--template={rev}:{node}\\n')
+            .path('/some/dir').stdout(LINESEP_BYTES.join([
+                        b'5:784bd',
+                    ])),
+            gpo.Expect('hg', 'log', '-r', '784bd',
+                       '--template={date|hgdate}' + os.linesep +
+                       '{author}' + os.linesep +
+                       "{files % '{file}" +
+                       os.pathsep + "'}" +
+                       os.linesep + '{desc|strip}')
+            .path('/some/dir').stdout(LINESEP_BYTES.join([
+                b'1273258009.0 -7200',
+                b'Joe Test <joetest@example.org>',
+                b'file1 file2',
+                b'Comment for rev 5',
+                b''])),
+        )
+
+        yield self.poller._setCurrentRev(4)
+
+        yield self.poller.poll()
+        yield self.check_current_rev(5)
+
+        self.assertEqual(len(self.master.data.updates.changesAdded), 1)
+        change = self.master.data.updates.changesAdded[0]
+        self.assertEqual(change['revision'], '784bd')
+        self.assertEqual(change['comments'], 'Comment for rev 5')
+
 
 class HgPollerNoTimestamp(TestHgPoller):
     """ Test HgPoller() without parsing revision commit timestamp """