Repository: buildbot/buildbot

Issue:
GitHub source step fails on missing magic `/merge` ref of GitHub
Currently I run into an issue where the changehook is too fast for the magic `merge` ref in GitHub to be created on Pull Requests.

An easy fix would be to try again in this case or wait a reasonable amount in `startVC()` of the GitHub step if the remote ref ends in `merge`.

Thoughts?



Discussion:
I though this was created on demand..
There is a retry option on the git step, maybe you can try to use/improve that.
The merge ref will be created for every Pull request, though the webhook is just too fast. Sometimes the merge isn't possible and that's a legit case to fail. But otherwise we should retry at least once. I take a look at the retry option, thanks!
I just modify source code. Use `/head` instead of `/merge`
@meritozh This is fine if you want to use `/head` There is some value in using the `/merge` ref though. I think I tried setting the retry option and still get sporadic failures, need to confirm this though.
I think use `/merge` is not suitable. I have face same issue. My repo is so large, so when develop make a PR, it always fails. If you use retry, you must set a `max_try_number`, but for PR which contains big changes (such as, a rebase PR, or updating multiple dependencies), you must set this value bigger. And `max_try_number` must only useful when bb can not found a remote branch, otherwise, bb will retry may time if a PR actually have some error. It's terrible for me.
`/merge` vs. `/head` is an issue for the webhook/poller though. If the retry is not working I might implement a configurable backoff timer for the GitHub source step.
Here is a message we got on the mailing list on similar issue, with reference to support from GH people saying that /merge magic ref is not documented:
```
From: David Alves

 I've seen sporadic build issues when pushing a new rev of a 
  GitHub pull request
  because buildbot pulls the merge ref and the merge ref is often 
  not immediately updated.
  This causes the build triggered by the updated rev to actually 
  use the merge ref
  corresponding to the previous rev, making the results of the 
  build unreliable without
  re-checking that the correct code was actually tested.
  When contacted about this Github claimed merge refs are an 
  undocumented feature and
  that they don't advise to use them.
  The alternative seems to be building from the head ref instead 
  of the merge ref.
  IMO this makes sense. Would you be open to patch to fix this?
```

We have always used the merge magic ref to build buildbot, and I don't recall having any issue of building the wrong code. However, we often have people sending patch based on an old version of master, so I really would like not to remove the feature completley.

I didn't know the feature was undocumented. I found it while studying how travis worked:
https://docs.travis-ci.com/user/pull-requests/

> Travis CI builds a pull request when it is first opened, and whenever commits are added to the pull request. Rather than build the commits that have been pushed to the branch the pull request is from, we build the merge between the source branch and the upstream branch.


They don't say how they actually merge the code.you can see that from the log:

```
6.54s$ git clone --depth=300 https://github.com/buildbot/buildbot.git buildbot/buildbot
Cloning into 'buildbot/buildbot'...
remote: Counting objects: 63986, done.
remote: Compressing objects: 100% (17548/17548), done.
remote: Total 63986 (delta 50483), reused 58925 (delta 46144), pack-reused 0
Receiving objects: 100% (63986/63986), 25.70 MiB | 15.83 MiB/s, done.
Resolving deltas: 100% (50483/50483), done.
Checking connectivity... done.
$ cd buildbot/buildbot
0.89s$ git fetch origin +refs/pull/4052/merge:
```

I am not against having it disabled via configuration flag.
One could easily change the GitHub step, with a MergeToTargetBranch=False.

A first step would be to just s/\/merge$/\/head/ on the branch name is this config is false.
A second step would be to perform the merge to target branch manually.
Such a feature would have to be done generically, as other github clones (Gitlab, bitbucket, gogs, etc), which have the PR feature will clearly benefit from this MergeToTargetBranch feature.


I support switching to `/head` and advising people to manually merge PR's to the target branch. Maybe it would be nice to implement this in a configurable way in the step?
@tardyp thanks for moving my suggestion/mail to the right place.

Here's another reference to the same problem: https://discourse.drone.io/t/github-claims-that-merge-refs-are-undocumented-feature/1100

Anecdotally, it seems this problem is accentuated in the case of forced pushes with amended commits and seems to go away after some variable time, but I've seen it last for hours.

While I do appreciate the usefulness of building against the merge ref and personally being totally ok with just enabling building from the head ref with a flag, I'd like to point out that this is a _silent_ failure. If the previous version of a patch built ok, but the updated rev doesn't, the user won't know and might merge something that breaks master (actually happened to me twice already).

Conceptually the usefulness of the build against the merge ref also has it's issues. Old patches will have been built against an old merge ref. If the merge ref is constantly updated as master moves forward (but the PR doesn't) and re-triggers would test on an updated merge ref, which I'm unsure about, it's now easy for the user to just rebase or re-merge on the latest master with a push of a button. Something the reviewer might require prior to allowing closure of the PR.

As a final (even more anecdotal) note, after using another build system that builds against head for a few years (gerrit+jenkins) on reasonably big project it's been really rare that merging a patch broke master, particularly because github/gerrit won't allow the merge if there are new conflicts.
@dralves The build isn't retriggered if master/other base is updated. The PR is only tested for every new commit. 

There are clearly pro's and con's for the `merge` ref.
I'll put up a PR so people (like me) can disable this feature on demand in in the hook configuration. This shouldn't be the new default behaviour though since this will break existing configs.



Resolving patch:
diff --git a/master/buildbot/www/hooks/github.py b/master/buildbot/www/hooks/github.py
--- a/master/buildbot/www/hooks/github.py
+++ b/master/buildbot/www/hooks/github.py
@@ -48,6 +48,7 @@ def __init__(self, secret, strict,
                  master=None,
                  skips=None,
                  github_api_endpoint=None,
+                 pullrequest_ref=None,
                  token=None,
                  debug=False,
                  verify=False):
@@ -55,6 +56,7 @@ def __init__(self, secret, strict,
         self._strict = strict
         self._token = token
         self._codebase = codebase
+        self.pullrequest_ref = pullrequest_ref
         self.github_property_whitelist = github_property_whitelist
         self.skips = skips
         self.github_api_endpoint = github_api_endpoint
@@ -156,7 +158,7 @@ def handle_push(self, payload, event):
     def handle_pull_request(self, payload, event):
         changes = []
         number = payload['number']
-        refname = 'refs/pull/{}/merge'.format(number)
+        refname = 'refs/pull/{}/{}'.format(number, self.pullrequest_ref)
         commits = payload['pull_request']['commits']
         title = payload['pull_request']['title']
         comments = payload['pull_request']['body']
@@ -327,6 +329,7 @@ def __init__(self, master, options):
             'github_property_whitelist': options.get('github_property_whitelist', None),
             'skips': options.get('skips', None),
             'github_api_endpoint': options.get('github_api_endpoint', None) or 'https://api.github.com',
+            'pullrequest_ref': options.get('pullrequest_ref', None) or 'merge',
             'token': options.get('token', None),
             'debug': options.get('debug', None) or False,
             'verify': options.get('verify', None) or False,


Test patch:
diff --git a/master/buildbot/test/unit/test_www_hooks_github.py b/master/buildbot/test/unit/test_www_hooks_github.py
--- a/master/buildbot/test/unit/test_www_hooks_github.py
+++ b/master/buildbot/test/unit/test_www_hooks_github.py
@@ -860,6 +860,35 @@ def test_git_pull_request_with_skip_message(self):
                 gitJsonPayloadPullRequest)
 
 
+class TestChangeHookConfiguredWithGitChangeCustomPullrequestRef(unittest.TestCase):
+
+    @defer.inlineCallbacks
+    def setUp(self):
+        self.changeHook = _prepare_github_change_hook(
+            strict=False, github_property_whitelist=["github.*"], pullrequest_ref="head")
+        self.master = self.changeHook.master
+        fake_headers = {'User-Agent': 'Buildbot'}
+        self._http = yield fakehttpclientservice.HTTPClientService.getFakeService(
+            self.master, self, 'https://api.github.com', headers=fake_headers,
+            debug=False, verify=False)
+        yield self.master.startService()
+
+    @defer.inlineCallbacks
+    def tearDown(self):
+        yield self.master.stopService()
+
+    @defer.inlineCallbacks
+    def test_git_pull_request_with_custom_ref(self):
+        commit = deepcopy([gitJsonPayloadPullRequest])
+        api_endpoint = '/repos/defunkt/github/commits/05c588ba8cd510ecbe112d020f215facb17817a7'
+        self._http.expect('get', api_endpoint, content_json=gitJsonPayloadCommit)
+        self.request = _prepare_request('pull_request', commit)
+        yield self.request.test_render(self.changeHook)
+        self.assertEqual(len(self.changeHook.master.addedChanges), 1)
+        change = self.changeHook.master.addedChanges[0]
+        self.assertEqual(change["branch"], "refs/pull/50/head")
+
+
 class TestChangeHookConfiguredWithCustomSkips(unittest.TestCase):
 
     @defer.inlineCallbacks