Repository: buildbot/buildbot

Issue:
Bitbucket Cloud web hook returns 500
Started three weeks ago (missed it due to vacations), my guess is that Bitbucket changed something. Error log looks like this:

```
2019-07-02 14:33:40+0300 [_GenericHTTPChannelProtocol,13,18.234.32.226] adding changes from web hook
        Traceback (most recent call last):
          File "/home/buildbot/sandbox/lib/python3.5/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
            result = g.send(result)
          File "/home/buildbot/sandbox/lib/python3.5/site-packages/buildbot/www/change_hook.py", line 106, in getAndSubmitChanges
            changes, src = yield self.getChanges(request)
          File "/home/buildbot/sandbox/lib/python3.5/site-packages/twisted/internet/defer.py", line 1613, in unwindGenerator
            return _cancellableInlineCallbacks(gen)
          File "/home/buildbot/sandbox/lib/python3.5/site-packages/twisted/internet/defer.py", line 1529, in _cancellableInlineCallbacks
            _inlineCallbacks(None, g, status)
        --- <exception caught here> ---
          File "/home/buildbot/sandbox/lib/python3.5/site-packages/buildbot/www/change_hook.py", line 106, in getAndSubmitChanges
            changes, src = yield self.getChanges(request)
          File "/home/buildbot/sandbox/lib/python3.5/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
            result = g.send(result)
          File "/home/buildbot/sandbox/lib/python3.5/site-packages/buildbot/www/change_hook.py", line 167, in getChanges
            changes, src = yield handler.getChanges(request)
          File "/home/buildbot/sandbox/lib/python3.5/site-packages/buildbot/www/hooks/bitbucketcloud.py", line 163, in getChanges
            return self.process(request)
          File "/home/buildbot/sandbox/lib/python3.5/site-packages/buildbot/www/hooks/bitbucketcloud.py", line 54, in process
            return handler(payload)
          File "/home/buildbot/sandbox/lib/python3.5/site-packages/buildbot/www/hooks/bitbucketcloud.py", line 97, in handle_repo_push
            payload['actor']['username']),
        builtins.KeyError: 'username'
```

Last line reveals the culprit.


Discussion:
I'm wondering if the Bitbucket API has changed.
@p12tic
Apparently they did: [Bitbucket Cloud REST APIs are changing to improve user privacy](https://developer.atlassian.com/cloud/bitbucket/bitbucket-api-changes-gdpr/) (from 12 April 2019). Specifically they removed the `username` field per this [API documentation note](https://confluence.atlassian.com/bitbucket/event-payloads-740262817.html#EventPayloads-entity_user) so merely removing it should fix it.
Here's a patch against [bitbucketcloud.py](https://github.com/buildbot/buildbot/blob/master/master/buildbot/www/hooks/bitbucketcloud.py) that gave me a working Bitbucket hooks. Two changes:
* replace the `username` property with `nickname` (per the above privacy changes in API)
* per [Bitbucket documentation](https://confluence.atlassian.com/bitbucket/event-payloads-740262817.html#EventPayloads-entity_repositoryRepository) the `project` property in `repository` is not guaranteed to be always present which was also crashing my Buildbot
```
--- bitbucketcloud.py	2019-08-01 13:45:47.525672898 +0000
+++ ./lib/python3.6/site-packages/buildbot/www/hooks/bitbucketcloud.py	2019-08-01 13:02:29.471261055 +0000
@@ -70,7 +70,10 @@
 
     def handle_repo_push(self, payload):
         changes = []
+        if payload['repository'].get('project'):
         project = payload['repository']['project']['name']
+        else:
+            project = 'none'
         repo_url = payload['repository']['links']['self']['href']
         web_url = payload['repository']['links']['html']['href']
 
@@ -94,7 +97,7 @@
                 'revlink': '{}/commits/{}'.format(web_url, commit_hash),
                 'repository': repo_url,
                 'author': '{} <{}>'.format(payload['actor']['display_name'],
-                                           payload['actor']['username']),
+                                           payload['actor']['nickname']),
                 'comments': 'Bitbucket Cloud commit {}'.format(commit_hash),
                 'branch': branch,
                 'project': project,
@@ -139,6 +142,10 @@
     def handle_pullrequest(self, payload, refname, category):
         pr_number = int(payload['pullrequest']['id'])
         repo_url = payload['repository']['links']['self']['href']
+        if payload['repository'].get('project'):
+            project = payload['repository']['project']['name']
+        else:
+            project = 'none'
         change = {
             'revision': payload['pullrequest']['fromRef']['commit']['hash'],
             'revlink': payload['pullrequest']['link'],
@@ -147,7 +154,7 @@
                                        payload['actor']['username']),
             'comments': 'Bitbucket Cloud Pull Request #{}'.format(pr_number),
             'branch': refname,
-            'project': payload['repository']['project']['name'],
+            'project': project,
             'category': category,
             'properties': {'pullrequesturl': payload['pullrequest']['link']}
         }
```
Hi,
Could you please submit a proper pull request?

Buildbot is 100% maintained by volunteers. 
We rely on the bitbucket community to update plugins like bitbucketcloud hook.

Thank you for your understanding.

Resolving patch:
diff --git a/master/buildbot/www/hooks/bitbucketcloud.py b/master/buildbot/www/hooks/bitbucketcloud.py
--- a/master/buildbot/www/hooks/bitbucketcloud.py
+++ b/master/buildbot/www/hooks/bitbucketcloud.py
@@ -70,7 +70,7 @@ def _get_payload(self, request):
 
     def handle_repo_push(self, payload):
         changes = []
-        project = payload['repository']['project']['name']
+        project = payload['repository'].get('project', {'name': 'none'})['name']
         repo_url = payload['repository']['links']['self']['href']
         web_url = payload['repository']['links']['html']['href']
 
@@ -94,7 +94,7 @@ def handle_repo_push(self, payload):
                 'revlink': '{}/commits/{}'.format(web_url, commit_hash),
                 'repository': repo_url,
                 'author': '{} <{}>'.format(payload['actor']['display_name'],
-                                           payload['actor']['username']),
+                                           payload['actor']['nickname']),
                 'comments': 'Bitbucket Cloud commit {}'.format(commit_hash),
                 'branch': branch,
                 'project': project,
@@ -139,15 +139,16 @@ def handle_pullrequest_rejected(self, payload):
     def handle_pullrequest(self, payload, refname, category):
         pr_number = int(payload['pullrequest']['id'])
         repo_url = payload['repository']['links']['self']['href']
+        project = payload['repository'].get('project', {'name': 'none'})['name']
         change = {
             'revision': payload['pullrequest']['fromRef']['commit']['hash'],
             'revlink': payload['pullrequest']['link'],
             'repository': repo_url,
             'author': '{} <{}>'.format(payload['actor']['display_name'],
-                                       payload['actor']['username']),
+                                       payload['actor']['nickname']),
             'comments': 'Bitbucket Cloud Pull Request #{}'.format(pr_number),
             'branch': refname,
-            'project': payload['repository']['project']['name'],
+            'project': project,
             'category': category,
             'properties': {'pullrequesturl': payload['pullrequest']['link']}
         }


Test patch:
diff --git a/master/buildbot/test/unit/test_www_hooks_bitbucketcloud.py b/master/buildbot/test/unit/test_www_hooks_bitbucketcloud.py
--- a/master/buildbot/test/unit/test_www_hooks_bitbucketcloud.py
+++ b/master/buildbot/test/unit/test_www_hooks_bitbucketcloud.py
@@ -31,7 +31,7 @@
 pushJsonPayload = """
 {
     "actor": {
-        "username": "John",
+        "nickname": "John",
         "display_name": "John Smith"
     },
     "repository": {
@@ -52,7 +52,7 @@
         "public": false,
         "ownerName": "CI",
         "owner": {
-            "username": "CI",
+            "nickname": "CI",
             "display_name": "CI"
         },
         "fullName": "CI/py-repo"
@@ -87,7 +87,7 @@
 pullRequestCreatedJsonPayload = """
 {
     "actor": {
-        "username": "John",
+        "nickname": "John",
         "display_name": "John Smith"
     },
     "pullrequest": {
@@ -111,7 +111,7 @@
                 "public": false,
                 "ownerName": "CI",
                 "owner": {
-                    "username": "CI",
+                    "nickname": "CI",
                     "display_name": "CI"
                 },
                 "fullName": "CI/py-repo"
@@ -143,7 +143,7 @@
                 "public": false,
                 "ownerName": "CI",
                 "owner": {
-                    "username": "CI",
+                    "nickname": "CI",
                     "display_name": "CI"
                 },
                 "fullName": "CI/py-repo"
@@ -175,7 +175,7 @@
         "public": false,
         "ownerName": "CI",
         "owner": {
-            "username": "CI",
+            "nickname": "CI",
             "display_name": "CI"
         },
         "fullName": "CI/py-repo"
@@ -186,7 +186,7 @@
 pullRequestUpdatedJsonPayload = """
 {
     "actor": {
-        "username": "John",
+        "nickname": "John",
         "display_name": "John Smith"
     },
     "pullrequest": {
@@ -210,7 +210,7 @@
                 "public": false,
                 "ownerName": "CI",
                 "owner": {
-                    "username": "CI",
+                    "nickname": "CI",
                     "display_name": "CI"
                 },
                 "fullName": "CI/py-repo"
@@ -242,7 +242,7 @@
                 "public": false,
                 "ownerName": "CI",
                 "owner": {
-                    "username": "CI",
+                    "nickname": "CI",
                     "display_name": "CI"
                 },
                 "fullName": "CI/py-repo"
@@ -274,7 +274,7 @@
         "public": false,
         "ownerName": "CI",
         "owner": {
-            "username": "CI",
+            "nickname": "CI",
             "display_name": "CI"
         },
         "fullName": "CI/py-repo"
@@ -285,7 +285,7 @@
 pullRequestRejectedJsonPayload = """
 {
     "actor": {
-        "username": "John",
+        "nickname": "John",
         "display_name": "John Smith"
     },
     "pullrequest": {
@@ -309,7 +309,7 @@
                 "public": false,
                 "ownerName": "CI",
                 "owner": {
-                    "username": "CI",
+                    "nickname": "CI",
                     "display_name": "CI"
                 },
                 "fullName": "CI/py-repo"
@@ -341,7 +341,7 @@
                 "public": false,
                 "ownerName": "CI",
                 "owner": {
-                    "username": "CI",
+                    "nickname": "CI",
                     "display_name": "CI"
                 },
                 "fullName": "CI/py-repo"
@@ -373,7 +373,7 @@
         "public": false,
         "ownerName": "CI",
         "owner": {
-            "username": "CI",
+            "nickname": "CI",
             "display_name": "CI"
         },
         "fullName": "CI/py-repo"
@@ -384,7 +384,7 @@
 pullRequestFulfilledJsonPayload = """
 {
     "actor": {
-        "username": "John",
+        "nickname": "John",
         "display_name": "John Smith"
     },
     "pullrequest": {
@@ -408,7 +408,7 @@
                 "public": false,
                 "ownerName": "CI",
                 "owner": {
-                    "username": "CI",
+                    "nickname": "CI",
                     "display_name": "CI"
                 },
                 "fullName": "CI/py-repo"
@@ -440,7 +440,7 @@
                 "public": false,
                 "ownerName": "CI",
                 "owner": {
-                    "username": "CI",
+                    "nickname": "CI",
                     "display_name": "CI"
                 },
                 "fullName": "CI/py-repo"
@@ -472,7 +472,7 @@
         "public": false,
         "ownerName": "CI",
         "owner": {
-            "username": "CI",
+            "nickname": "CI",
             "display_name": "CI"
         },
         "fullName": "CI/py-repo"
@@ -483,7 +483,7 @@
 deleteTagJsonPayload = """
 {
     "actor": {
-        "username": "John",
+        "nickname": "John",
         "display_name": "John Smith"
     },
     "repository": {
@@ -504,7 +504,7 @@
         "ownerName": "BUIL",
         "public": false,
         "owner": {
-            "username": "CI",
+            "nickname": "CI",
             "display_name": "CI"
         },
         "fullName": "CI/py-repo"
@@ -532,7 +532,7 @@
 deleteBranchJsonPayload = """
 {
     "actor": {
-        "username": "John",
+        "nickname": "John",
         "display_name": "John Smith"
     },
     "repository": {
@@ -553,7 +553,7 @@
         "ownerName": "CI",
         "public": false,
         "owner": {
-            "username": "CI",
+            "nickname": "CI",
             "display_name": "CI"
         },
         "fullName": "CI/py-repo"
@@ -581,7 +581,7 @@
 newTagJsonPayload = """
 {
     "actor": {
-        "username": "John",
+        "nickname": "John",
         "display_name": "John Smith"
     },
     "repository": {
@@ -602,7 +602,7 @@
         "public": false,
         "ownerName": "CI",
         "owner": {
-            "username": "CI",
+            "nickname": "CI",
             "display_name": "CI"
         },
         "fullName": "CI/py-repo"