Repository: zulip/zulip

Issue:
Semaphore CI's webhook notifications don't work
It looks like the payload format for the webhook notifications has changed. The old format described [here](https://semaphoreci.com/docs/branches-and-builds-api.html) is expected by the webhook code. The test fixtures also have payloads that look very similar to what has been described here. 

The new payload format is documented [here](https://docs.semaphoreci.com/essentials/webhook-notifications/#notification-payload)
Semaphore CI's webhook notifications don't work
It looks like the payload format for the webhook notifications has changed. The old format described [here](https://semaphoreci.com/docs/branches-and-builds-api.html) is expected by the webhook code. The test fixtures also have payloads that look very similar to what has been described here. 

The new payload format is documented [here](https://docs.semaphoreci.com/essentials/webhook-notifications/#notification-payload)


Discussion:
Hello @zulip/server-integrations members, this issue was labeled with the "area: integrations" label, so you may want to check it out!

<!-- areaLabelAddition -->
@zulipbot  claim
Welcome to Zulip, @abhinav824! We just sent you an invite to collaborate on this repository at https://github.com/zulip/zulip/invitations. Please accept this invite in order to claim this issue and begin a fun, rewarding experience contributing to Zulip!

Here's some tips to get you off to a good start:
* Join me on the [Zulip developers' server](https://chat.zulip.org), to get help, chat about this issue, and meet the other developers.
* [Unwatch this repository](https://help.github.com/articles/unwatching-repositories/), so that you don't get 100 emails a day.

As you work on this issue, you'll also want to refer to the [Zulip code contribution guide](https://zulip.readthedocs.io/en/latest/contributing/index.html), as well as the rest of the developer documentation on that site.

See you on the other side (that is, the pull request side)!
The notification payload seems less useful than what it was before - links to the commit/changes, and a link to the build are missing from the payload. :see_no_evil: 
I've also done some research on this and i want to mention a few things :
1. There are two versions of semaphore - **Semaphore classic** and **Semaphore 2.0**, which are operating as two distinct products. Have a look on this [page](https://docs.semaphoreci.com/guided-tour/migration-guide-for-semaphore-classic-users/#semaphore-classic-and-semaphore-20-are-distinct-products).
1.  So, our existing code should work for semaphore classic (well, i'm not sure whether it's working or not) and we need to extend our support for semaphore 2.0. (The old payload corresponds to semaphore classic and the new one is for Semaphore 2.0).
1.  For extending our support to semaphore 2.0, i think we need to understand the basic working [concept](https://docs.semaphoreci.com/guided-tour/concepts/#concepts) behind it (I believe it's different from the classic one). Maybe, then we'll be able to extract the useful information from the new notification payload and send a notification.
1. And yes, i also think that the new payload is somewhat less useful. Anyway we can generate commit links using the commit id's but, maybe we cannot generate link to the build.
@punchagan I have two kinds of approach to this issue. 
We can handle both the payloads in the existing view and trigger the notification according to the payload we'll receive or we can have a separate integration for Semaphore 2.0. 
So, how should i proceed now ?
From a user's perspective, it might be best to just have a single integration and do the right thing in the webhook code. 

But, the help instructions in the `/integrations` page would probably be slightly different for the two versions, since the UI in Semaphore would be different, and the notification in Zulip itself would be slightly different because the payloads are different. 

Following from the Bitbucket example, having a separate integration might be the way to go. 
Hello @zulip/server-integrations members, this issue was labeled with the "area: integrations" label, so you may want to check it out!

<!-- areaLabelAddition -->
@zulipbot  claim
Welcome to Zulip, @abhinav824! We just sent you an invite to collaborate on this repository at https://github.com/zulip/zulip/invitations. Please accept this invite in order to claim this issue and begin a fun, rewarding experience contributing to Zulip!

Here's some tips to get you off to a good start:
* Join me on the [Zulip developers' server](https://chat.zulip.org), to get help, chat about this issue, and meet the other developers.
* [Unwatch this repository](https://help.github.com/articles/unwatching-repositories/), so that you don't get 100 emails a day.

As you work on this issue, you'll also want to refer to the [Zulip code contribution guide](https://zulip.readthedocs.io/en/latest/contributing/index.html), as well as the rest of the developer documentation on that site.

See you on the other side (that is, the pull request side)!
The notification payload seems less useful than what it was before - links to the commit/changes, and a link to the build are missing from the payload. :see_no_evil: 
I've also done some research on this and i want to mention a few things :
1. There are two versions of semaphore - **Semaphore classic** and **Semaphore 2.0**, which are operating as two distinct products. Have a look on this [page](https://docs.semaphoreci.com/guided-tour/migration-guide-for-semaphore-classic-users/#semaphore-classic-and-semaphore-20-are-distinct-products).
1.  So, our existing code should work for semaphore classic (well, i'm not sure whether it's working or not) and we need to extend our support for semaphore 2.0. (The old payload corresponds to semaphore classic and the new one is for Semaphore 2.0).
1.  For extending our support to semaphore 2.0, i think we need to understand the basic working [concept](https://docs.semaphoreci.com/guided-tour/concepts/#concepts) behind it (I believe it's different from the classic one). Maybe, then we'll be able to extract the useful information from the new notification payload and send a notification.
1. And yes, i also think that the new payload is somewhat less useful. Anyway we can generate commit links using the commit id's but, maybe we cannot generate link to the build.
@punchagan I have two kinds of approach to this issue. 
We can handle both the payloads in the existing view and trigger the notification according to the payload we'll receive or we can have a separate integration for Semaphore 2.0. 
So, how should i proceed now ?
From a user's perspective, it might be best to just have a single integration and do the right thing in the webhook code. 

But, the help instructions in the `/integrations` page would probably be slightly different for the two versions, since the UI in Semaphore would be different, and the notification in Zulip itself would be slightly different because the payloads are different. 

Following from the Bitbucket example, having a separate integration might be the way to go. 

Resolving patch:
diff --git a/zerver/lib/integrations.py b/zerver/lib/integrations.py
--- a/zerver/lib/integrations.py
+++ b/zerver/lib/integrations.py
@@ -327,7 +327,7 @@ def __init__(self, name: str, *args: Any, **kwargs: Any) -> None:
     WebhookIntegration('pivotal', ['project-management'], display_name='Pivotal Tracker'),
     WebhookIntegration('raygun', ['monitoring'], display_name="Raygun"),
     WebhookIntegration('reviewboard', ['version-control'], display_name="ReviewBoard"),
-    WebhookIntegration('semaphore', ['continuous-integration', 'deployment'], stream_name='builds'),
+    WebhookIntegration('semaphore', ['continuous-integration', 'deployment']),
     WebhookIntegration('sentry', ['monitoring']),
     WebhookIntegration('slack', ['communication']),
     WebhookIntegration('solano', ['continuous-integration'], display_name='Solano Labs'),
diff --git a/zerver/webhooks/semaphore/view.py b/zerver/webhooks/semaphore/view.py
--- a/zerver/webhooks/semaphore/view.py
+++ b/zerver/webhooks/semaphore/view.py
@@ -1,5 +1,7 @@
 # Webhooks for external integrations.
-from typing import Any, Dict
+from typing import Any, Dict, Tuple, Optional
+
+from urllib.parse import urlparse
 
 from django.http import HttpRequest, HttpResponse
 
@@ -9,6 +11,8 @@
 from zerver.lib.webhooks.common import check_send_webhook_message
 from zerver.models import UserProfile
 
+# Semaphore Classic Templates
+
 BUILD_TEMPLATE = """
 [Build {build_number}]({build_url}) {status}:
 * **Commit**: [{commit_hash}: {commit_message}]({commit_url})
@@ -22,13 +26,83 @@
 * **Server**: {server_name}
 """.strip()
 
+# Semaphore 2.0 Templates
+
+# Currently, Semaphore 2.0 only supports GitHub, while Semaphore Classic
+# supports Bitbucket too. The payload does not have URLs for commits, tags,
+# pull requests, etc. So, we use separate templates for GitHub and construct
+# the URLs ourselves. For any other repository hosting services we use
+# templates that don't have any links in them.
+
+GH_PUSH_TEMPLATE = """
+[{pipeline_name}]({workflow_url}) pipeline **{pipeline_result}**:
+* **Commit**: [({commit_hash})]({commit_url}) {commit_message}
+* **Branch**: {branch_name}
+* **Author**: [{author_name}]({author_url})
+""".strip()
+
+PUSH_TEMPLATE = """
+[{pipeline_name}]({workflow_url}) pipeline **{pipeline_result}**:
+* **Commit**: ({commit_hash}) {commit_message}
+* **Branch**: {branch_name}
+* **Author**: {author_name}
+""".strip()
+
+GH_PULL_REQUEST_TEMPLATE = """
+[{pipeline_name}]({workflow_url}) pipeline **{pipeline_result}**:
+* **Pull Request**: [{pull_request_title}]({pull_request_url})
+* **Branch**: {branch_name}
+* **Author**: [{author_name}]({author_url})
+""".strip()
+
+PULL_REQUEST_TEMPLATE = """
+[{pipeline_name}]({workflow_url}) pipeline **{pipeline_result}**:
+* **Pull Request**: {pull_request_title} (#{pull_request_number})
+* **Branch**: {branch_name}
+* **Author**: {author_name}
+""".strip()
+
+GH_TAG_TEMPLATE = """
+[{pipeline_name}]({workflow_url}) pipeline **{pipeline_result}**:
+* **Tag**: [{tag_name}]({tag_url})
+* **Author**: [{author_name}]({author_url})
+""".strip()
+
+TAG_TEMPLATE = """
+[{pipeline_name}]({workflow_url}) pipeline **{pipeline_result}**:
+* **Tag**: {tag_name}
+* **Author**: {author_name}
+""".strip()
+
+DEFAULT_TEMPLATE = """
+[{pipeline_name}]({workflow_url}) pipeline **{pipeline_result}** for {event_name} event
+""".strip()
+
 TOPIC_TEMPLATE = "{project}/{branch}"
 
+GITHUB_URL_TEMPLATES = {
+    'commit': '{repo_url}/commit/{commit_id}',
+    'pull_request': '{repo_url}/pull/{pr_number}',
+    'tag': '{repo_url}/tree/{tag_name}',
+    'user': 'https://github.com/{username}',
+}
+
+
 @api_key_only_webhook_view('Semaphore')
 @has_request_variables
 def api_semaphore_webhook(request: HttpRequest, user_profile: UserProfile,
                           payload: Dict[str, Any]=REQ(argument_type='body')) -> HttpResponse:
 
+    content, project_name, branch_name = (
+        semaphore_classic(payload) if 'event' in payload else semaphore_2(payload)
+    )
+    subject = (
+        TOPIC_TEMPLATE.format(project=project_name, branch=branch_name) if branch_name else project_name
+    )
+    check_send_webhook_message(request, user_profile, subject, content)
+    return json_success()
+
+def semaphore_classic(payload: Dict[str, Any]) -> Tuple[str, str, str]:
     # semaphore only gives the last commit, even if there were multiple commits
     # since the last build
     branch_name = payload["branch_name"]
@@ -76,10 +150,65 @@ def api_semaphore_webhook(request: HttpRequest, user_profile: UserProfile,
         content = "{event}: {result}".format(
             event=event, result=result)
 
-    subject = TOPIC_TEMPLATE.format(
-        project=project_name,
-        branch=branch_name
+    return content, project_name, branch_name
+
+def semaphore_2(payload: Dict[str, Any]) -> Tuple[str, str, Optional[str]]:
+    repo_url = payload["repository"]["url"]
+    project_name = payload["project"]["name"]
+    organization_name = payload["organization"]["name"]
+    author_name = payload["revision"]["sender"]["login"]
+    workflow_id = payload['workflow']['id']
+    context = dict(
+        author_name=author_name,
+        author_url=GITHUB_URL_TEMPLATES['user'].format(repo_url=repo_url, username=author_name),
+        pipeline_name=payload["pipeline"]["name"],
+        pipeline_result=payload["pipeline"]["result"],
+        workflow_url='https://{org}.semaphoreci.com/workflows/{id}'.format(
+            org=organization_name, id=workflow_id)
     )
 
-    check_send_webhook_message(request, user_profile, subject, content)
-    return json_success()
+    if payload["revision"]["reference_type"] == "branch":  # push event
+        commit_id = payload["revision"]["commit_sha"]
+        branch_name = payload["revision"]["branch"]["name"]
+        context.update(
+            branch_name=branch_name,
+            commit_id=commit_id,
+            commit_hash=commit_id[:7],
+            commit_message=payload["revision"]["commit_message"],
+            commit_url=GITHUB_URL_TEMPLATES['commit'].format(repo_url=repo_url, commit_id=commit_id),
+        )
+        template = GH_PUSH_TEMPLATE if is_github_repo(repo_url) else PUSH_TEMPLATE
+        content = template.format(**context)
+    elif payload["revision"]["reference_type"] == "pull_request":
+        pull_request = payload["revision"]["pull_request"]
+        branch_name = pull_request["branch_name"]
+        pull_request_title = pull_request["name"]
+        pull_request_number = pull_request["number"]
+        pull_request_url = GITHUB_URL_TEMPLATES['pull_request'].format(
+            repo_url=repo_url, pr_number=pull_request_number)
+        context.update(
+            branch_name=branch_name,
+            pull_request_title=pull_request_title,
+            pull_request_url=pull_request_url,
+            pull_request_number=pull_request_number,
+        )
+        template = GH_PULL_REQUEST_TEMPLATE if is_github_repo(repo_url) else PULL_REQUEST_TEMPLATE
+        content = template.format(**context)
+    elif payload["revision"]["reference_type"] == "tag":
+        branch_name = ''
+        tag_name = payload["revision"]["tag"]["name"]
+        tag_url = GITHUB_URL_TEMPLATES['tag'].format(repo_url=repo_url, tag_name=tag_name)
+        context.update(
+            tag_name=tag_name,
+            tag_url=tag_url,
+        )
+        template = GH_TAG_TEMPLATE if is_github_repo(repo_url) else TAG_TEMPLATE
+        content = template.format(**context)
+    else:  # should never get here: unknown event
+        branch_name = ''
+        context.update(event_name=payload["revision"]["reference_type"])
+        content = DEFAULT_TEMPLATE.format(**context)
+    return content, project_name, branch_name
+
+def is_github_repo(repo_url: str) -> bool:
+    return urlparse(repo_url).hostname == 'github.com'


Test patch:
diff --git a/zerver/webhooks/semaphore/tests.py b/zerver/webhooks/semaphore/tests.py
--- a/zerver/webhooks/semaphore/tests.py
+++ b/zerver/webhooks/semaphore/tests.py
@@ -1,4 +1,6 @@
 # -*- coding: utf-8 -*-
+import ujson
+from mock import patch
 from zerver.lib.test_classes import WebhookTestCase
 
 
@@ -10,6 +12,7 @@ class SemaphoreHookTests(WebhookTestCase):
     # contain information on the repo and branch, and the message has links and
     # details about the build, deploy, server, author, and commit
 
+    # Tests for Semaphore Classic
     def test_semaphore_build(self) -> None:
         expected_topic = u"knighthood/master"  # repo/branch
         expected_message = """
@@ -31,5 +34,89 @@ def test_semaphore_deploy(self) -> None:
         self.send_and_test_stream_message('deploy', expected_topic, expected_message,
                                           content_type="application/x-www-form-urlencoded")
 
+    # Tests For Semaphore 2.0
+
+    def test_semaphore2_push(self) -> None:
+        expected_topic = u"notifications/rw/webhook_impl"  # repo/branch
+        expected_message = """
+[Notifications](https://semaphore.semaphoreci.com/workflows/acabe58e-4bcc-4d39-be06-e98d71917703) pipeline **stopped**:
+* **Commit**: [(2d9f5fc)](https://github.com/renderedtext/notifications/commit/2d9f5fcec1ca7c68fa7bd44dd58ec4ff65814563) Implement webhooks for SemaphoreCI
+* **Branch**: rw/webhook_impl
+* **Author**: [radwo](https://github.com/radwo)
+""".strip()
+        self.send_and_test_stream_message('push', expected_topic, expected_message,
+                                          content_type="application/json")
+
+    def test_semaphore2_push_non_gh_repo(self) -> None:
+        expected_topic = u"notifications/rw/webhook_impl"  # repo/branch
+        expected_message = """
+[Notifications](https://semaphore.semaphoreci.com/workflows/acabe58e-4bcc-4d39-be06-e98d71917703) pipeline **stopped**:
+* **Commit**: (2d9f5fc) Implement webhooks for SemaphoreCI
+* **Branch**: rw/webhook_impl
+* **Author**: radwo
+""".strip()
+        with patch('zerver.webhooks.semaphore.view.is_github_repo', return_value=False):
+            self.send_and_test_stream_message('push', expected_topic, expected_message,
+                                              content_type="application/json")
+
+    def test_semaphore_pull_request(self) -> None:
+        expected_topic = u"notifications/test-notifications"
+        expected_message = """
+[Notifications](https://semaphore.semaphoreci.com/workflows/84383f37-d025-4811-b719-61c6acc92a1e) pipeline **failed**:
+* **Pull Request**: [Testing PR notifications](https://github.com/renderedtext/notifications/pull/3)
+* **Branch**: test-notifications
+* **Author**: [radwo](https://github.com/radwo)
+""".strip()
+        self.send_and_test_stream_message('pull_request', expected_topic, expected_message,
+                                          content_type="application/json")
+
+    def test_semaphore_pull_request_non_gh_repo(self) -> None:
+        expected_topic = u"notifications/test-notifications"
+        expected_message = """
+[Notifications](https://semaphore.semaphoreci.com/workflows/84383f37-d025-4811-b719-61c6acc92a1e) pipeline **failed**:
+* **Pull Request**: Testing PR notifications (#3)
+* **Branch**: test-notifications
+* **Author**: radwo
+""".strip()
+        with patch('zerver.webhooks.semaphore.view.is_github_repo', return_value=False):
+            self.send_and_test_stream_message('pull_request', expected_topic, expected_message,
+                                              content_type="application/json")
+
+    def test_semaphore_tag(self) -> None:
+        expected_topic = u"notifications"
+        expected_message = """
+[Notifications](https://semaphore.semaphoreci.com/workflows/a8704319-2422-4828-9b11-6b2afa3554e6) pipeline **stopped**:
+* **Tag**: [v1.0.1](https://github.com/renderedtext/notifications/tree/v1.0.1)
+* **Author**: [radwo](https://github.com/radwo)
+""".strip()
+        self.send_and_test_stream_message('tag', expected_topic, expected_message,
+                                          content_type="application/json")
+
+    def test_semaphore_tag_non_gh_repo(self) -> None:
+        expected_topic = u"notifications"
+        expected_message = """
+[Notifications](https://semaphore.semaphoreci.com/workflows/a8704319-2422-4828-9b11-6b2afa3554e6) pipeline **stopped**:
+* **Tag**: v1.0.1
+* **Author**: radwo
+""".strip()
+        with patch('zerver.webhooks.semaphore.view.is_github_repo', return_value=False):
+            self.send_and_test_stream_message('tag', expected_topic, expected_message,
+                                              content_type="application/json")
+
+    def test_semaphore_unknown_event(self) -> None:
+        expected_topic = u"notifications"
+        expected_message = """
+[Notifications](https://semaphore.semaphoreci.com/workflows/a8704319-2422-4828-9b11-6b2afa3554e6) pipeline **stopped** for unknown event
+""".strip()
+        with patch('zerver.webhooks.semaphore.tests.SemaphoreHookTests.get_body', self.get_unknown_event):
+            self.send_and_test_stream_message('tag', expected_topic, expected_message,
+                                              content_type="application/json")
+
     def get_body(self, fixture_name: str) -> str:
         return self.webhook_fixture_data("semaphore", fixture_name, file_type="json")
+
+    def get_unknown_event(self, fixture_name: str) -> str:
+        """Return modified payload with revision.reference_type changed"""
+        fixture_data = ujson.loads(self.webhook_fixture_data("semaphore", fixture_name, file_type="json"))
+        fixture_data['revision']['reference_type'] = 'unknown'
+        return fixture_data