Repository: zulip/zulip

Issue:
Sentry integration "Test Plugin" failure
Hello, I'm setting up Sentry.io integration and got this error when I tried "Test Plugin" (meaning "Test Outgoing Webhook to Zulip"); I'm assuming it's the response payload that came back from Zulip Cloud:

>"There was an internal error with the Plugin, {\"result\":\"error\",\"msg\":\"The 'Raven SDK' event isn't currently supported by the Sentry webhook\",\"webhook_name\":\"Sentry\",\"event_type\":\"Raven SDK\",\"code\":\"UNSUPPORTED_WEBHOOK_EVENT_TYPE\"}\n"

I'm not sure if there are any events that do work because I'm new to Sentry and not sure how to trigger test events other than the Test Plugin event.

**Zulip Server and web app version:**

- [x] Zulip Cloud (`*.zulipchat.com`)
- [ ] Zulip Server 7.0+
- [ ] Zulip Server 6.0+
- [ ] Zulip Server 5.0 or older
- [ ] Other or not sure



Discussion:
@zulipbot add "area: integrations"
Hello @zulip/server-integrations members, this issue was labeled with the "area: integrations" label, so you may want to check it out!

<!-- areaLabelAddition -->

@aryairani Thanks for reporting this issue. 

This issue was recently fixed in 16563a321. I tried to reproduce this by pressing the **Send Test Notification** button while creating an alert rule and a notification was received as expected. I also inspected the received webhook and all seems to be alright.

Can you confirm if you are creating an "Alert Rule" as mentioned in the [documentation](https://zulip.com/integrations/doc/sentry)? 
Hi @sbansal1999,

This is under 
`https://<myorg>.sentry.io/settings/projects/<myproj>/plugins/webhooks/`
Pressing [Test Plugin], with Callback Url 
`https://<myzulip>.zulipchat.com/api/v1/external/sentry?api_key=<mykey>&stream=<mystream>`
Is this expected to work?

I'm working on getting permission to "Create New Integration"; fwiw the don't currently have permissions to create a custom integration for the whole org, I'm "only" admin for the project.

The webhook also appears under` https://<myorg>.sentry.io/settings/projects/<myproj>/alerts/`. Is this also not right?

Ok thanks!
Alright, I guess I got it sorted out, thanks for the hint.
I wonder if it might be worthwhile to add to a note that the Sentry Webhook integration is not the right way to configure Sentry alerts via Webhook to Zulip?
@aryairani  Our [Sentry documentation](https://zulip.com/integrations/doc/sentry) says:

>NOTE: Zulip also supports configuring this as a webhook in Sentry — which, while easier to configure (Navigate to Settings > Integrations > WebHooks) may not include the full breadth of event types.  

I tried creating a webhook using the `Integrations` list -- as you did. I was able to trigger issues on my sample project and then the events resulted in correct messages being sent by the bot. But as you pointed out when testing it by using the "Test Plugin" button it returns an error. 

So practically speaking, the webhook will still be able to receive events related to issues but won't respond to the test 

The only problematic thing here is that Sentry doesn't explicitly mention its condition for the sample event; hence making it difficult to identify them. So, I don't think there is a need to delete this from the documentation.
I see, sorry I missed that. Fwiw I find the integration docs to have a bit of a "wall of text" vibe, which makes it hard to follow (not just the Sentry integration docs); but I'm not a designer myself so I don't have any concrete suggestions on that front. 

Thanks for your help! Should I close the ticket?
#25976 is actively being worked on, which should help address the big block of text in various integrations documentation pages for generating the URL. But until that's done, I don't think there's much to be done about that block of text in step 2.

I did a little revision pass of the Sentry specific instructions (steps 3 and 4), and have #26224 up with an updated version of the current documentation that breaks up that text a bit and emphasized that warning/note about the alternative implementation via Sentry webhook.

Resolving patch:
diff --git a/zerver/webhooks/sentry/view.py b/zerver/webhooks/sentry/view.py
--- a/zerver/webhooks/sentry/view.py
+++ b/zerver/webhooks/sentry/view.py
@@ -81,6 +81,15 @@
 }
 
 
+def is_sample_event(event: Dict[str, Any]) -> bool:
+    # This is just a heuristic to detect the sample event, this should
+    # not be used for making important behavior decisions.
+    title = event.get("title", "")
+    if title == "This is an example Python exception":
+        return True
+    return False
+
+
 def convert_lines_to_traceback_string(lines: Optional[List[str]]) -> str:
     traceback = ""
     if lines is not None:
@@ -103,12 +112,10 @@ def handle_event_payload(event: Dict[str, Any]) -> Tuple[str, str]:
 
     # We shouldn't support the officially deprecated Raven series of
     # Python SDKs.
-    if platform_name == "python" and int(event["version"]) < 7:
+    if platform_name == "python" and int(event["version"]) < 7 and not is_sample_event(event):
         # The sample event is still an old "version" -- accept it even
         # though we don't accept events from the old Python SDK.
-        tags = event.get("tags", [])
-        if ["sample_event", "yes"] not in tags:
-            raise UnsupportedWebhookEventTypeError("Raven SDK")
+        raise UnsupportedWebhookEventTypeError("Raven SDK")
     context = {
         "title": subject,
         "level": event["level"],


Test patch:
diff --git a/zerver/webhooks/sentry/tests.py b/zerver/webhooks/sentry/tests.py
--- a/zerver/webhooks/sentry/tests.py
+++ b/zerver/webhooks/sentry/tests.py
@@ -251,7 +251,7 @@ def test_deprecated_exception_message(self) -> None:
 ```"""
         self.check_webhook("deprecated_exception_message", expected_topic, expected_message)
 
-    def test_sample_event(self) -> None:
+    def test_sample_event_through_alert(self) -> None:
         expected_topic = "This is an example Python exception"
         expected_message = """\
 **New message event:** [This is an example Python exception](https://sentry.io/organizations/nitk-46/issues/4218258981/events/b6eff1a49b1f4132850b1238d968da70/)
@@ -259,4 +259,27 @@ def test_sample_event(self) -> None:
 **level:** error
 **timestamp:** 2023-05-31 11:06:16
 ```"""
-        self.check_webhook("sample_event", expected_topic, expected_message)
+        self.check_webhook("sample_event_through_alert", expected_topic, expected_message)
+
+    def test_sample_event_through_plugin(self) -> None:
+        expected_topic = "This is an example Python exception"
+        expected_message = """\
+**New message event:** [This is an example Python exception](https://nitk-46.sentry.io/issues/4218258981/events/4dc4fc2858aa450eb658be9e5b8ad149/)
+```quote
+**level:** error
+**timestamp:** 2023-07-09 20:41:24
+```"""
+        self.check_webhook("sample_event_through_plugin", expected_topic, expected_message)
+
+    def test_raven_sdk_python_event(self) -> None:
+        payload = self.get_body("raven_sdk_python_event")
+        result = self.client_post(
+            self.url,
+            payload,
+            content_type="application/json",
+        )
+        self.assert_json_success(result)
+        self.assert_in_response(
+            "The 'Raven SDK' event isn't currently supported by the Sentry webhook; ignoring",
+            result,
+        )