Repository: getsentry/sentry

Issue:
Slack alert message doesn't match up with why alert was actually triggered
### Environment

SaaS (https://sentry.io/)

### Steps to Reproduce

Hi Team,

A customer reported to us that the messages in their Slack alerts doesn't match up with the real reason the alert was triggered (displayed correctly once you click into the alert and go to the alert page). I was able to replicate this by setting up a similar type of alert.
1. Set up a metric alert to trigger when `Number of errors is X% higher in Y minutes compared to same time one week ago`
2. Alert action is to be notified via Slack, specifically a Slack channel
3. Receive Slack alert
<img width="1269" alt="Screenshot 2023-05-04 at 3 45 03 PM" src="https://user-images.githubusercontent.com/98921471/236560084-f64cf7cf-7856-4921-8c3d-ca70433afda8.png">


### Expected Result

When the alert triggers, I expect the message in Slack to say: "157 events in the last 10 minutes". This is verified by the alert details of why the alert triggered.

Alternatively, it can also be expected to say: "Number of errors is 89.16% higher in 10 minutes compared to the same time a week ago".
<img width="1277" alt="Screenshot 2023-05-04 at 3 45 14 PM" src="https://user-images.githubusercontent.com/98921471/236560515-38e4361c-0c8e-4c53-8986-fcd5a67f40fe.png">


### Actual Result

The Slack alert message says "168.478260.. events in the last 10 minutes". This doesn't match up with the 157 events that I see in the alert details.

For the customer, it said "400.0 events in the last 60 minutes" when it should've said "4 events in the last 60 minutes", which is 2 orders of magnitudes off.
<img width="544" alt="Screenshot 2023-05-04 at 3 44 30 PM" src="https://user-images.githubusercontent.com/98921471/236560823-894e8da5-6918-416c-8f65-f182456424d7.png">


### Product Area

Alerts

### Link

_No response_

### DSN

_No response_

### Version

_No response_

┆Issue is synchronized with this [Jira Improvement](https://getsentry.atlassian.net/browse/FEEDBACK-1825) by [Unito](https://www.unito.io)



Discussion:
Assigning to @getsentry/support for [routing](https://open.sentry.io/triage/#2-route), due by **<time datetime=2023-05-08T20:19:33.252Z>Monday, May 8th at 1:19 pm</time> (sfo)**. ⏲️
Routing to @getsentry/product-owners-alerts for [triage](https://develop.sentry.dev/processing-tickets/#3-triage), due by **<time datetime=2023-05-09T20:33:39.228Z>Tuesday, May 9th at 1:33 pm</time> (sfo)**. ⏲️
This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it `Status: Backlog` or `Status: In Progress`, I will leave it alone ... forever!

----

"A weed is but an unloved flower." ― _Ella Wheeler Wilcox_ 🥀
Hello, is there an update on this issue? Our team is seeing very similar behavior with our alerting, to the point that we have disabled them.

For example, we received an alert that says "1200.0 events in the last 30 minutes", yet when I check the `Issues` page for the same project, there weren't even 1200 events total for the entire previous week. Our rules seem correct, and we are unsure what else there is to try.

In fact, we suspected that the reported event count was roughly 100x higher than it should be, which seems to align with what was claimed by the customer in this issue:

> For the customer, it said "400.0 events in the last 60 minutes" when it should've said "4 events in the last 60 minutes", which is 2 orders of magnitudes off.
@brianthi @scefali to comment on status of this issue in our backlog.

Resolving patch:
diff --git a/src/sentry/integrations/metric_alerts.py b/src/sentry/integrations/metric_alerts.py
--- a/src/sentry/integrations/metric_alerts.py
+++ b/src/sentry/integrations/metric_alerts.py
@@ -11,6 +11,7 @@
 from sentry.incidents.models import (
     INCIDENT_STATUS,
     AlertRule,
+    AlertRuleThresholdType,
     Incident,
     IncidentStatus,
     IncidentTrigger,
@@ -25,6 +26,16 @@
     "percentage(users_crashed, users)": "% users crash free rate",
 }
 LOGO_URL = absolute_uri(get_asset_url("sentry", "images/sentry-email-avatar.png"))
+# These should be the same as the options in the frontend
+# COMPARISON_DELTA_OPTIONS
+TEXT_COMPARISON_DELTA = {
+    5: ("same time 5 minutes ago"),  # 5 minutes
+    15: ("same time 15 minutes ago"),  # 15 minutes
+    60: ("same time one hour ago"),  # one hour
+    1440: ("same time one day ago"),  # one day
+    10080: ("same time one week ago"),  # one week
+    43200: ("same time one month ago"),  # 30 days
+}
 
 
 def get_metric_count_from_incident(incident: Incident) -> str:
@@ -67,14 +78,27 @@ def get_incident_status_text(alert_rule: AlertRule, metric_value: str) -> str:
 
     time_window = alert_rule.snuba_query.time_window // 60
     interval = "minute" if time_window == 1 else "minutes"
-    text = _("%(metric_and_agg_text)s in the last %(time_window)d %(interval)s") % {
+    # % change alerts have a comparison delta
+    if alert_rule.comparison_delta:
+        metric_and_agg_text = f"{agg_text.capitalize()} {int(metric_value)}%"
+        higher_or_lower = (
+            "higher" if alert_rule.threshold_type == AlertRuleThresholdType.ABOVE.value else "lower"
+        )
+        comparison_delta_minutes = alert_rule.comparison_delta // 60
+        comparison_string = TEXT_COMPARISON_DELTA.get(
+            comparison_delta_minutes, f"same time {comparison_delta_minutes} minutes ago"
+        )
+        return _(
+            f"{metric_and_agg_text} {higher_or_lower} in the last {time_window} {interval} "
+            f"compared to the {comparison_string}"
+        )
+
+    return _("%(metric_and_agg_text)s in the last %(time_window)d %(interval)s") % {
         "metric_and_agg_text": metric_and_agg_text,
         "time_window": time_window,
         "interval": interval,
     }
 
-    return text
-
 
 def incident_attachment_info(incident, new_status: IncidentStatus, metric_value=None):
     alert_rule = incident.alert_rule


Test patch:
diff --git a/tests/sentry/integrations/test_metric_alerts.py b/tests/sentry/integrations/test_metric_alerts.py
--- a/tests/sentry/integrations/test_metric_alerts.py
+++ b/tests/sentry/integrations/test_metric_alerts.py
@@ -124,6 +124,52 @@ def test_with_incident_trigger(self):
             == "http://testserver/_static/{version}/sentry/images/sentry-email-avatar.png"
         )
 
+    def test_percent_change_alert(self):
+        # 1 hour comparison_delta
+        alert_rule = self.create_alert_rule(comparison_delta=60)
+        date_started = self.now
+        incident = self.create_incident(
+            self.organization,
+            title="Incident #1",
+            projects=[self.project],
+            alert_rule=alert_rule,
+            status=IncidentStatus.CLOSED.value,
+            date_started=date_started,
+        )
+        trigger = self.create_alert_rule_trigger(alert_rule, CRITICAL_TRIGGER_LABEL, 100)
+        self.create_alert_rule_trigger_action(
+            alert_rule_trigger=trigger, triggered_for_incident=incident
+        )
+        metric_value = 123.12
+        data = incident_attachment_info(incident, IncidentStatus.CRITICAL, metric_value)
+        assert (
+            data["text"]
+            == "Events 123% higher in the last 10 minutes compared to the same time one hour ago"
+        )
+
+    def test_percent_change_alert_custom_comparison_delta(self):
+        # 12 hour comparison_delta
+        alert_rule = self.create_alert_rule(comparison_delta=60 * 12)
+        date_started = self.now
+        incident = self.create_incident(
+            self.organization,
+            title="Incident #1",
+            projects=[self.project],
+            alert_rule=alert_rule,
+            status=IncidentStatus.CLOSED.value,
+            date_started=date_started,
+        )
+        trigger = self.create_alert_rule_trigger(alert_rule, CRITICAL_TRIGGER_LABEL, 100)
+        self.create_alert_rule_trigger_action(
+            alert_rule_trigger=trigger, triggered_for_incident=incident
+        )
+        metric_value = 123.12
+        data = incident_attachment_info(incident, IncidentStatus.CRITICAL, metric_value)
+        assert (
+            data["text"]
+            == "Events 123% higher in the last 10 minutes compared to the same time 720 minutes ago"
+        )
+
 
 MOCK_NOW = timezone.now().replace(hour=13, minute=0, second=0, microsecond=0)