Repository: getsentry/sentry

Issue:
Issue alerts not alerting on regressions
Issue alerts with "The issue changes state from resolved to unresolved" are not getting triggered when an issue is unresolved and has reappeared (resolved from not being seen).


Discussion:
2 weeks is a pretty serious delay for something like this. We are currently depending on this for alerting to work  properly.
Routing to @getsentry/product-owners-issues for [triage](https://develop.sentry.dev/processing-tickets/#3-triage) ⏲️
Hi
Do we have any update on when this issue will be sorted out ?
We are also heavily depending on alert rules for our alerting system.
Hey folks, I'm actively looking into this and hope to have an update soon. The issue isn't present in most organizations, so we're hoping to narrow down what exactly is going on here.
Any updates?
Our organization is also affected
Hi folks - thanks for adding to the issue. It's immensely helpful to find other affected organizations - @liferthedude, @leighst-anchorage, @simonot and others, can I ask what your org slug is? Or please file a support ticket with additional info! 🙏 
support ticket #100699 was placed earlier today

On Mon, Aug 14, 2023 at 10:05 PM Snigdha Sharma ***@***.***>
wrote:

> Hi folks - thanks for adding to the issue. It's immensely helpful to find
> other affected organizations - @liferthedude
> <https://github.com/liferthedude> @Simonot <https://github.com/Simonot>
> and others, can I ask what your org slug is? Or please file a support
> ticket with additional info! 🙏
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/getsentry/sentry/issues/52849#issuecomment-1677908774>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AOCCEA63L3DW7AHOMVVACB3XVJZIBANCNFSM6AAAAAA2JVSPZQ>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>

Please see support ticket #100051 reported on August 04, 2023. Thanks!
Also Support ticket #100820
Support ticket #101017
Support ticket #101274
Support ticket
"Your request (99846) has been updated. To add additional comments, reply to this email."

I am not sure this is the support ticket. So I will put back the content of the ticket here:
"
Hi Sentry team,
It seems that since a few weeks/(months ?) we have issue with the Alert Rules.
Indeed, we have some issues (mainly regression) that do not get catched by the Alert Rule. Therefore, the issue is not sent to our slack alerting channel.
What we can see is that:
- If you check at the dashbaord of the alert rule, the issue XXX (XXX is for the example) is not present
- If you then Edit the alert, you can see the issue XXX in the Preview
=> Which is strange, because if present in the preview, it means that the condition matches the Alert Rule

We have tried the following action last Thursday:
- Editing all alert rules with a new configuration
- Editing them back to the original configuration
=> The goal was to be sure we will catch everything in the Preview.
=> It work for a day or two, but we still have missing issues in the Alert Rules.

So the final issue is:
- We have Alert Rules configured to catch any new issue or regression
- We can see in the Preview of the Alert Rules Edition that we should catch some issue
- Nevertheless, in run time, only part of the issues are catched by the Alert Rule.

Here is an example with real data:
- Alert Rule critical issues carriers: https://shipup.sentry.io/alerts/rules/shipup-api-prod/9351537/
- We have in the Preview of the Edition the issue Tamdis::EventProducerWorker (https://shipup.sentry.io/issues/4325599743/?referrer=issue-stream&stream_index=0) (see screenshot attached)
- Same Alert Rule dashboard (https://shipup.sentry.io/alerts/rules/shipup-api-prod/9351537/details/?pageStatsPeriod=7d)
- We do see Tamdis alert (see screenshot), and we never received any slack message for it. As it was not catched by the Alert Rule

Is there any sort of Rate Limiting on Alert Rules ?
Or have we missconfigured something ?
Any help would be very much apprecitated.

Best regards
"

I can give more images or reopen a ticket with more fresh data if needed
@snigdhas @scttcper 

This issue is a simple enough to reproduce:
1. Create a alert rule that triggers on regressions
2. Send a test event to the project to create a new issue.
3. Go to the project settings -> general settings and set `Auto Resolve` to any value. 
4. Wait for the issue created on step2 to auto-resolve.
5. Send a new event to the same issue to trigger a regression

Expected: the alert created on step1 is triggered
Actual: the alert created on step1 is not triggered.

I just reproduced it:
Issue: https://dev-curumas.sentry.io/issues/4174538071/activity/?project=5591101&query=is%3Aunresolved&referrer=issue-stream&stream_index=0
Alert rule: https://dev-curumas.sentry.io/alerts/rules/nextjs/14529739/details/


Resolving patch:
diff --git a/src/sentry/tasks/post_process.py b/src/sentry/tasks/post_process.py
--- a/src/sentry/tasks/post_process.py
+++ b/src/sentry/tasks/post_process.py
@@ -684,6 +684,8 @@ def update_event_groups(event: Event, group_states: Optional[GroupStates] = None
         # deprecated event.group and event.group_id usage, kept here for backwards compatibility
         event.group, _ = get_group_with_redirect(event.group_id)
         event.group_id = event.group.id
+        # We buffer updates to last_seen, assume its at least >= the event datetime
+        event.group.last_seen = max(event.datetime, event.group.last_seen)
 
     # Re-bind Group since we're reading the Event object
     # from cache, which may contain a stale group and project
@@ -691,6 +693,8 @@ def update_event_groups(event: Event, group_states: Optional[GroupStates] = None
     rebound_groups = []
     for group_state in group_states:
         rebound_group = get_group_with_redirect(group_state["id"])[0]
+        # We buffer updates to last_seen, assume its at least >= the event datetime
+        rebound_group.last_seen = max(event.datetime, rebound_group.last_seen)
 
         # We fetch buffered updates to group aggregates here and populate them on the Group. This
         # helps us avoid problems with processing group ignores and alert rules that rely on these


Test patch:
diff --git a/tests/sentry/tasks/test_post_process.py b/tests/sentry/tasks/test_post_process.py
--- a/tests/sentry/tasks/test_post_process.py
+++ b/tests/sentry/tasks/test_post_process.py
@@ -463,6 +463,42 @@ def test_group_refresh(self, mock_processor):
             EventMatcher(event, group=group2), True, False, True, False
         )
 
+    @patch("sentry.rules.processor.RuleProcessor")
+    def test_group_last_seen_buffer(self, mock_processor):
+        first_event_date = datetime.now(timezone.utc) - timedelta(days=90)
+        event1 = self.create_event(
+            data={"message": "testing"},
+            project_id=self.project.id,
+        )
+        group1 = event1.group
+        group1.update(last_seen=first_event_date)
+
+        event2 = self.create_event(data={"message": "testing"}, project_id=self.project.id)
+
+        # Mock set the last_seen value to the first event date
+        # To simulate the update to last_seen being buffered
+        event2.group.last_seen = first_event_date
+        event2.group.update(last_seen=first_event_date)
+        assert event2.group_id == group1.id
+
+        mock_callback = Mock()
+        mock_futures = [Mock()]
+
+        mock_processor.return_value.apply.return_value = [(mock_callback, mock_futures)]
+
+        self.call_post_process_group(
+            is_new=False,
+            is_regression=True,
+            is_new_group_environment=False,
+            event=event2,
+        )
+        mock_processor.assert_called_with(
+            EventMatcher(event2, group=group1), False, True, False, False
+        )
+        sent_group_date = mock_processor.call_args[0][0].group.last_seen
+        # Check that last_seen was updated to be at least the new event's date
+        self.assertAlmostEqual(sent_group_date, event2.datetime, delta=timedelta(seconds=10))
+
 
 class ServiceHooksTestMixin(BasePostProgressGroupMixin):
     @patch("sentry.tasks.servicehooks.process_service_hook")