Repository: getsentry/sentry
Pull request: feat(ingest): Add exception metrics to event_manager and post_process_group
Adds exception metrics to be used in Issue Platform upcoming SLOs

Problem statement:
Make ingestor fault tolerant and production ready
Right now we haven't done a comprehensive overview of metrics and error handling for the ingestor (and associated pipeline steps like `post_process` and `eventstream`). We should:
1. Ensure metric consistency in our new ingestor pipeline with our current processing pipeline
2. Make sure that all new code pathways fail gracefully
    a. Error handling where appropriate
    b. Invalid payloads are rejected with helpful error messages and don't cause unhandled internal errors
3. Set up SLO's for existing event and transaction processing with an eye towards occurrence SLO's

Hints from issue discussion:
Need to add more metrics to understand amount of load. Logging is in a decent place
Need to make SLOs for events and transactions as well
https://www.notion.so/sentry/SLO-Update-Q4-FY23-3a100322fcd945528e4d58e89e339379

Commit message: re-raise exception

Patch to review:
diff --git a/src/sentry/tasks/post_process.py b/src/sentry/tasks/post_process.py
index 8d1f959a54bdb6..f1ecc27e21d9b3 100644
--- a/src/sentry/tasks/post_process.py
+++ b/src/sentry/tasks/post_process.py
@@ -448,6 +448,10 @@ def run_post_process_job(job: PostProcessJob):
         try:
             pipeline_step(job)
         except Exception:
+            metrics.incr(
+                "sentry.tasks.post_process.post_process_group.exception",
+                tags={"issue_category": group_event.group.issue_category},
+            )
             logger.exception(
                 f"Failed to process pipeline step {pipeline_step.__name__}",
                 extra={"event": group_event, "group": group_event.group},
diff --git a/src/sentry/tasks/store.py b/src/sentry/tasks/store.py
index f3df570cdb9d3f..f6c446c7c4aee9 100644
--- a/src/sentry/tasks/store.py
+++ b/src/sentry/tasks/store.py
@@ -708,6 +708,9 @@ def _do_save_event(
             if cache_key:
                 with metrics.timer("tasks.store.do_save_event.delete_cache"):
                     processing.event_processing_store.delete_by_key(cache_key)
+        except Exception:
+            metrics.incr("events.save_event.exception", tags={"event_type": event_type})
+            raise
 
         finally:
             reprocessing2.mark_event_reprocessed(data)


Code review comments:
diff_hunk: @@ -448,6 +448,10 @@ def run_post_process_job(job: PostProcessJob):
         try:
             pipeline_step(job)
         except Exception:
+            metrics.incr(
+                "sentry.tasks.post_process.post_process_group.exception",
+                tags={"issue_category": group_event.group.issue_category},
path: src/sentry/tasks/post_process.py
text: @user1:
`issue_category` is an enum. You'll want to convert this to either the int value, or maybe better, use the lower case name of the specific enum type.