Repository: getsentry/sentry
Pull request #91707: ref(span-buffer): Add backpressure
Author: login: untitaker
Labels: name: Scope: Backend
description: Automatically applied to PRs that change backend components
Created: 2025-05-15T11:16:58Z | Merged: 2025-05-20T10:08:25Z

1. If the flusher dies, the consumer will crash too instead of filling up redis further. I believe this is what we recently saw ~13h ago in US: A flusher died because of a kafka blip, but the consumer kept filling up a particular shard.
2. If the flusher is not fast enough and there are too many segments that would be "ready" to flush, we apply backpressure.
3. If redis OOMs, we stop the entire consumer and therefore prevent data from being flushed out prematurely.

I'm a bit concerned that point 3 is not acceptable for oncall since it basically stops the consumer until redis is scaled up (unlike other memory-based backpressure where we eventually recover automatically), but from a product POV it's also bad to flush partial segments prematurely. In practice we'll have to overprovision redis' memory, and set early alerts to scale up redis soon enough.


Diff:
diff --git a/src/sentry/consumers/__init__.py b/src/sentry/consumers/__init__.py
index 4ab89167ccb7..62a69ec2843a 100644
--- a/src/sentry/consumers/__init__.py
+++ b/src/sentry/consumers/__init__.py
@@ -436,6 +436,11 @@ def ingest_transactions_options() -> list[click.Option]:
                 is_flag=True,
                 default=False,
             ),
+            click.Option(
+                ["--max-memory-percentage", "max_memory_percentage"],
+                default=1.0,
+                help="Maximum memory usage of the Redis cluster in % (0.0-1.0) before the consumer backpressures.",
+            ),
             *multiprocessing_options(default_max_batch_size=100),
         ],
     },
diff --git a/src/sentry/processing/backpressure/memory.py b/src/sentry/processing/backpressure/memory.py
index 8b42ca9e1e83..cdba89a00beb 100644
--- a/src/sentry/processing/backpressure/memory.py
+++ b/src/sentry/processing/backpressure/memory.py
@@ -4,6 +4,7 @@
 
 import rb
 import requests
+from redis import StrictRedis
 from rediscluster import RedisCluster
 
 
@@ -47,7 +48,7 @@ def query_rabbitmq_memory_usage(host: str) -> ServiceMemory:
 # Based on configuration, this could be:
 # - a `rediscluster` Cluster (actually `RetryingRedisCluster`)
 # - a `rb.Cluster` (client side routing cluster client)
-Cluster = Union[RedisCluster, rb.Cluster]
+Cluster = Union[RedisCluster, rb.Cluster, StrictRedis]
 
 
 def get_memory_usage(node_id: str, info: Mapping[str, Any]) -> ServiceMemory:
@@ -68,12 +69,14 @@ def get_host_port_info(node_id: str, cluster: Cluster) -> NodeInfo:
             # RedisCluster node mapping
             node = cluster.connection_pool.nodes.nodes.get(node_id)
             return NodeInfo(node["host"], node["port"])
-        else:
+        elif isinstance(cluster, rb.Cluster):
             # rb.Cluster node mapping
             node = cluster.hosts[node_id]
             return NodeInfo(node.host, node.port)
     except Exception:
-        return NodeInfo(None, None)
+        pass
+
+    return NodeInfo(None, None)
 
 
 def iter_cluster_memory_usage(cluster: Cluster) -> Generator[ServiceMemory]:
@@ -83,6 +86,8 @@ def iter_cluster_memory_usage(cluster: Cluster) -> Generator[ServiceMemory]:
     if isinstance(cluster, RedisCluster):
         # `RedisCluster` returns these as a dictionary, with the node-id as key
         cluster_info = cluster.info()
+    elif isinstance(cluster, StrictRedis):
+        cluster_info = {"main": cluster.info()}
     else:
         # rb.Cluster returns a promise with a dictionary with a _local_ node-id as key
         with cluster.all() as client:
diff --git a/src/sentry/spans/buffer.py b/src/sentry/spans/buffer.py
index f289eb58632f..b60ccea8455f 100644
--- a/src/sentry/spans/buffer.py
+++ b/src/sentry/spans/buffer.py
@@ -64,7 +64,7 @@
 from __future__ import annotations
 
 import itertools
-from collections.abc import MutableMapping, Sequence
+from collections.abc import Generator, MutableMapping, Sequence
 from typing import Any, NamedTuple
 
 import rapidjson
@@ -72,6 +72,7 @@
 from django.utils.functional import cached_property
 from sentry_redis_tools.clients import RedisCluster, StrictRedis
 
+from sentry.processing.backpressure.memory import ServiceMemory, iter_cluster_memory_usage
 from sentry.utils import metrics, redis
 
 # SegmentKey is an internal identifier used by the redis buffer that is also
@@ -306,6 +307,27 @@ def _group_by_parent(self, spans: Sequence[Span]) -> dict[tuple[str, str], list[
 
         return trees
 
+    def record_stored_segments(self):
+        with metrics.timer("spans.buffer.get_stored_segments"):
+            with self.client.pipeline(transaction=False) as p:
+                for shard in self.assigned_shards:
+                    key = self._get_queue_key(shard)
+                    p.zcard(key)
+
+                result = p.execute()
+
+        assert len(result) == len(self.assigned_shards)
+
+        for shard_i, queue_size in zip(self.assigned_shards, result):
+            metrics.timing(
+                "spans.buffer.flush_segments.queue_size",
+                queue_size,
+                tags={"shard_i": shard_i},
+            )
+
+    def get_memory_info(self) -> Generator[ServiceMemory]:
+        return iter_cluster_memory_usage(self.client)
+
     def flush_segments(self, now: int, max_segments: int = 0) -> dict[SegmentKey, FlushedSegment]:
         cutoff = now
 
@@ -318,13 +340,11 @@ def flush_segments(self, now: int, max_segments: int = 0) -> dict[SegmentKey, Fl
                     p.zrangebyscore(
                         key, 0, cutoff, start=0 if max_segments else None, num=max_segments or None
                     )
-                    p.zcard(key)
                     queue_keys.append(key)
 
-                result = iter(p.execute())
+                result = p.execute()
 
         segment_keys: list[tuple[QueueKey, SegmentKey]] = []
-        queue_sizes = []
 
         with metrics.timer("spans.buffer.flush_segments.load_segment_data"):
             with self.client.pipeline(transaction=False) as p:
@@ -335,18 +355,8 @@ def flush_segments(self, now: int, max_segments: int = 0) -> dict[SegmentKey, Fl
                         segment_keys.append((queue_key, segment_key))
                         p.smembers(segment_key)
 
-                    # ZCARD output
-                    queue_sizes.append(next(result))
-
                 segments = p.execute()
 
-        for shard_i, queue_size in zip(self.assigned_shards, queue_sizes):
-            metrics.timing(
-                "spans.buffer.flush_segments.queue_size",
-                queue_size,
-                tags={"shard_i": shard_i},
-            )
-
         return_segments = {}
 
         num_has_root_spans = 0
diff --git a/src/sentry/spans/consumers/process/factory.py b/src/sentry/spans/consumers/process/factory.py
index 8eeb86c05f68..a31a019fa5a4 100644
--- a/src/sentry/spans/consumers/process/factory.py
+++ b/src/sentry/spans/consumers/process/factory.py
@@ -37,6 +37,7 @@ def __init__(
         input_block_size: int | None,
         output_block_size: int | None,
         produce_to_pipe: Callable[[KafkaPayload], None] | None = None,
+        max_memory_percentage: float = 1.0,
     ):
         super().__init__()
 
@@ -44,6 +45,7 @@ def __init__(
         self.max_batch_size = max_batch_size
         self.max_batch_time = max_batch_time
         self.max_flush_segments = max_flush_segments
+        self.max_memory_percentage = max_memory_percentage
         self.input_block_size = input_block_size
         self.output_block_size = output_block_size
         self.num_processes = num_processes
@@ -66,8 +68,9 @@ def create_with_partitions(
 
         flusher = self._flusher = SpanFlusher(
             buffer,
-            self.max_flush_segments,
-            self.produce_to_pipe,
+            max_flush_segments=self.max_flush_segments,
+            max_memory_percentage=self.max_memory_percentage,
+            produce_to_pipe=self.produce_to_pipe,
             next_step=committer,
         )
 
@@ -93,19 +96,19 @@ def create_with_partitions(
             next_step=run_task,
         )
 
-        # We use the produce timestamp to drive the clock for flushing, so that
-        # consumer backlogs do not cause segments to be flushed prematurely.
-        # The received timestamp in the span is too old for this purpose if
-        # Relay starts buffering, and we don't want that effect to propagate
-        # into this system.
-        def add_produce_timestamp_cb(message: Message[KafkaPayload]) -> tuple[int, KafkaPayload]:
+        def prepare_message(message: Message[KafkaPayload]) -> tuple[int, KafkaPayload]:
+            # We use the produce timestamp to drive the clock for flushing, so that
+            # consumer backlogs do not cause segments to be flushed prematurely.
+            # The received timestamp in the span is too old for this purpose if
+            # Relay starts buffering, and we don't want that effect to propagate
+            # into this system.
             return (
                 int(message.timestamp.timestamp() if message.timestamp else time.time()),
                 message.payload,
             )
 
         add_timestamp = RunTask(
-            function=add_produce_timestamp_cb,
+            function=prepare_message,
             next_step=batch,
         )
 
@@ -133,7 +136,7 @@ def process_batch(
             parent_span_id=val.get("parent_span_id"),
             project_id=val["project_id"],
             payload=payload.value,
-            is_segment_span=(val.get("parent_span_id") is None or val.get("is_remote")),
+            is_segment_span=bool(val.get("parent_span_id") is None or val.get("is_remote")),
         )
         spans.append(span)
 
diff --git a/src/sentry/spans/consumers/process/flusher.py b/src/sentry/spans/consumers/process/flusher.py
index 7b800f3d2872..a6ebfdc3d66c 100644
--- a/src/sentry/spans/consumers/process/flusher.py
+++ b/src/sentry/spans/consumers/process/flusher.py
@@ -1,12 +1,14 @@
+import logging
 import multiprocessing
 import threading
 import time
 from collections.abc import Callable
 
 import rapidjson
+import sentry_sdk
 from arroyo import Topic as ArroyoTopic
 from arroyo.backends.kafka import KafkaPayload, KafkaProducer, build_kafka_configuration
-from arroyo.processing.strategies.abstract import ProcessingStrategy
+from arroyo.processing.strategies.abstract import MessageRejected, ProcessingStrategy
 from arroyo.types import FilteredPayload, Message
 
 from sentry.conf.types.kafka_definition import Topic
@@ -14,6 +16,10 @@
 from sentry.utils import metrics
 from sentry.utils.kafka_config import get_kafka_producer_cluster_options, get_topic_definition
 
+MAX_PROCESS_RESTARTS = 10
+
+logger = logging.getLogger(__name__)
+
 
 class SpanFlusher(ProcessingStrategy[FilteredPayload | int]):
     """
@@ -33,15 +39,19 @@ def __init__(
         self,
         buffer: SpansBuffer,
         max_flush_segments: int,
+        max_memory_percentage: float,
         produce_to_pipe: Callable[[KafkaPayload], None] | None,
         next_step: ProcessingStrategy[FilteredPayload | int],
     ):
         self.buffer = buffer
         self.max_flush_segments = max_flush_segments
+        self.max_memory_percentage = max_memory_percentage
         self.next_step = next_step
 
         self.stopped = multiprocessing.Value("i", 0)
+        self.redis_was_full = False
         self.current_drift = multiprocessing.Value("i", 0)
+        self.should_backpressure = multiprocessing.Value("i", 0)
 
         from sentry.utils.arroyo import _get_arroyo_subprocess_initializer
 
@@ -59,6 +69,7 @@ def __init__(
                 initializer,
                 self.stopped,
                 self.current_drift,
+                self.should_backpressure,
                 self.buffer,
                 self.max_flush_segments,
                 produce_to_pipe,
@@ -66,6 +77,8 @@ def __init__(
             daemon=True,
         )
 
+        self.process_restarts = 0
+
         self.process.start()
 
     @staticmethod
@@ -73,10 +86,13 @@ def main(
         initializer: Callable | None,
         stopped,
         current_drift,
+        should_backpressure,
         buffer: SpansBuffer,
         max_flush_segments: int,
         produce_to_pipe: Callable[[KafkaPayload], None] | None,
     ) -> None:
+        sentry_sdk.set_tag("sentry_spans_buffer_component", "flusher")
+
         try:
             if initializer:
                 initializer()
@@ -102,6 +118,10 @@ def produce(payload: KafkaPayload) -> None:
                 now = int(time.time()) + current_drift.value
                 flushed_segments = buffer.flush_segments(max_segments=max_flush_segments, now=now)
 
+                should_backpressure.value = len(flushed_segments) >= max_flush_segments * len(
+                    buffer.assigned_shards
+                )
+
                 if not flushed_segments:
                     time.sleep(1)
                     continue
@@ -139,8 +159,61 @@ def poll(self) -> None:
         self.next_step.poll()
 
     def submit(self, message: Message[FilteredPayload | int]) -> None:
+        # Note that submit is not actually a hot path. Their message payloads
+        # are mapped from *batches* of spans, and there are a handful of spans
+        # per second at most. If anything, self.poll() might even be called
+        # more often than submit()
+        if not self.process.is_alive():
+            metrics.incr("sentry.spans.buffer.flusher_dead")
+            if self.process_restarts < MAX_PROCESS_RESTARTS:
+                self.process.start()
+                self.process_restarts += 1
+            else:
+                raise RuntimeError(
+                    "flusher process has crashed.\n\nSearch for sentry_spans_buffer_component:flusher in Sentry to get the original error."
+                )
+
+        self.buffer.record_stored_segments()
+
+        # We pause insertion into Redis if the flusher is not making progress
+        # fast enough. We could backlog into Redis, but we assume, despite best
+        # efforts, it is still always going to be less durable than Kafka.
+        # Minimizing our Redis memory usage also makes COGS easier to reason
+        # about.
+        #
+        # should_backpressure is true if there are many segments to flush, but
+        # the flusher can't get all of them out.
+        if self.should_backpressure.value:
+            metrics.incr("sentry.spans.buffer.flusher.backpressure")
+            raise MessageRejected()
+
+        # We set the drift. The backpressure based on redis memory comes after.
+        # If Redis is full for a long time, the drift will grow into a large
+        # negative value, effectively pausing flushing as well.
         if isinstance(message.payload, int):
-            self.current_drift.value = message.payload - int(time.time())
+            self.current_drift.value = drift = message.payload - int(time.time())
+            metrics.timing("sentry.spans.buffer.flusher.drift", drift)
+
+        # We also pause insertion into Redis if Redis is too full. In this case
+        # we cannot allow the flusher to progress either, as it would write
+        # partial/fragmented segments to buffered-segments topic. We have to
+        # wait until the situation is improved manually.
+        if self.max_memory_percentage < 1.0:
+            memory_infos = list(self.buffer.get_memory_info())
+            used = sum(x.used for x in memory_infos)
+            available = sum(x.available for x in memory_infos)
+            if available > 0 and used / available > self.max_memory_percentage:
+                if not self.redis_was_full:
+                    logger.fatal("Pausing consumer due to Redis being full")
+                metrics.incr("sentry.spans.buffer.flusher.hard_backpressure")
+                self.redis_was_full = True
+                # Pause consumer if Redis memory is full. Because the drift is
+                # set before we emit backpressure, the flusher effectively
+                # stops as well. Alternatively we may simply crash the consumer
+                # but this would also trigger a lot of rebalancing.
+                raise MessageRejected()
+
+        self.redis_was_full = False
         self.next_step.submit(message)
 
     def terminate(self) -> None:
diff --git a/tests/sentry/spans/consumers/process/test_consumer.py b/tests/sentry/spans/consumers/process/test_consumer.py
index 07d2a613898a..fbf8fdc60637 100644
--- a/tests/sentry/spans/consumers/process/test_consumer.py
+++ b/tests/sentry/spans/consumers/process/test_consumer.py
@@ -1,4 +1,3 @@
-import threading
 from datetime import datetime
 
 import rapidjson
@@ -8,16 +7,7 @@
 from sentry.spans.consumers.process.factory import ProcessSpansStrategyFactory
 
 
-class FakeProcess(threading.Thread):
-    """
-    Pretend this is multiprocessing.Process
-    """
-
-    def terminate(self):
-        pass
-
-
-def test_basic(monkeypatch, request):
+def test_basic(monkeypatch):
     # Flush very aggressively to make test pass instantly
     monkeypatch.setattr("time.sleep", lambda _: None)
 
@@ -61,11 +51,6 @@ def add_commit(offsets, force=False):
         )
     )
 
-    @request.addfinalizer
-    def _():
-        step.join()
-        fac.shutdown()
-
     step.poll()
     fac._flusher.current_drift.value = 9000  # "advance" our "clock"
 
diff --git a/tests/sentry/spans/consumers/process/test_flusher.py b/tests/sentry/spans/consumers/process/test_flusher.py
new file mode 100644
index 000000000000..4654865f8284
--- /dev/null
+++ b/tests/sentry/spans/consumers/process/test_flusher.py
@@ -0,0 +1,83 @@
+import time
+from time import sleep
+
+import rapidjson
+from arroyo.processing.strategies.noop import Noop
+
+from sentry.spans.buffer import Span, SpansBuffer
+from sentry.spans.consumers.process.flusher import SpanFlusher
+
+
+def _payload(span_id: bytes) -> bytes:
+    return rapidjson.dumps({"span_id": span_id}).encode("ascii")
+
+
+def test_backpressure(monkeypatch):
+    # Flush very aggressively to make join() faster
+    monkeypatch.setattr("time.sleep", lambda _: None)
+
+    buffer = SpansBuffer(assigned_shards=list(range(1)))
+
+    messages = []
+
+    def append(msg):
+        messages.append(msg)
+        sleep(1.0)
+
+    flusher = SpanFlusher(
+        buffer,
+        max_flush_segments=1,
+        max_memory_percentage=1.0,
+        produce_to_pipe=append,
+        next_step=Noop(),
+    )
+
+    now = time.time()
+
+    for i in range(200):
+        trace_id = f"{i:0>32x}"
+
+        spans = [
+            Span(
+                payload=_payload(b"a" * 16),
+                trace_id=trace_id,
+                span_id="a" * 16,
+                parent_span_id="b" * 16,
+                project_id=1,
+            ),
+            Span(
+                payload=_payload(b"d" * 16),
+                trace_id=trace_id,
+                span_id="d" * 16,
+                parent_span_id="b" * 16,
+                project_id=1,
+            ),
+            Span(
+                payload=_payload(b"c" * 16),
+                trace_id=trace_id,
+                span_id="c" * 16,
+                parent_span_id="b" * 16,
+                project_id=1,
+            ),
+            Span(
+                payload=_payload(b"b" * 16),
+                trace_id=trace_id,
+                span_id="b" * 16,
+                parent_span_id=None,
+                is_segment_span=True,
+                project_id=1,
+            ),
+        ]
+
+        buffer.process_spans(spans, now=int(now))
+
+    # Advance drift to trigger idle timeout of all segments. The flusher should
+    # have way too much to do due to `max_flush_segments=1` and enter
+    # backpressure state.
+
+    flusher.current_drift.value = 20000
+    sleep(0.1)
+
+    assert messages
+
+    assert flusher.should_backpressure.value
diff --git a/tests/sentry/spans/test_buffer.py b/tests/sentry/spans/test_buffer.py
index e7cf39043f2f..a45347d1b13d 100644
--- a/tests/sentry/spans/test_buffer.py
+++ b/tests/sentry/spans/test_buffer.py
@@ -172,6 +172,8 @@ def test_basic(buffer: SpansBuffer, spans):
     buffer.done_flush_segments(rv)
     assert buffer.flush_segments(now=30) == {}
 
+    assert list(buffer.get_memory_info())
+
     assert_clean(buffer.client)
 
 


Reviews:
user: login: fpacifici
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T17:55:36Z
user: login: untitaker
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-19T14:43:57Z
user: login: jan-auer
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-20T09:14:18Z

Review comments:
diff_hunk: @@ -306,6 +306,26 @@ def _group_by_parent(self, spans: Sequence[Span]) -> dict[tuple[str, str], list[
 
         return trees
 
+    def get_current_queue_size(self) -> int:
+        with metrics.timer("spans.buffer.get_current_queue_size"):
+            with self.client.pipeline(transaction=False) as p:
+                for shard in self.assigned_shards:
+                    key = self._get_queue_key(shard)
+                    p.zcard(key)
path: src/sentry/spans/buffer.py
user: login: fpacifici
body: Please consider measuring this in bytes rather than set cardinality as spans can be very different from each other. 

A way to do this is to keep a sharded counter of the size of what you write, when you add a batch of spans you increment, when you flush you reduce. Then you can rely on the amount of data in redis rather than the number of spans.

Or, better, measure the amount of free memory in redis like backpressure does. Then you cannot go wrong with the signal to trigger backpressure.
created_at: 2025-05-16T17:52:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -306,6 +306,26 @@ def _group_by_parent(self, spans: Sequence[Span]) -> dict[tuple[str, str], list[
 
         return trees
 
+    def get_current_queue_size(self) -> int:
+        with metrics.timer("spans.buffer.get_current_queue_size"):
+            with self.client.pipeline(transaction=False) as p:
+                for shard in self.assigned_shards:
+                    key = self._get_queue_key(shard)
+                    p.zcard(key)
path: src/sentry/spans/buffer.py
user: login: untitaker
body: done. also had a conversation with jan and had to split up into two kinds of backpressure. see updated pr description
created_at: 2025-05-19T14:43:57Z
author_association: MEMBER
diff_hunk: @@ -139,8 +155,41 @@ def poll(self) -> None:
         self.next_step.poll()
 
     def submit(self, message: Message[FilteredPayload | int]) -> None:
+        # Note that submit is not actually a hot path. Their message payloads
+        # are mapped from *batches* of spans, and there are a handful of spans
+        # per second at most. If anything, self.poll() might even be called
+        # more often than submit()
+        if not self.process.is_alive():
+            metrics.incr("sentry.spans.buffer.flusher_dead")
+            if self.process_restarts < MAX_PROCESS_RESTARTS:
+                self.process.start()
+                self.process_restarts += 1
+            else:
+                raise RuntimeError("flusher process is dead")
+
+        # Execute this to record metrics.
+        self.buffer.get_stored_segments()
+
+        if self.should_backpressure.value:
+            metrics.incr("sentry.spans.buffer.flusher.backpressure")
+            raise MessageRejected()
+
         if isinstance(message.payload, int):
             self.current_drift.value = message.payload - int(time.time())
+
+        if self.max_memory_percentage < 1.0:
+            memory_infos = list(self.buffer.get_memory_info())
+            used = sum(x.used for x in memory_infos)
+            available = sum(x.available for x in memory_infos)
+            if available > 0 and used / available > self.max_memory_percentage:
+                logger.fatal("Pausing consumer due to Redis being full")
path: src/sentry/spans/consumers/process/flusher.py
user: login: jan-auer
body: Let's debounce this so we don't spam Sentry with too many errors
created_at: 2025-05-20T09:09:00Z
author_association: MEMBER
diff_hunk: @@ -139,8 +155,41 @@ def poll(self) -> None:
         self.next_step.poll()
 
     def submit(self, message: Message[FilteredPayload | int]) -> None:
+        # Note that submit is not actually a hot path. Their message payloads
+        # are mapped from *batches* of spans, and there are a handful of spans
+        # per second at most. If anything, self.poll() might even be called
+        # more often than submit()
+        if not self.process.is_alive():
+            metrics.incr("sentry.spans.buffer.flusher_dead")
+            if self.process_restarts < MAX_PROCESS_RESTARTS:
+                self.process.start()
+                self.process_restarts += 1
+            else:
+                raise RuntimeError("flusher process is dead")
path: src/sentry/spans/consumers/process/flusher.py
user: login: jan-auer
body: ```suggestion
                raise RuntimeError("flusher process has crashed")
```

Ideally we would have the inner error linked here, which is not trivial. We can note in the runbook to check for errors from the flusher process. Can we initialize the SDK with a special tag within the flusher process so this gets easier?
created_at: 2025-05-20T09:10:29Z
author_association: MEMBER
diff_hunk: @@ -306,6 +307,29 @@ def _group_by_parent(self, spans: Sequence[Span]) -> dict[tuple[str, str], list[
 
         return trees
 
+    def get_stored_segments(self) -> int:
path: src/sentry/spans/buffer.py
user: login: jan-auer
body: The return value of this is no longer needed and we use this just to record metrics. Let's rename the function accordingly.
created_at: 2025-05-20T09:12:00Z
author_association: MEMBER
diff_hunk: @@ -139,8 +155,41 @@ def poll(self) -> None:
         self.next_step.poll()
 
     def submit(self, message: Message[FilteredPayload | int]) -> None:
path: src/sentry/spans/consumers/process/flusher.py
user: login: jan-auer
body: Please add a short docstring that shows the two conditions we return with back pressure. Let's also document in which of the two cases the flusher itself will stop and why.
created_at: 2025-05-20T09:13:52Z
author_association: MEMBER

PR comments:
user: login: codecov[bot]
created_at: 2025-05-15T11:33:52Z
body: ## [Codecov](https://app.codecov.io/gh/getsentry/sentry/pull/91707?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getsentry) Report
Attention: Patch coverage is `80.00000%` with `17 lines` in your changes missing coverage. Please review.

### :warning: Parser warning

> The parser emitted a warning. Please review your JUnit XML file:
> <pre><code class="language-python">Warning while parsing testcase attributes: Limit of string is 1000 chars, for name, we got 2083 at 1:157236 in /home/runner/work/sentry/sentry/.artifacts/pytest.junit.xml</code></pre>


| [Files with missing lines](https://app.codecov.io/gh/getsentry/sentry/pull/91707?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getsentry) | Patch % | Lines |
|---|---|---|
| [src/sentry/spans/consumers/process/flusher.py](https://app.codecov.io/gh/getsentry/sentry/pull/91707?src=pr&el=tree&filepath=src%2Fsentry%2Fspans%2Fconsumers%2Fprocess%2Fflusher.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getsentry#diff-c3JjL3NlbnRyeS9zcGFucy9jb25zdW1lcnMvcHJvY2Vzcy9mbHVzaGVyLnB5) | 52.94% | [16 Missing :warning: ](https://app.codecov.io/gh/getsentry/sentry/pull/91707?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getsentry) |
| [src/sentry/processing/backpressure/memory.py](https://app.codecov.io/gh/getsentry/sentry/pull/91707?src=pr&el=tree&filepath=src%2Fsentry%2Fprocessing%2Fbackpressure%2Fmemory.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getsentry#diff-c3JjL3NlbnRyeS9wcm9jZXNzaW5nL2JhY2twcmVzc3VyZS9tZW1vcnkucHk=) | 85.71% | [1 Missing :warning: ](https://app.codecov.io/gh/getsentry/sentry/pull/91707?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getsentry) |

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##           master   #91707      +/-   ##
==========================================
+ Coverage   87.62%   87.67%   +0.04%     
==========================================
  Files       10353    10342      -11     
  Lines      586779   586792      +13     
  Branches    22585    22528      -57     
==========================================
+ Hits       514194   514457     +263     
+ Misses      72157    71892     -265     
- Partials      428      443      +15     
```

</details>
author_association: NONE
user: login: sentry[bot]
created_at: 2025-05-20T16:13:16Z
body: ## Suspect Issues
This pull request was deployed and Sentry observed the following issues:

- ‼️ **AssertionError: cannot start a process twice** `sentry.spans.consumers.process.flusher in submit` [View Issue](https://sentry.sentry.io/issues/6618650414/?referrer=github-pr-bot)
- ‼️ **AssertionError: cannot start a process twice** `sentry.spans.consumers.process.flusher in submit` [View Issue](https://sentry.sentry.io/issues/6618991478/?referrer=github-pr-bot)
- ‼️ **AssertionError** `sentry.utils.kafka in run_processor_with_signals` [View Issue](https://sentry.sentry.io/issues/6619102266/?referrer=github-pr-bot)
- ‼️ **KafkaException: KafkaError{code=_DESTROY,val=-197,str="Failed to get committed offsets: Local: Broker handle destroyed"}** `sentry.utils.kafka in run_processor_with_signals` [View Issue](https://sentry.sentry.io/issues/6619248419/?referrer=github-pr-bot)

<sub>Did you find this useful? React with a 👍 or 👎</sub>
author_association: CONTRIBUTOR