Repository: getsentry/sentry-python

Issue:
Transport causes CPU Spikes on Larger Envelopes
### How do you use Sentry?

Sentry Saas (sentry.io)

### Version

1.31

### Steps to Reproduce

Sending this to Sentry causes the transport to create CPU spikes that are very visible in our kubernetes pods.

```python
from sentry_sdk import capture_message, push_scope

for _ in range(50):
    with push_scope() as scope:
        scope.add_attachment(bytes=b"Hello World! " * 10000, filename="attachment.txt")
        capture_message('Hello, hello.')
```

It also happens if gzip compression is disabled. It appears to be related to how `capture_envelope` works within the transport.

### Expected Result

I would not expect the CPU usage to spike.

### Actual Result

It spikes.


Discussion:
possible urllib3 stuff
https://github.com/psf/requests/issues/1910

I did some profiling today: (because sending is done in a worker thread, I profiled them on my local machine because in sentry we sould have not enough samples)

Profling setup:

I used `profiling_sample_rate=1.0` in the sentry_sdk.init() for profiling. Additionally I patched 
I profiled the background workers thread `.run()` method and dump the data on `flush()`: https://github.com/getsentry/sentry-python/pull/2439/files

a) send 10 envelopes with big attachments (10000 hello worlds):
https://sentry-sdks.sentry.io/profiling/profile/sentry-python/6b7fc1ca0e30427cbf9d76ff2e1998dd/flamegraph/

![10_10000](https://github.com/getsentry/sentry-python/assets/202325/14ed7698-ab33-4516-a63e-960ce977f4c6)

b) send 50 envelopes with big attachments (10000 hello worlds):
https://sentry-sdks.sentry.io/profiling/profile/sentry-python/d446df68dd324870a5d0fb416a29e29b/flamegraph/
![50_10000](https://github.com/getsentry/sentry-python/assets/202325/b44aadb6-bec2-40dd-a11d-503a95858190)

c) send 50 envelopes with small attachments (10 hello worlds):
https://sentry-sdks.sentry.io/profiling/profile/sentry-python/ac9a8a1ae4f648a582ce375fe6e1e474/flamegraph/

![50_10](https://github.com/getsentry/sentry-python/assets/202325/a0e989ed-9e82-4ee7-973a-0ac02e9e3cff)


So there is nothing that jumps out. Most of the time is spent in the http lib (which was kind of our guess). I tried to disable SSL certificate verification, but this made not difference at all. 

One thing of not is that sending 10 envelopes takes around 350ms and sending 50 envelopes takes 2500ms (no matter what size the attachments) So that is not linear growth there.

We could create an AsyncHttpTransport (swapping out urllib3 with aiohttp) to see if the result changes drastically. But this is not done in a day.
Another note. I also tracked the cpu utilization of my test process. It maxed out at between 54% (10 small envelopes) 56% (10 large envelopes), and 61% (50 large envelopes). So I could not reproduce a cup utilization spike.

Resolving patch:
diff --git a/sentry_sdk/consts.py b/sentry_sdk/consts.py
--- a/sentry_sdk/consts.py
+++ b/sentry_sdk/consts.py
@@ -43,6 +43,7 @@
             "profiler_mode": Optional[ProfilerMode],
             "otel_powered_performance": Optional[bool],
             "transport_zlib_compression_level": Optional[int],
+            "transport_num_pools": Optional[int],
             "enable_metrics": Optional[bool],
             "before_emit_metric": Optional[Callable[[str, MetricTags], bool]],
         },
diff --git a/sentry_sdk/transport.py b/sentry_sdk/transport.py
--- a/sentry_sdk/transport.py
+++ b/sentry_sdk/transport.py
@@ -157,6 +157,14 @@ def __init__(
         )  # type: DefaultDict[Tuple[str, str], int]
         self._last_client_report_sent = time.time()
 
+        compresslevel = options.get("_experiments", {}).get(
+            "transport_zlib_compression_level"
+        )
+        self._compresslevel = 9 if compresslevel is None else int(compresslevel)
+
+        num_pools = options.get("_experiments", {}).get("transport_num_pools")
+        self._num_pools = 2 if num_pools is None else int(num_pools)
+
         self._pool = self._make_pool(
             self.parsed_dsn,
             http_proxy=options["http_proxy"],
@@ -165,11 +173,6 @@ def __init__(
             proxy_headers=options["proxy_headers"],
         )
 
-        compresslevel = options.get("_experiments", {}).get(
-            "transport_zlib_compression_level"
-        )
-        self._compresslevel = 9 if compresslevel is None else int(compresslevel)
-
         from sentry_sdk import Hub
 
         self.hub_cls = Hub
@@ -439,7 +442,7 @@ def _send_envelope(
     def _get_pool_options(self, ca_certs):
         # type: (Optional[Any]) -> Dict[str, Any]
         return {
-            "num_pools": 2,
+            "num_pools": self._num_pools,
             "cert_reqs": "CERT_REQUIRED",
             "ca_certs": ca_certs or certifi.where(),
         }


Test patch:
diff --git a/tests/test_transport.py b/tests/test_transport.py
--- a/tests/test_transport.py
+++ b/tests/test_transport.py
@@ -132,6 +132,25 @@ def test_transport_works(
     assert any("Sending event" in record.msg for record in caplog.records) == debug
 
 
+@pytest.mark.parametrize(
+    "num_pools,expected_num_pools",
+    (
+        (None, 2),
+        (2, 2),
+        (10, 10),
+    ),
+)
+def test_transport_num_pools(make_client, num_pools, expected_num_pools):
+    _experiments = {}
+    if num_pools is not None:
+        _experiments["transport_num_pools"] = num_pools
+
+    client = make_client(_experiments=_experiments)
+
+    options = client.transport._get_pool_options([])
+    assert options["num_pools"] == expected_num_pools
+
+
 def test_transport_infinite_loop(capturing_server, request, make_client):
     client = make_client(
         debug=True,