Repository: getsentry/sentry-python

Issue:
Accumulating memory usage with 0.10.x
Following on from a discussion in #419, I am seeing increased memory usage over time with version 0.10.x.

In particular, I was hitting out-of-memory conditions with a long-running Celery task that uses the Django ORM.

I used [`tracemalloc`](https://docs.python.org/3/library/tracemalloc.html) to take snapshots at the beginning and end of the task.

Under 0.10.1, here are the top 20 differences between the snapshots taken at the end and start of the task (with it running for four minutes):

```
/venv/lib/python3.7/site-packages/sentry_sdk/utils.py:746: size=33.6 MiB (+33.6 MiB), count=67076 (+67076), average=525 B
/venv/lib/python3.7/site-packages/django/db/models/query.py:199: size=11.4 MiB (+11.3 MiB), count=73026 (+72307), average=164 B
/venv/lib/python3.7/site-packages/django/db/models/sql/compiler.py:472: size=10.7 MiB (+10.7 MiB), count=66784 (+66784), average=168 B
/venv/lib/python3.7/site-packages/django/db/models/query.py:192: size=10.8 MiB (+10.7 MiB), count=67265 (+66702), average=168 B
/venv/lib/python3.7/site-packages/sentry_sdk/tracing.py:123: size=9453 KiB (+9453 KiB), count=67215 (+67215), average=144 B
/venv/lib/python3.7/site-packages/django/db/models/lookups.py:159: size=8197 KiB (+8197 KiB), count=131148 (+131148), average=64 B
/venv/lib/python3.7/site-packages/django/db/models/sql/query.py:307: size=6432 KiB (+6107 KiB), count=3093 (+2840), average=2129 B
/venv/lib/python3.7/site-packages/django/db/models/lookups.py:19: size=4957 KiB (+4948 KiB), count=71804 (+71636), average=71 B
/venv/lib/python3.7/site-packages/django/db/models/sql/query.py:2197: size=4868 KiB (+4863 KiB), count=69230 (+69166), average=72 B
/venv/lib/python3.7/site-packages/django/db/models/expressions.py:159: size=4815 KiB (+4782 KiB), count=68264 (+67877), average=72 B
/venv/lib/python3.7/site-packages/sentry_sdk/hub.py:406: size=4710 KiB (+4710 KiB), count=66977 (+66977), average=72 B
/venv/lib/python3.7/site-packages/sentry_sdk/tracing.py:89: size=4254 KiB (+4254 KiB), count=67019 (+67018), average=65 B
/venv/lib/python3.7/site-packages/sentry_sdk/utils.py:753: size=3669 KiB (+3669 KiB), count=67072 (+67072), average=56 B
/venv/lib/python3.7/site-packages/sentry_sdk/utils.py:194: size=2620 KiB (+2620 KiB), count=67069 (+67069), average=40 B
/venv/lib/python3.7/site-packages/sentry_sdk/tracing.py:99: size=2618 KiB (+2618 KiB), count=67019 (+67018), average=40 B
/venv/lib/python3.7/site-packages/sentry_sdk/tracing.py:186: size=2618 KiB (+2618 KiB), count=67018 (+67018), average=40 B
/usr/lib/python3.7/uuid.py:204: size=2521 KiB (+2521 KiB), count=46067 (+46066), average=56 B
/venv/lib/python3.7/site-packages/sentry_sdk/utils.py:736: size=2097 KiB (+2097 KiB), count=67068 (+67068), average=32 B
/venv/lib/python3.7/site-packages/sentry_sdk/utils.py:723: size=2096 KiB (+2096 KiB), count=67067 (+67067), average=32 B
/venv/lib/python3.7/site-packages/django/db/utils.py:96: size=1738 KiB (+1738 KiB), count=26978 (+26978), average=66 B
```

Under 0.9.5, it looks like this:

```
/venv/lib/python3.7/site-packages/django/db/models/sql/query.py:307: size=6432 KiB (+6107 KiB), count=3086 (+2833), average=2134 B
/usr/lib/python3.7/uuid.py:204: size=2511 KiB (+2511 KiB), count=45919 (+45918), average=56 B
/venv/lib/python3.7/site-packages/django/db/utils.py:96: size=1738 KiB (+1738 KiB), count=26976 (+26976), average=66 B
/venv/lib/python3.7/site-packages/django/db/models/base.py:411: size=1696 KiB (+1696 KiB), count=11438 (+11438), average=152 B
/venv/lib/python3.7/site-packages/psycopg2/extras.py:678: size=1256 KiB (+1256 KiB), count=22969 (+22967), average=56 B
/usr/lib/python3.7/uuid.py:161: size=987 KiB (+987 KiB), count=22966 (+22966), average=44 B
/venv/lib/python3.7/site-packages/django/db/models/sql/datastructures.py:46: size=984 KiB (+886 KiB), count=13368 (+12044), average=75 B
/venv/lib/python3.7/site-packages/django/db/models/sql/query.py:310: size=882 KiB (+840 KiB), count=6025 (+5661), average=150 B
/venv/lib/python3.7/site-packages/django/db/models/query.py:1231: size=870 KiB (+818 KiB), count=9018 (+8493), average=99 B
/venv/lib/python3.7/site-packages/django/db/models/base.py:408: size=750 KiB (+750 KiB), count=7837 (+7837), average=98 B
/venv/lib/python3.7/site-packages/django/db/models/sql/datastructures.py:116: size=754 KiB (+732 KiB), count=8880 (+8509), average=87 B
/venv/lib/python3.7/site-packages/django/db/models/query.py:199: size=747 KiB (+664 KiB), count=6387 (+5668), average=120 B
/venv/lib/python3.7/site-packages/django/db/models/query.py:190: size=717 KiB (+664 KiB), count=6143 (+5666), average=119 B
/venv/lib/python3.7/site-packages/django/db/models/sql/query.py:337: size=702 KiB (+663 KiB), count=5999 (+5661), average=120 B
/venv/lib/python3.7/site-packages/django/db/models/sql/query.py:354: size=652 KiB (+620 KiB), count=2980 (+2832), average=224 B
/venv/lib/python3.7/site-packages/django/db/models/sql/query.py:311: size=645 KiB (+620 KiB), count=2950 (+2832), average=224 B
/venv/lib/python3.7/site-packages/django/db/models/sql/query.py:309: size=618 KiB (+580 KiB), count=3844 (+3537), average=165 B
/venv/lib/python3.7/site-packages/django/utils/deconstruct.py:17: size=762 KiB (+517 KiB), count=8924 (+4472), average=87 B
/venv/lib/python3.7/site-packages/django/db/models/fields/related.py:664: size=555 KiB (+494 KiB), count=9637 (+8572), average=59 B
/venv/lib/python3.7/site-packages/django/db/models/base.py:513: size=479 KiB (+479 KiB), count=7142 (+7142), average=69 B
```

Django is on version 2.2.2, Python on 3.7.3.

Are all the executed SQL statements being kept in memory?


Discussion:
>Are all the executed SQL statements being kept in memory?

That's correct. This is obviously wrong now that I think about it, thanks for raising this! We will have a fix next week.

Resolving patch:
diff --git a/sentry_sdk/hub.py b/sentry_sdk/hub.py
--- a/sentry_sdk/hub.py
+++ b/sentry_sdk/hub.py
@@ -479,6 +479,9 @@ def start_span(
             sample_rate = client and client.options["traces_sample_rate"] or 0
             span.sampled = random.random() < sample_rate
 
+        if span.sampled:
+            span.init_finished_spans()
+
         return span
 
     def finish_span(
@@ -517,7 +520,9 @@ def finish_span(
                 "contexts": {"trace": span.get_trace_context()},
                 "timestamp": span.timestamp,
                 "start_timestamp": span.start_timestamp,
-                "spans": [s.to_json() for s in span._finished_spans if s is not span],
+                "spans": [
+                    s.to_json() for s in (span._finished_spans or ()) if s is not span
+                ],
             }
         )
 
diff --git a/sentry_sdk/tracing.py b/sentry_sdk/tracing.py
--- a/sentry_sdk/tracing.py
+++ b/sentry_sdk/tracing.py
@@ -6,13 +6,14 @@
 
 from sentry_sdk.utils import capture_internal_exceptions, concat_strings
 from sentry_sdk._compat import PY2
+from sentry_sdk._types import MYPY
 
 if PY2:
     from collections import Mapping
 else:
     from collections.abc import Mapping
 
-if False:
+if MYPY:
     import typing
 
     from typing import Optional
@@ -76,15 +77,16 @@ class Span(object):
 
     def __init__(
         self,
-        trace_id=None,
-        span_id=None,
-        parent_span_id=None,
-        same_process_as_parent=True,
-        sampled=None,
-        transaction=None,
-        op=None,
-        description=None,
+        trace_id=None,  # type: Optional[str]
+        span_id=None,  # type: Optional[str]
+        parent_span_id=None,  # type: Optional[str]
+        same_process_as_parent=True,  # type: bool
+        sampled=None,  # type: Optional[bool]
+        transaction=None,  # type: Optional[str]
+        op=None,  # type: Optional[str]
+        description=None,  # type: Optional[str]
     ):
+        # type: (...) -> None
         self.trace_id = trace_id or uuid.uuid4().hex
         self.span_id = span_id or uuid.uuid4().hex[16:]
         self.parent_span_id = parent_span_id
@@ -95,12 +97,16 @@ def __init__(
         self.description = description
         self._tags = {}  # type: Dict[str, str]
         self._data = {}  # type: Dict[str, Any]
-        self._finished_spans = []  # type: List[Span]
+        self._finished_spans = None  # type: Optional[List[Span]]
         self.start_timestamp = datetime.now()
 
         #: End timestamp of span
         self.timestamp = None
 
+    def init_finished_spans(self):
+        if self._finished_spans is None:
+            self._finished_spans = []
+
     def __repr__(self):
         return (
             "<%s(transaction=%r, trace_id=%r, span_id=%r, parent_span_id=%r, sampled=%r)>"
@@ -184,7 +190,8 @@ def set_data(self, key, value):
 
     def finish(self):
         self.timestamp = datetime.now()
-        self._finished_spans.append(self)
+        if self._finished_spans is not None:
+            self._finished_spans.append(self)
 
     def to_json(self):
         return {


Test patch:
diff --git a/tests/test_tracing.py b/tests/test_tracing.py
--- a/tests/test_tracing.py
+++ b/tests/test_tracing.py
@@ -1,3 +1,6 @@
+import weakref
+import gc
+
 import pytest
 
 from sentry_sdk import Hub, capture_message
@@ -93,3 +96,34 @@ def test_sampling_decided_only_for_transactions(sentry_init, capture_events):
 
     with Hub.current.span() as span:
         assert span.sampled is None
+
+
+@pytest.mark.parametrize(
+    "args,expected_refcount",
+    [({"traces_sample_rate": 1.0}, 100), ({"traces_sample_rate": 0.0}, 0)],
+)
+def test_memory_usage(sentry_init, capture_events, args, expected_refcount):
+    sentry_init(**args)
+
+    references = weakref.WeakSet()
+
+    with Hub.current.span(transaction="hi"):
+        for i in range(100):
+            with Hub.current.span(
+                op="helloworld", description="hi {}".format(i)
+            ) as span:
+
+                def foo():
+                    pass
+
+                references.add(foo)
+                span.set_tag("foo", foo)
+                pass
+
+        del foo
+        del span
+
+        # required only for pypy (cpython frees immediately)
+        gc.collect()
+
+        assert len(references) == expected_refcount