Repository: pydicom/pydicom

Issue:
Memory leaks when accessing sequence tags with Dataset.__getattr__.
**Describe the bug**
Accessing sequences via `Dataset.__getattr__` seems to leak memory. The bug occurred for me when I was processing many DICOMs and manipulating some tags contained in sequences and each leaked a bit of memory, ultimately crashing the process.

**Expected behavior**
Memory should not leak. It works correctly when you replace the `__getattr__` call with `__getitem__` (by manually constructing the necessary tag beforehand).

Without being an expert in the codebase, one difference I think that could explain it is that `__getattr__` sets `value.parent = self` for sequences while `__getitem__` doesn't seem to do that. Maybe this loop of references somehow confuses Python's garbage collection?

**Steps To Reproduce**
This increases the memory consumption of the Python process by about 700 MB on my machine. The DICOM file I've tested it with is 27MB and has one item in `SourceImageSequence`. Note that the memory leak plateaus after a while in this example, maybe because it's the same file. In my actual workflow when iterating over many different files, the process filled all memory and crashed.

```python
import pydicom
for i in range(100):
  dcm = pydicom.dcmread("my_dicom.dcm")
  test = dcm.SourceImageSequence
```

For comparison, this keeps the memory constant. `(0x0008, 0x2112)` is `SourceImageSequence`: 

```python
import pydicom
import pydicom.tag
for i in range(100):
  dcm = pydicom.dcmread("my_dicom.dcm")
  test = dcm[pydicom.tag.TupleTag((0x0008, 0x2112))]
```

**Your environment**

```bash
Linux-4.15.0-72-generic-x86_64-with-Ubuntu-18.04-bionic
Python  3.6.8 (default, Jan 14 2019, 11:02:34)
pydicom  1.3.0
```



Discussion:
I could reproduce this with a smaller sequence (2MB) under Windows. The memory usage got up until it was about twice the initial value (after a few hundred iterations), after that got back down to the initial value and started to rise again.
If removing the code that sets the parent, this does not happen - the memory remains at the initial value.
So, your assessment about the garbage collection delayed due to cyclic references seems to be correct.
I will have a look how to resolve this - probably later this week.
I'm not seeing any memory leakage using [memory-profiler](https://pypi.org/project/memory-profiler/). What are you using to profile?

I get similar results (~38 MB, no sign of gradual increase) for each of the following branches and Python 3.6.5, current `master`:
```python
import time
from pydicom import dcmread
from pydicom.data import get_testdata_file

fname = get_testdata_file("rtplan.dcm")
branch = 'A'

if branch == 'A':
    for ii in range(1):
        ds = dcmread(fname)
elif branch == 'B':
    for ii in range(1):
        ds = dcmread(fname)
        seq = ds.BeamSequence
elif branch == 'C':
    for ii in range(10000):
        ds = dcmread(fname)
elif branch == 'D':
    for ii in range(10000):
        ds = dcmread(fname)
        seq = ds.BeamSequence
elif branch == 'E':
    for ii in range(10000):
        ds = dcmread(fname)
        seq = ds[0x300a, 0x00b0].value

time.sleep(1)
```
It looks like there's a bit of overhead for ds.BeamSequence but not a lot?
@scaramallion - I didn't use a profiler, just watched the private memory consumption. I can add a screenshot if I get to it tonight. A couple of remarks:
- there is no real leak here, as the memory is freed eventually, just not immediately as with the scenario without setting a parent
- the test data you used is quite small - I used a real rtstruct file for the test with a sequence of about 2MB size, so that is more visible in this case
- not setting the parent didn't fail any test, so while I'm sure there was a scenario where this was not needed, there is no tests for it; I have to check if and when this is really needed
- the solution is not complete anyway, as with the access via `__getitem__` the parent is not set

Patch:
diff --git a/pydicom/dataset.py b/pydicom/dataset.py
--- a/pydicom/dataset.py
+++ b/pydicom/dataset.py
@@ -769,13 +769,7 @@ def __getattr__(self, name):
         if tag is not None:  # `name` isn't a DICOM element keyword
             tag = Tag(tag)
             if tag in self._dict:  # DICOM DataElement not in the Dataset
-                data_elem = self[tag]
-                value = data_elem.value
-                if data_elem.VR == 'SQ':
-                    # let a sequence know its parent dataset, as sequence items
-                    # may need parent dataset tags to resolve ambiguous tags
-                    value.parent = self
-                return value
+                return self[tag].value
 
         # no tag or tag not contained in the dataset
         if name == '_dict':
@@ -854,6 +848,10 @@ def __getitem__(self, key):
         data_elem = self._dict[tag]
 
         if isinstance(data_elem, DataElement):
+            if data_elem.VR == 'SQ' and data_elem.value:
+                # let a sequence know its parent dataset, as sequence items
+                # may need parent dataset tags to resolve ambiguous tags
+                data_elem.value.parent = self
             return data_elem
         elif isinstance(data_elem, tuple):
             # If a deferred read, then go get the value now
@@ -1289,7 +1287,7 @@ def convert_pixel_data(self, handler_name=''):
         Raises
         ------
         ValueError
-            If `name` is not a valid handler name.
+            If `handler_name` is not a valid handler name.
         NotImplementedError
             If the given handler or any handler, if none given, is able to
             decompress pixel data with the current transfer syntax
diff --git a/pydicom/filewriter.py b/pydicom/filewriter.py
--- a/pydicom/filewriter.py
+++ b/pydicom/filewriter.py
@@ -70,8 +70,8 @@ def _correct_ambiguous_vr_element(elem, ds, is_little_endian):
         #   For references, see the list at
         #   https://github.com/darcymason/pydicom/pull/298
         # PixelRepresentation is usually set in the root dataset
-        while 'PixelRepresentation' not in ds and ds.parent:
-            ds = ds.parent
+        while 'PixelRepresentation' not in ds and ds.parent and ds.parent():
+            ds = ds.parent()
         # if no pixel data is present, none if these tags is used,
         # so we can just ignore a missing PixelRepresentation in this case
         if ('PixelRepresentation' not in ds and 'PixelData' not in ds or
diff --git a/pydicom/sequence.py b/pydicom/sequence.py
--- a/pydicom/sequence.py
+++ b/pydicom/sequence.py
@@ -3,6 +3,7 @@
 
 Sequence is a list of pydicom Dataset objects.
 """
+import weakref
 
 from pydicom.dataset import Dataset
 from pydicom.multival import MultiValue
@@ -57,7 +58,8 @@ def __init__(self, iterable=None):
 
     @property
     def parent(self):
-        """Return the parent :class:`~pydicom.dataset.Dataset`."""
+        """Return a weak reference to the parent
+        :class:`~pydicom.dataset.Dataset`."""
         return self._parent
 
     @parent.setter
@@ -66,7 +68,7 @@ def parent(self, value):
         :class:`Sequence` items.
         """
         if value != self._parent:
-            self._parent = value
+            self._parent = weakref.ref(value)
             for item in self._list:
                 item.parent = self._parent
 


Test patch:
diff --git a/pydicom/tests/test_filewriter.py b/pydicom/tests/test_filewriter.py
--- a/pydicom/tests/test_filewriter.py
+++ b/pydicom/tests/test_filewriter.py
@@ -875,47 +875,90 @@ def test_write_new_ambiguous(self):
         assert ds[0x00283002].VR == 'US'
         assert ds.LUTDescriptor == [1, 0]
 
-    def test_ambiguous_element_in_sequence_explicit(self):
-        """Test that writing a sequence with an ambiguous element
-        as explicit transfer syntax works."""
-        # regression test for #804
+    def dataset_with_modality_lut_sequence(self, pixel_repr):
         ds = Dataset()
-        ds.PixelRepresentation = 0
+        ds.PixelRepresentation = pixel_repr
         ds.ModalityLUTSequence = [Dataset()]
         ds.ModalityLUTSequence[0].LUTDescriptor = [0, 0, 16]
         ds.ModalityLUTSequence[0].LUTExplanation = None
         ds.ModalityLUTSequence[0].ModalityLUTType = 'US'  # US = unspecified
         ds.ModalityLUTSequence[0].LUTData = b'\x0000\x149a\x1f1c\xc2637'
-
         ds.is_little_endian = True
+        return ds
+
+    def test_ambiguous_element_in_sequence_explicit_using_attribute(self):
+        """Test that writing a sequence with an ambiguous element
+        as explicit transfer syntax works if accessing the tag via keyword."""
+        # regression test for #804
+        ds = self.dataset_with_modality_lut_sequence(pixel_repr=0)
         ds.is_implicit_VR = False
         fp = BytesIO()
         ds.save_as(fp, write_like_original=True)
-
         ds = dcmread(fp, force=True)
         assert 'US' == ds.ModalityLUTSequence[0][0x00283002].VR
 
-    def test_ambiguous_element_in_sequence_implicit(self):
+        ds = self.dataset_with_modality_lut_sequence(pixel_repr=1)
+        ds.is_implicit_VR = False
+        fp = BytesIO()
+        ds.save_as(fp, write_like_original=True)
+        ds = dcmread(fp, force=True)
+        assert 'SS' == ds.ModalityLUTSequence[0][0x00283002].VR
+
+    def test_ambiguous_element_in_sequence_explicit_using_index(self):
+        """Test that writing a sequence with an ambiguous element
+        as explicit transfer syntax works if accessing the tag
+        via the tag number."""
+        ds = self.dataset_with_modality_lut_sequence(pixel_repr=0)
+        ds.is_implicit_VR = False
+        fp = BytesIO()
+        ds.save_as(fp, write_like_original=True)
+        ds = dcmread(fp, force=True)
+        assert 'US' == ds[0x00283000][0][0x00283002].VR
+
+        ds = self.dataset_with_modality_lut_sequence(pixel_repr=1)
+        ds.is_implicit_VR = False
+        fp = BytesIO()
+        ds.save_as(fp, write_like_original=True)
+        ds = dcmread(fp, force=True)
+        assert 'SS' == ds[0x00283000][0][0x00283002].VR
+
+    def test_ambiguous_element_in_sequence_implicit_using_attribute(self):
         """Test that reading a sequence with an ambiguous element
-        from a file with implicit transfer syntax works."""
+        from a file with implicit transfer syntax works if accessing the
+        tag via keyword."""
         # regression test for #804
-        ds = Dataset()
-        ds.PixelRepresentation = 0
-        ds.ModalityLUTSequence = [Dataset()]
-        ds.ModalityLUTSequence[0].LUTDescriptor = [0, 0, 16]
-        ds.ModalityLUTSequence[0].LUTExplanation = None
-        ds.ModalityLUTSequence[0].ModalityLUTType = 'US'  # US = unspecified
-        ds.ModalityLUTSequence[0].LUTData = b'\x0000\x149a\x1f1c\xc2637'
-
-        ds.is_little_endian = True
+        ds = self.dataset_with_modality_lut_sequence(pixel_repr=0)
         ds.is_implicit_VR = True
         fp = BytesIO()
         ds.save_as(fp, write_like_original=True)
         ds = dcmread(fp, force=True)
-        # we first have to access the value to trigger correcting the VR
-        assert 16 == ds.ModalityLUTSequence[0].LUTDescriptor[2]
         assert 'US' == ds.ModalityLUTSequence[0][0x00283002].VR
 
+        ds = self.dataset_with_modality_lut_sequence(pixel_repr=1)
+        ds.is_implicit_VR = True
+        fp = BytesIO()
+        ds.save_as(fp, write_like_original=True)
+        ds = dcmread(fp, force=True)
+        assert 'SS' == ds.ModalityLUTSequence[0][0x00283002].VR
+
+    def test_ambiguous_element_in_sequence_implicit_using_index(self):
+        """Test that reading a sequence with an ambiguous element
+        from a file with implicit transfer syntax works if accessing the tag
+        via the tag number."""
+        ds = self.dataset_with_modality_lut_sequence(pixel_repr=0)
+        ds.is_implicit_VR = True
+        fp = BytesIO()
+        ds.save_as(fp, write_like_original=True)
+        ds = dcmread(fp, force=True)
+        assert 'US' == ds[0x00283000][0][0x00283002].VR
+
+        ds = self.dataset_with_modality_lut_sequence(pixel_repr=1)
+        ds.is_implicit_VR = True
+        fp = BytesIO()
+        ds.save_as(fp, write_like_original=True)
+        ds = dcmread(fp, force=True)
+        assert 'SS' == ds[0x00283000][0][0x00283002].VR
+
 
 class TestCorrectAmbiguousVRElement(object):
     """Test filewriter.correct_ambiguous_vr_element"""