Repository: pandas-dev/pandas

Issue:
Index._engine creates cyclic reference
#### Code Sample, a copy-pastable example if possible

```python
import gc
import pandas as pd

gc.disable()

idx = pd.Int64Index(data=[0])

# trigger a call to idx._engine using public API, but a unit test might also be
# able to use the private API here
idx.is_monotonic_increasing
del idx

remaining_indices = [o for o in gc.get_objects() if isinstance(o, pd.Int64Index)]
assert remaining_indices == []
```
#### Problem description

The (indirect) call to `idx._engine` creates a cyclic reference, therefore indices are not removed without calling the GC:

```python
import objgraph
objgraph.show_backrefs(
    remaining_indices,
    filename='obj.png',
    shortnames=False,
    max_depth=10,
)
```

![obj](https://user-images.githubusercontent.com/1529400/61873731-f6553080-aee6-11e9-8f2f-6e50c881ada5.png)

On certain payloads (e.g. on `dask.distributed` clusters), this might increase memory consumptions significantly.

#### Expected Output
Index is cleared on `del idx` call.

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.6.6.final.0
python-bits      : 64
OS               : Linux
OS-release       : 4.9.125-linuxkit
machine          : x86_64
processor        :
byteorder        : little
LC_ALL           : en_US.UTF-8
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 0.25.0
numpy            : 1.16.4
pytz             : 2019.1
dateutil         : 2.8.0
pip              : 10.0.1
setuptools       : 39.1.0
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : None
IPython          : None
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : None
IPython          : None
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : None
matplotlib       : None
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
s3fs             : None
scipy            : None
sqlalchemy       : None
tables           : None
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None

</details>

Index._engine creates cyclic reference
#### Code Sample, a copy-pastable example if possible

```python
import gc
import pandas as pd

gc.disable()

idx = pd.Int64Index(data=[0])

# trigger a call to idx._engine using public API, but a unit test might also be
# able to use the private API here
idx.is_monotonic_increasing
del idx

remaining_indices = [o for o in gc.get_objects() if isinstance(o, pd.Int64Index)]
assert remaining_indices == []
```
#### Problem description

The (indirect) call to `idx._engine` creates a cyclic reference, therefore indices are not removed without calling the GC:

```python
import objgraph
objgraph.show_backrefs(
    remaining_indices,
    filename='obj.png',
    shortnames=False,
    max_depth=10,
)
```

![obj](https://user-images.githubusercontent.com/1529400/61873731-f6553080-aee6-11e9-8f2f-6e50c881ada5.png)

On certain payloads (e.g. on `dask.distributed` clusters), this might increase memory consumptions significantly.

#### Expected Output
Index is cleared on `del idx` call.

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.6.6.final.0
python-bits      : 64
OS               : Linux
OS-release       : 4.9.125-linuxkit
machine          : x86_64
processor        :
byteorder        : little
LC_ALL           : en_US.UTF-8
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 0.25.0
numpy            : 1.16.4
pytz             : 2019.1
dateutil         : 2.8.0
pip              : 10.0.1
setuptools       : 39.1.0
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : None
IPython          : None
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : None
IPython          : None
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : None
matplotlib       : None
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
s3fs             : None
scipy            : None
sqlalchemy       : None
tables           : None
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None

</details>



Discussion:
cyclic references are very natural in python; have a destructor thru ``__del__`` is not. I suppose you could replace these as weakrefs. If you want to try go for it (though not sure if this would work / pass)
cyclic references are very natural in python; have a destructor thru ``__del__`` is not. I suppose you could replace these as weakrefs. If you want to try go for it (though not sure if this would work / pass)

Resolving patch:
diff --git a/asv_bench/benchmarks/index_object.py b/asv_bench/benchmarks/index_object.py
--- a/asv_bench/benchmarks/index_object.py
+++ b/asv_bench/benchmarks/index_object.py
@@ -1,3 +1,4 @@
+import gc
 import numpy as np
 import pandas.util.testing as tm
 from pandas import (
@@ -225,4 +226,21 @@ def time_intersection_both_duplicate(self, N):
         self.intv.intersection(self.intv2)
 
 
+class GC:
+    params = [1, 2, 5]
+
+    def create_use_drop(self):
+        idx = Index(list(range(1000 * 1000)))
+        idx._engine
+
+    def peakmem_gc_instances(self, N):
+        try:
+            gc.disable()
+
+            for _ in range(N):
+                self.create_use_drop()
+        finally:
+            gc.enable()
+
+
 from .pandas_vb_common import setup  # noqa: F401
diff --git a/pandas/core/indexes/base.py b/pandas/core/indexes/base.py
--- a/pandas/core/indexes/base.py
+++ b/pandas/core/indexes/base.py
@@ -689,7 +689,11 @@ def _cleanup(self):
     @cache_readonly
     def _engine(self):
         # property, for now, slow to look up
-        return self._engine_type(lambda: self._ndarray_values, len(self))
+
+        # to avoid a refernce cycle, bind `_ndarray_values` to a local variable, so
+        # `self` is not passed into the lambda.
+        _ndarray_values = self._ndarray_values
+        return self._engine_type(lambda: _ndarray_values, len(self))
 
     # --------------------------------------------------------------------
     # Array-Like Methods


Test patch:
diff --git a/pandas/tests/indexes/test_base.py b/pandas/tests/indexes/test_base.py
--- a/pandas/tests/indexes/test_base.py
+++ b/pandas/tests/indexes/test_base.py
@@ -1,5 +1,6 @@
 from collections import defaultdict
 from datetime import datetime, timedelta
+import gc
 from io import StringIO
 import math
 import operator
@@ -2424,6 +2425,13 @@ def test_deprecated_contains(self):
             with tm.assert_produces_warning(FutureWarning):
                 index.contains(1)
 
+    def test_engine_reference_cycle(self):
+        # https://github.com/pandas-dev/pandas/issues/27585
+        index = pd.Index([1, 2, 3])
+        nrefs_pre = len(gc.get_referrers(index))
+        index._engine
+        assert len(gc.get_referrers(index)) == nrefs_pre
+
 
 class TestMixedIntIndex(Base):
     # Mostly the tests from common.py for which the results differ