Repository: dask/distributed

Issue:
Fail to garbage collect Pandas dataframes
Sometimes we fail to clean up Pandas dataframes.  Here are a couple of examples using dask.dataframe and dask.array on a "cluster" in the local environment.  In the dask.array case we cleanly remove used data.  In the dask.dataframe case the memory of our process remains high even after we delete things.

### Clean case with dask.array

```python
In [1]: import psutil

In [2]: proc = psutil.Process()

In [3]: proc.memory_info().rss / 1e6
Out[3]: 39.768064

In [4]: from distributed import Client, LocalCluster

In [5]: cluster = LocalCluster(nanny=False, n_workers=4, threads_per_worker=1)

In [6]: client = Client(cluster)

In [7]: bokeh.command.util - CRITICAL - Cannot start Bokeh server, port 8787 is already in use
In [7]: 

In [7]: import dask.array as da

In [8]: x = da.random.random(100000000, chunks=(1000000,)).persist()

In [9]: proc.memory_info().rss / 1e6
Out[9]: 872.411136

In [10]: proc.memory_info().rss / 1e6
Out[10]: 872.411136

In [11]: proc.memory_info().rss / 1e6
Out[11]: 872.411136

In [12]: del x

In [13]: proc.memory_info().rss / 1e6
Out[13]: 72.081408

In [14]: proc.memory_info().rss / 1e6
Out[14]: 72.081408
```

### Leaking case with Pandas dataframes

```python
In [1]: import psutil

In [2]: proc = psutil.Process()

In [3]: proc.memory_info().rss / 1e6
Out[3]: 38.109184

In [4]: proc.memory_info().rss / 1e6
Out[4]: 38.37952

In [5]: from distributed import Client, LocalCluster

In [6]: proc.memory_info().rss / 1e6
Out[6]: 51.851264

In [7]: cluster = LocalCluster(nanny=False, n_workers=4, threads_per_worker=1)

In [8]: client = Client(cluster)bokeh.command.util - CRITICAL - Cannot start Boe
In [8]: client = Client(cluster)

In [9]: proc.memory_info().rss / 1e6
Out[9]: 65.073152

In [10]: import dask.dataframe as dd

In [11]: df = dd.demo.daily_stock('GOOG', '2008', '2010', freq='1s', 
    ...:                          random_state=1234).persist()

In [12]: proc.memory_info().rss / 1e6
Out[12]: 584.060928

In [13]: proc.memory_info().rss / 1e6
Out[13]: 685.867008

In [14]: proc.memory_info().rss / 1e6
Out[14]: 804.74112

In [15]: proc.memory_info().rss / 1e6
Out[15]: 927.670272

In [16]: proc.memory_info().rss / 1e6
Out[16]: 996.499456

In [17]: proc.memory_info().rss / 1e6
Out[17]: 997.445632

In [18]: proc.memory_info().rss / 1e6
Out[18]: 997.445632

In [19]: del df

In [20]: cluster.scheduler.tasks
Out[20]: {}

In [21]: proc.memory_info().rss / 1e6
Out[21]: 997.711872
```

cc @pitrou and @jreback any thoughts on how to track this down or avoid it?


Discussion:
@jreback are there any known problems with memory leaks in Pandas?

@pitrou do you have any thoughts on how to identify where this leak occurs?
A word of warning: `rss` tells you the size currently allocated by the kernel, but this could be memory which has been freed by the application while being held by the libc (a quick look at Pandas suggests it uses C `malloc()`). Try allocating a new dataframe of the same size and see whether memory size grows or not.

As a sidenote, `dd.demo.daily_stock` fails here. What is the `pandas_datareader` module?
```
>>> df = dd.demo.daily_stock('GOOG', '2008', '2010', freq='1s', random_state=1234).persist()
Traceback (most recent call last):
  File "<ipython-input-21-f70de842f2db>", line 1, in <module>
    df = dd.demo.daily_stock('GOOG', '2008', '2010', freq='1s', random_state=1234).persist()
  File "/home/antoine/dask/dask/dataframe/io/demo.py", line 186, in daily_stock
    from pandas_datareader import data
ModuleNotFoundError: No module named 'pandas_datareader'
```
```
In [1]: import psutil

In [2]: proc = psutil.Process()

In [3]: proc.memory_info().rss / 1e6
Out[3]: 74.919936

In [4]: proc.memory_info().rss / 1e6
Out[4]: 74.981376

In [5]: df = pd.DataFrame(np.random.randn(100000, 10))

In [6]: proc.memory_info().rss / 1e6
Out[6]: 83.06688

In [7]: proc.memory_info().rss / 1e6
Out[7]: 83.070976

In [8]: del df

In [9]: proc.memory_info().rss / 1e6
Out[9]: 83.095552

In [10]: df = pd.DataFrame(np.random.randn(100000, 10))

In [11]: proc.memory_info().rss / 1e6
Out[11]: 83.140608
```

```
In [1]: import psutil

In [2]: proc = psutil.Process()

In [3]: proc.memory_info().rss / 1e6
Out[3]: 74.661888

In [4]: arr = np.random.randn(100000, 10)

In [5]: proc.memory_info().rss / 1e6
Out[5]: 82.767872

In [6]: del arr

In [7]: proc.memory_info().rss / 1e6
Out[7]: 82.788352

In [8]: arr = np.random.randn(100000, 10)

In [9]: proc.memory_info().rss / 1e6
Out[9]: 82.849792

In [10]: del arr

In [11]: proc.memory_info().rss / 1e6
Out[11]: 82.870272
```
you need to install ``pandas-datareader`` (pip/conda)
Scratching the two last messages -- apparently this was because of IPython, not because of the Dask dataframe type (could reproduce with simply `object()`).
Note that `distributed` is out of the picture. With no distributed client running:
```
>>> import dask.dataframe as dd
>>> df = dd.demo.daily_stock('GOOG', '2008', '2010', freq='1s', random_state=1235).persist()
>>> import psutil
>>> import weakref, sys, gc
>>> wr = weakref.ref(df)
>>> sys.getrefcount(df)
2
>>> del df
>>> wr() is None
True
>>> psutil.Process().memory_info().rss / 1e6
991.62112
```
So it seems that deleting a Dask dataframe can leave a small number of underlying dataframes around (3 here):
```
>>> import dask.dataframe as dd
>>> import pandas as pd
>>> df = dd.demo.daily_stock('GOOG', '2008', '2010', freq='1h', random_state=1235).persist()
>>> 
>>> len(df.dask)
505
>>> len([x for x in df.dask.values() if isinstance(x, pd.DataFrame)])
505
>>> frames = [x for x in df.dask.values() if isinstance(x, pd.DataFrame)]
>>> len(frames)
505
>>> import weakref
>>> frames = [weakref.ref(x) for x in df.dask.values() if isinstance(x, pd.DataFrame)]
>>> len(frames)
505
>>> len([f for f in frames if f() is not None])
505
>>> wr = weakref.ref(df)
>>> del df
>>> wr() is not None
False
>>> len([f for f in frames if f() is not None])
3
```

Let's try to see what those frames are. They seem a bit random:
```
>>> a, b, c = [f() for f in frames if f() is not None]
>>> a.head()

                       close     high      low     open
timestamp                                              
2008-05-15 09:00:00  577.799  579.489  576.904  579.001
2008-05-15 10:00:00  579.922  579.922  575.611  575.611
2008-05-15 11:00:00  582.927  582.951  581.008  581.008
2008-05-15 12:00:00  582.438  582.438  581.623  582.312
2008-05-15 13:00:00  581.126  582.190  581.126  581.651
>>> b.head()

                       close     high      low     open
timestamp                                              
2008-05-22 09:00:00  554.211  554.211  551.951  551.951
2008-05-22 10:00:00  553.099  553.654  552.542  553.654
2008-05-22 11:00:00  552.344  554.152  552.344  554.152
2008-05-22 12:00:00  553.075  553.973  552.619  553.194
2008-05-22 13:00:00  546.648  552.450  546.648  552.450
>>> c.head()

                       close     high      low     open
timestamp                                              
2008-05-23 09:00:00  545.999  548.058  545.427  546.961
2008-05-23 10:00:00  546.713  546.713  541.679  545.211
2008-05-23 11:00:00  546.575  546.575  545.145  545.145
2008-05-23 12:00:00  552.030  552.030  547.238  548.377
2008-05-23 13:00:00  548.420  553.001  548.420  550.993
```

What is referring to the first dangling frame:
```
>>> [(type(r), r) for r in gc.get_referrers(a) if r is not globals()]

[(tuple, (                       close     high      low     open
   timestamp                                              
   2008-05-15 09:00:00  577.799  579.489  576.904  579.001
   2008-05-15 10:00:00  579.922  579.922  575.611  575.611
   2008-05-15 11:00:00  582.927  582.951  581.008  581.008
   2008-05-15 12:00:00  582.438  582.438  581.623  582.312
   2008-05-15 13:00:00  581.126  582.190  581.126  581.651
   2008-05-15 14:00:00  581.065  581.417  580.007  581.417
   2008-05-15 15:00:00  580.856  581.015  579.504  580.427
   2008-05-15 16:00:00  581.001  581.001  581.001  581.001,
   None,
   140221674002176)),
 (dict,
  {'name': 'loc',
   'ndim': 2,
   'obj':                        close     high      low     open
   timestamp                                              
   2008-05-15 09:00:00  577.799  579.489  576.904  579.001
   2008-05-15 10:00:00  579.922  579.922  575.611  575.611
   2008-05-15 11:00:00  582.927  582.951  581.008  581.008
   2008-05-15 12:00:00  582.438  582.438  581.623  582.312
   2008-05-15 13:00:00  581.126  582.190  581.126  581.651
   2008-05-15 14:00:00  581.065  581.417  580.007  581.417
   2008-05-15 15:00:00  580.856  581.015  579.504  580.427
   2008-05-15 16:00:00  581.001  581.001  581.001  581.001}),
 (dict,
  {'name': 'iloc',
   'ndim': 2,
   'obj':                        close     high      low     open
   timestamp                                              
   2008-05-15 09:00:00  577.799  579.489  576.904  579.001
   2008-05-15 10:00:00  579.922  579.922  575.611  575.611
   2008-05-15 11:00:00  582.927  582.951  581.008  581.008
   2008-05-15 12:00:00  582.438  582.438  581.623  582.312
   2008-05-15 13:00:00  581.126  582.190  581.126  581.651
   2008-05-15 14:00:00  581.065  581.417  580.007  581.417
   2008-05-15 15:00:00  580.856  581.015  579.504  580.427
   2008-05-15 16:00:00  581.001  581.001  581.001  581.001})]
```
Similar results can be had for the other dangling frames: they are referred to by a `(frame, None, integer timestamp)` tuple and two `{'name': 'loc' or 'iloc', 'ndim': 2, 'obj': frame}` dict.

At this point I don't know where to go next, as I don't know anything about Pandas internals. The `loc` and `iloc` things seem to suggest there may be some caching going on?
``.iloc/.loc`` are the pandas indexers, they keep a reference to the object (I don't think you need to explicity break this cycle as the gc would normally), but maybe here you do.
```
In [1]: df = pd.DataFrame(np.random.randn(10,2))

In [2]: df.iloc[0:5]
Out[2]: 
          0         1
0  0.060184  1.252374
1 -0.017412  0.741759
2  0.462292  0.092218
3  0.064403  0.790817
4  1.481308  0.962824

In [3]: df.iloc
Out[3]: <pandas.core.indexing._iLocIndexer at 0x10e748b70>

In [4]: df.iloc.obj
Out[4]: 
          0         1
0  0.060184  1.252374
1 -0.017412  0.741759
2  0.462292  0.092218
3  0.064403  0.790817
4  1.481308  0.962824
5 -0.826619  0.980654
6 -1.133114 -0.504807
7 -0.017072 -1.646783
8  0.928021 -0.175613
9 -0.332223  2.219120
```
Oh, so you mean one needs to call `gc.collect()` (or wait for the GC to run automatically) in order for a Pandas dataframe to be *actually* collected:
```
>>> df = pd.DataFrame(np.random.randn(10,2))
>>> df.head()

          0         1
0 -0.390915 -0.197607
1 -1.096394 -0.347774
2 -0.724065  1.348458
3 -0.803112  1.071055
4  0.116906 -1.077839
>>> df.loc[:5].head()

          0         1
0 -0.390915 -0.197607
1 -1.096394 -0.347774
2 -0.724065  1.348458
3 -0.803112  1.071055
4  0.116906 -1.077839
>>> df.iloc[:5].head()

          0         1
0 -0.390915 -0.197607
1 -1.096394 -0.347774
2 -0.724065  1.348458
3 -0.803112  1.071055
4  0.116906 -1.077839
>>> import gc, sys, weakref
>>> sys.getrefcount(df)
4
>>> wr = weakref.ref(df)
>>> del df
>>> wr() is None
False
>>> gc.collect()
47
>>> wr() is None
True
```

I understand why some workers may make it look like memory hasn't been reclaimed after "deleting" a Dask dataframe and its underlying dataframes: all the underlying dataframes are not necessarily reclaimed up front, and depending on the allocator's heuristics, no memory at all may be returned to the OS.
> Oh, so you mean one needs to call gc.collect() (or wait for the GC to run automatically) in order for a Pandas dataframe to be actually collected:

those are the semantics of a cyclical ref, so yes. There are indexing allocations that are potentially done (e.g. not in all circumstances are these required), that may allocate c-structures. These have there own ``__dealloc__`` routines. so a collection is necessary. So you cannot guarantee *when* these are deleted. (again semantics of ``__del__``). 


> those are the semantics of a cyclical ref, so yes

Wouldn't it be better to only keep a weak reference? Users don't necessarily expect allocated data to linger around, especially for large objects such as dataframes.
yes it would

(this was done a long time ago though)

if u want to do a PR would be welcome!
Regardless, even with `gc.collect()`, some frames are still lying around. They seem to hang on some Dask structures.
I created an issue on the pandas side. But I don't think this particular issue a pandas issue per-se, rather something in dask is holding onto a reference (and thus the object is not actually completely deleted).
Ok, so in the pure Dask case (without Distributed), it is an issue with `multiprocessing.Pool` not releasing the references to the latest task(s). Distributed uses `concurrent.futures`, so a separate investigation will be needed.
`multiprocessing` bug reported upstream at http://bugs.python.org/issue29861
Ok, so in the Distributed case, this is an issue inside Distributed (in addition to the Pandas-specific `iloc` issue). I will submit a PR soon.

Resolving patch:
diff --git a/distributed/core.py b/distributed/core.py
--- a/distributed/core.py
+++ b/distributed/core.py
@@ -266,6 +266,7 @@ def handle_comm(self, comm, shutting_down=shutting_down):
                         logger.warn("Lost connection to %r while sending result for op %r: %s",
                                     address, op, e)
                         break
+                msg = result = None
                 if close_desired:
                     yield comm.close()
                 if comm.closed():


Test patch:
diff --git a/distributed/tests/test_core.py b/distributed/tests/test_core.py
--- a/distributed/tests/test_core.py
+++ b/distributed/tests/test_core.py
@@ -4,14 +4,17 @@
 from functools import partial
 import os
 import socket
+import weakref
 
 from tornado import gen, ioloop
 import pytest
 
+from distributed.compatibility import finalize
 from distributed.core import (
     pingpong, Server, rpc, connect, send_recv,
     coerce_to_address, ConnectionPool, CommClosedError)
 from distributed.metrics import time
+from distributed.protocol import to_serialize
 from distributed.utils import get_ip, get_ipv6
 from distributed.utils_test import (
     slow, loop, gen_test, gen_cluster, has_ipv6,
@@ -26,6 +29,32 @@
     EXTERNAL_IP6 = get_ipv6()
 
 
+def echo(comm, x):
+    return x
+
+
+class CountedObject(object):
+    """
+    A class which counts the number of live instances.
+    """
+    n_instances = 0
+
+    # Use __new__, as __init__ can be bypassed by pickle.
+    def __new__(cls):
+        cls.n_instances += 1
+        obj = object.__new__(cls)
+        finalize(obj, cls._finalize)
+        return obj
+
+    @classmethod
+    def _finalize(cls, *args):
+        cls.n_instances -= 1
+
+
+def echo_serialize(comm, x):
+    return {'result': to_serialize(x)}
+
+
 def test_server(loop):
     """
     Simple Server test.
@@ -218,6 +247,48 @@ def test_rpc_inputs():
         r.close_rpc()
 
 
+@gen.coroutine
+def check_rpc_message_lifetime(*listen_args):
+    # Issue #956: rpc arguments and result shouldn't be kept alive longer
+    # than necessary
+    server = Server({'echo': echo_serialize})
+    server.listen(*listen_args)
+
+    # Sanity check
+    obj = CountedObject()
+    assert CountedObject.n_instances == 1
+    del obj
+    assert CountedObject.n_instances == 0
+
+    with rpc(server.address) as remote:
+        obj = CountedObject()
+        res = yield remote.echo(x=to_serialize(obj))
+        assert isinstance(res['result'], CountedObject)
+        # Make sure resource cleanup code in coroutines runs
+        yield gen.sleep(0.05)
+
+        w1 = weakref.ref(obj)
+        w2 = weakref.ref(res['result'])
+        del obj, res
+
+        assert w1() is None
+        assert w2() is None
+        # If additional instances were created, they were deleted as well
+        assert CountedObject.n_instances == 0
+
+@gen_test()
+def test_rpc_message_lifetime_default():
+    yield check_rpc_message_lifetime()
+
+@gen_test()
+def test_rpc_message_lifetime_tcp():
+    yield check_rpc_message_lifetime('tcp://')
+
+@gen_test()
+def test_rpc_message_lifetime_inproc():
+    yield check_rpc_message_lifetime('inproc://')
+
+
 @gen.coroutine
 def check_rpc_with_many_connections(listen_arg):
     @gen.coroutine
@@ -245,9 +316,6 @@ def test_rpc_with_many_connections_inproc():
     yield check_rpc_with_many_connections('inproc://')
 
 
-def echo(comm, x):
-    return x
-
 @gen.coroutine
 def check_large_packets(listen_arg):
     """ tornado has a 100MB cap by default """