Repository: dask/distributed

Issue:
Reusing variables leaks significant memory on long runs
Use case: allocate a large variable and keep reusing it. Build a computation chain and stick the large variable in there. At end submit the large computation (dummy bellow) and also remember the resulting future in another variable (the reason for this is async computation, another client connects later and checks if the second variable finished).

Observation: the code bellows leaks at a rate of 1MB/s. After the a few hours the computation crashed due to out of memory. Sorry for the formatting bellow. I can't get it right

```python
from distributed import LocalCluster, Client, Variable
from distributed.utils_test import gen_cluster, inc, loop, cluster, slowinc
import time
from dask import delayed

def dummy(large):
    print(large[30], 'dummy')
    return bytearray(50)

def dummy_2(x):
    print(str(x[12]), 'dummy2')

def create_large():
    return bytearray(50)

def use_var():
    with LocalCluster(n_workers=1,processes=False, scheduler_port=8786, diagnostics_port=8787) as lc:
        while True:
            key = ''
            with Client(lc.scheduler_address) as client:
                v = Variable(name="var", client=client)
                large = None
                try:
                    large = v.get(timeout=1)
                    time.sleep(0.2)
                    print('found')
                except:
                    print('not found__creating..')
                    large = client.submit(create_large)
                    v.set(large)

                f1 = delayed(dummy)(large)
                f2 = delayed(dummy_2)(f1)

                f = client.compute(f2)

                key = f.key
                v2 = Variable(name=key, client=client)
                v2.set(f)

            while True:
                with Client(lc.scheduler_address) as client:
                    v = Variable(name=key, client=client)
                    f = v.get()
                    time.sleep(0.2)
               
                    if f.status == 'finished':
                        v.delete()
                        break
                    else:
                        time.sleep(1)
if __name__ == '__main__':
    use_var()
```



Discussion:
![figure_1](https://user-images.githubusercontent.com/11028129/27612238-e24185fe-5b95-11e7-9d26-11d06ca41990.png)

actually, this simpler code shows the memory accumulation also:

```python
def dummy_2():
    pass

def use_var():
    with LocalCluster(n_workers=1,processes=False, scheduler_port=8786, diagnostics_port=8787) as lc:

        while True:
            key = ''
            with Client(lc.scheduler_address) as client:
                f2 = delayed(dummy_2)
                f = client.compute(f2)

                key = f.key
                v2 = Variable(name=key, client=client)
                v2.set(f)

            while True:
                with Client(lc.scheduler_address) as client:
                    v = Variable(name=key, client=client)
                    f = v.get()
                    time.sleep(0.2)
                    print(f)

                    if f.status == 'finished':
                        print('finished')
                        v.delete()
                        break
                    else:
                        print('not finished')
                        time.sleep(1)


if __name__ == '__main__':
    use_var()
```

@danielb2814, I edited the formatting for you.
pitrou, thanks!

Actually it might not be variable al all but the establishing of new client connections. Apparently each new client connection allocates around 200KB which does not get released on scope exit.

This can be seen with:

```
while True:
                with Client(lc.scheduler_address) as client:`
                     time.sleep(1)


```
Actually, it doesn't seem to be the client that leaks but the scheduler. I'm investigating.

Resolving patch:
diff --git a/distributed/scheduler.py b/distributed/scheduler.py
--- a/distributed/scheduler.py
+++ b/distributed/scheduler.py
@@ -1141,7 +1141,7 @@ def add_client(self, comm, client=None):
         finally:
             if not comm.closed():
                 self.comms[client].send({'op': 'stream-closed'})
-                yield self.comms[client].close()
+            yield self.comms[client].close()
             del self.comms[client]
             logger.info("Close client connection: %s", client)
 
diff --git a/distributed/sizeof.py b/distributed/sizeof.py
--- a/distributed/sizeof.py
+++ b/distributed/sizeof.py
@@ -101,6 +101,6 @@ def safe_sizeof(obj, default_size=1e6):
     try:
         return sizeof(obj)
     except Exception:
-        logger.warn('Sizeof calculation failed.  Defaulting to 1MB',
-                    exc_info=True)
+        logger.warning('Sizeof calculation failed.  Defaulting to 1MB',
+                       exc_info=True)
         return int(default_size)


Test patch: