Repository: bokeh/bokeh

Issue:
memory leak on add_next_tick_callback
ubuntu/18.04
python/3.6
bokeh/0.12.15dev8 onwards

The following code, which is derived from [this example](https://bokeh.pydata.org/en/latest/docs/user_guide/server.html#updating-from-unlocked-callbacks) (the size of an object being passed to `add_next_tick_callback` is enlarged for emphasis), continuously drains memory starting from bokeh/0.12.15dev8 (in particular, https://github.com/bokeh/bokeh/commit/de32960039d19599f03e32ab0aedc090918b4602)

```python
from functools import partial

from concurrent.futures import ThreadPoolExecutor
from tornado import gen
import numpy as np

from bokeh.document import without_document_lock
from bokeh.models import ColumnDataSource
from bokeh.plotting import curdoc, figure

source = ColumnDataSource(data=dict(x=[0], y=[0], color=["blue"]))

i = 0

doc = curdoc()

executor = ThreadPoolExecutor(max_workers=2)

def blocking_task():
    # receive/generate new data
    return np.random.rand(1024, 1024)

@gen.coroutine
def locked_update(res):
    # keep it simple as in example, but practically the plot may be updated with res values
    source.stream(dict(x=[source.data['x'][-1]+1], y=[i], color=["blue"]))

@gen.coroutine
@without_document_lock
def unlocked_task():
    global i
    i += 1
    res = yield executor.submit(blocking_task)
    doc.add_next_tick_callback(partial(locked_update, res=res))

p = figure(x_range=[0, 100], y_range=[0, 20])
l = p.circle(x='x', y='y', color='color', source=source)

doc.add_periodic_callback(unlocked_task, 1000)
doc.add_root(p)
```
Seems that, the `partial` objects passed to 'add_next_tick_callback' are piling up in memory and never get released/garbage collected.

Here is a minimal reproducible example:
```python
from functools import partial

import numpy as np
from bokeh.plotting import curdoc
from tornado import gen

doc = curdoc()

@gen.coroutine
def locked_update(res):
    pass

@gen.coroutine
def update():
    res = np.random.rand(1024, 1024)
    doc.add_next_tick_callback(partial(locked_update, res))

doc.add_periodic_callback(update, 1000)
```



Discussion:
Just to be clear, this *only* happens when also using `functools.partial`?
As a quick check, the same can be reproduced with a lambda function
```python
import numpy as np
from bokeh.plotting import curdoc
from tornado import gen

doc = curdoc()

@gen.coroutine
def update():
    res = np.random.rand(1024, 1024)
    doc.add_next_tick_callback(lambda res=res: None)

doc.add_periodic_callback(update, 1000)
```
Here is a simple gc debugging version that shows a growing total number of tracked objects in bokeh/0.12.16 in comparison to a more-or-less stable (still, slowly creeping after stabilization) number in bokeh/0.12.14
```python
import gc

import numpy as np
from bokeh.plotting import curdoc
from tornado import gen

gc.set_debug(gc.DEBUG_STATS)

doc = curdoc()

@gen.coroutine
def update():
    res = np.random.rand(1024, 1024)
    doc.add_next_tick_callback(lambda res=res: None)

doc.add_periodic_callback(update, 1000)
```

![index](https://user-images.githubusercontent.com/13196195/41406642-36305cde-6fcd-11e8-8063-612e078561a7.png)

<details>
  <summary>bokeh/0.12.14 (10 min runtime)</summary>

```
gc: collecting generation 0...
gc: objects in each generation: 900 1883 91978
gc: done, 90 unreachable, 0 uncollectable, 0.0026s elapsed
gc: collecting generation 0...
gc: objects in each generation: 814 2536 91968
gc: done, 630 unreachable, 0 uncollectable, 0.0002s elapsed
gc: collecting generation 0...
gc: objects in each generation: 918 2550 91968
gc: done, 735 unreachable, 0 uncollectable, 0.0002s elapsed
gc: collecting generation 0...
gc: objects in each generation: 1217 2179 91963
gc: done, 885 unreachable, 0 uncollectable, 0.0003s elapsed
gc: collecting generation 0...
gc: objects in each generation: 935 2335 91963
gc: done, 750 unreachable, 0 uncollectable, 0.0002s elapsed
gc: collecting generation 0...
gc: objects in each generation: 935 2349 91963
gc: done, 750 unreachable, 0 uncollectable, 0.0003s elapsed
gc: collecting generation 1...
gc: objects in each generation: 935 2363 91963
gc: done, 1222 unreachable, 0 uncollectable, 0.0198s elapsed
gc: collecting generation 0...
gc: objects in each generation: 935 0 93805
gc: done, 750 unreachable, 0 uncollectable, 0.0003s elapsed
gc: collecting generation 0...
gc: objects in each generation: 935 14 93805
gc: done, 750 unreachable, 0 uncollectable, 0.0003s elapsed
gc: collecting generation 0...
gc: objects in each generation: 935 28 93805
gc: done, 750 unreachable, 0 uncollectable, 0.0002s elapsed
gc: collecting generation 0...
gc: objects in each generation: 935 42 93805
gc: done, 750 unreachable, 0 uncollectable, 0.0002s elapsed
gc: collecting generation 0...
gc: objects in each generation: 935 56 93805
gc: done, 750 unreachable, 0 uncollectable, 0.0003s elapsed
```
</details>

<details>
  <summary>bokeh/0.12.16 (already after 5 min)</summary>

```
gc: collecting generation 0...
gc: objects in each generation: 702 1976 92424
gc: done, 96 unreachable, 0 uncollectable, 0.0002s elapsed
gc: collecting generation 0...
gc: objects in each generation: 790 2405 92414
gc: done, 132 unreachable, 0 uncollectable, 0.0001s elapsed
gc: collecting generation 0...
gc: objects in each generation: 818 2937 92414
gc: done, 132 unreachable, 0 uncollectable, 0.0001s elapsed
gc: collecting generation 0...
gc: objects in each generation: 817 3429 92414
gc: done, 132 unreachable, 0 uncollectable, 0.0002s elapsed
gc: collecting generation 0...
gc: objects in each generation: 762 3941 92414
gc: done, 126 unreachable, 0 uncollectable, 0.0001s elapsed
gc: collecting generation 0...
gc: objects in each generation: 1110 4066 92409
gc: done, 162 unreachable, 0 uncollectable, 0.0014s elapsed
gc: collecting generation 0...
gc: objects in each generation: 819 4810 92409
gc: done, 132 unreachable, 0 uncollectable, 0.0001s elapsed
gc: collecting generation 1...
gc: objects in each generation: 765 5322 92409
gc: done, 556 unreachable, 0 uncollectable, 0.0006s elapsed
gc: collecting generation 0...
gc: objects in each generation: 828 0 97792
gc: done, 144 unreachable, 0 uncollectable, 0.0001s elapsed
gc: collecting generation 0...
gc: objects in each generation: 821 556 97772
gc: done, 132 unreachable, 0 uncollectable, 0.0001s elapsed
gc: collecting generation 0...
gc: objects in each generation: 745 1088 97772
gc: done, 126 unreachable, 0 uncollectable, 0.0001s elapsed
gc: collecting generation 0...
gc: objects in each generation: 762 1558 97772
gc: done, 126 unreachable, 0 uncollectable, 0.0001s elapsed
gc: collecting generation 0...
gc: objects in each generation: 762 2048 97772
gc: done, 126 unreachable, 0 uncollectable, 0.0002s elapsed
gc: collecting generation 0...
gc: objects in each generation: 741 2558 97772
gc: done, 126 unreachable, 0 uncollectable, 0.0001s elapsed
gc: collecting generation 0...
gc: objects in each generation: 762 3028 97772
gc: done, 126 unreachable, 0 uncollectable, 0.0001s elapsed
```
</details>

 <br/>

And one more test on reference count for the same callable object passed to `add_next_tick_callback`
```python
import gc

from bokeh.plotting import curdoc
from tornado import gen

doc = curdoc()

func = lambda: None

@gen.coroutine
def update():
    doc.add_next_tick_callback(func)

    gc.collect()
    print(len(gc.get_referrers(func)))

doc.add_periodic_callback(update, 1000)
```

<details>
  <summary>bokeh/0.12.14</summary>

```
4
4
4
4
4
4
4
4
4
4
4
```
</details>

<details>
  <summary>bokeh/0.12.16</summary>

```
4
6
8
10
12
14
16
18
20
22
24
26
28
30
```
</details>

 <br/>

Overall, it seems, this is a reference leak issue. And it starts to drain memory when different callable objects (especially, if they reference large data structures themselves) are captured through `add_next_tick_callback`.
@ivan-usov thanks for the detailed reports, the minimal test cases should make it easy to add tests to maintain that things get released in these circumstances. I guess to reiterate my previous question differently, can you confirm that this is *not* happening with "plain" functions? I'm just trying to figure out where to start to focus an investigation. 
Actually, I just found the location of the leaked references inside the bokeh server. They get accumulated in `_next_tick_removers_by_callable` of `_CallbackGroup` in `bokeh/util/tornado.py` module.
@ivan-usov do you think you can work up a PR? I'd like to stay focused on the upcoming release if it is something you are able to help with
@bryevdv  
> can you confirm that this is not happening with "plain" functions?

<details>
  <summary>Yes, it also bugs with plain functions</summary>

```python
import gc

from bokeh.plotting import curdoc
from tornado import gen

doc = curdoc()

@gen.coroutine
def func():
    pass

@gen.coroutine
def update():
    cb = doc.add_next_tick_callback(func)

    gc.collect()
    print(len(gc.get_referrers(func)))

doc.add_periodic_callback(update, 1000)
```
```
4
6
8
10
12
14
16
```
</details>
 <br/>

I still need to understand why is it happening, but it should be possible to work up a PR in the upcoming days.
Awesome, thanks!

It's probably worth looking at what happens when sessions close, too if possible. I believe a previous round of work took care of leaks when sessions were closed, and both tests and runtime checks were added. So I have pretty high confidence that that should be OK, but always worth double-checking. At the time tho, the idea of leaks after repeated calls *in* a session, like these case, did not come up or get addressed.

Resolving patch:
diff --git a/bokeh/document/document.py b/bokeh/document/document.py
--- a/bokeh/document/document.py
+++ b/bokeh/document/document.py
@@ -1006,9 +1006,11 @@ def _remove_session_callback(self, callback_obj, originator):
             else:
                 callback_objs = [callback_obj]
                 self._session_callbacks.remove(callback_obj)
-                for cb_objs in self._callback_objs_by_callable[originator].values():
+                for cb, cb_objs in list(self._callback_objs_by_callable[originator].items()):
                     try:
                         cb_objs.remove(callback_obj)
+                        if not cb_objs:
+                            del self._callback_objs_by_callable[originator][cb]
                     except KeyError:
                         pass
         except KeyError:
diff --git a/bokeh/util/tornado.py b/bokeh/util/tornado.py
--- a/bokeh/util/tornado.py
+++ b/bokeh/util/tornado.py
@@ -167,20 +167,22 @@ def remover():
         return remover
 
     def _execute_remover(self, callback_id, removers):
-            try:
-                with self._removers_lock:
-                    if callable(callback_id):
-                        remover = self._create_deprecated_remover(callback_id, removers)
-                    else:
-                        remover = removers.pop(callback_id)
-                        for cb_ids in self._get_removers_ids_by_callable(removers).values():
-                            try:
-                                cb_ids.remove(callback_id)
-                            except KeyError:
-                                pass
-            except KeyError:
-                raise ValueError("Removing a callback twice (or after it's already been run)")
-            remover()
+        try:
+            with self._removers_lock:
+                if callable(callback_id):
+                    remover = self._create_deprecated_remover(callback_id, removers)
+                else:
+                    remover = removers.pop(callback_id)
+                    for cb, cb_ids in list(self._get_removers_ids_by_callable(removers).items()):
+                        try:
+                            cb_ids.remove(callback_id)
+                            if not cb_ids:
+                                del self._get_removers_ids_by_callable(removers)[cb]
+                        except KeyError:
+                            pass
+        except KeyError:
+            raise ValueError("Removing a callback twice (or after it's already been run)")
+        remover()
 
     def add_next_tick_callback(self, callback, callback_id=None):
         """ Adds a callback to be run on the next tick.


Test patch: