Repository: bokeh/bokeh

Issue:
Memory leak in bokeh application
In current master, I noticed a memory leak when using a bokeh application. It's only slowly increasing, thus making it hard to detect. I bisected it to commit 18802dd4fe4915d146 (PR #5425).

When I add a `print(len(sys.modules))`, it keeps increasing on every page load, and when I comment the line https://github.com/bokeh/bokeh/pull/5425/files#diff-dc51006b23833be68da1b9ca8a8167a6R63 there's no leak.
Adding a print to [`document.__del__`](https://github.com/bokeh/bokeh/pull/5425/files#diff-665e6cdee790b076b7b90d3ddd57f504R305), I see that it never gets called.

I'm using python 3.5.2 and I have the same issue on a server running python 3.4.3.

@bryevdv @mattpap



Discussion:
Hi @bkueng thanks for the investigation. Since you are already a bit familiar with the code and also set up to observe the problem, I am hoping you might be amenable to trying a few more things out. (If you aren't able to spend any time, I understand but do please let us know)

First, I should mention, the new module-per-session behavior is 100% expected. Every connection gets it's own unique session and document, and a new module is created from the app script for the purpose of running the code that creates the new document in isolation. What should also happen, and perhaps what is not happening, is that these things should get cleaned up when they are no longer in use. 

Off the cuff, I note that the module has reference to the document, and the doc gets a ref to the module, which appears circular, and then also sys.modules gets a ref to the module as well, so that all might explain why there is never any clean up. The proper place for cleanup is when a session is terminated. I believe this is where sessions get discarded after they have timed out:

https://github.com/bokeh/bokeh/blob/master/bokeh/server/application_context.py#L194-L227

So a few things to check:

* are sessions actually getting discarded after timeouts? 
* if so, do some targeted `del` statements in the discard code solve the problem?

I would think maybe here in `session.destroy` is a place where some `del` statements could be added:

https://github.com/bokeh/bokeh/blob/master/bokeh/server/session.py#L113-L116

e.g. could try things like del the module out of `sys.modules`, del the module from `doc._modules`, etc. If some of these measures help the problem, then we can make sure the deletions are happening in the right place, and then add tests to ensure  the correct deletions are maintained. 
Since you say it is slow-increasing, I've added this to `0.12.6` but if a solution can be reached in the next several weeks I am happy to include it in `0.12.5`
With slow I refer mostly to testing and reproducing, on my server it's actually fast enough to let it run out of memory every 1 or 2 days.

> are sessions actually getting discarded after timeouts?

Yes after a couple of seconds, session.destroy() is called.

This is my instrumentation & fix-up diff:
```
diff --git a/bokeh/application/handlers/code.py b/bokeh/application/handlers/code.py
index 1aea410..90600c0 100644
--- a/bokeh/application/handlers/code.py
+++ b/bokeh/application/handlers/code.py
@@ -61,6 +61,7 @@ class CodeHandler(Handler):
         # stored modules are used to provide correct paths to
         # custom models resolver.
         sys.modules[module.__name__] = module
+        print(len(sys.modules))
         doc._modules.append(module)
 
         old_doc = curdoc()
diff --git a/bokeh/document.py b/bokeh/document.py
index f836116..e20263f 100644
--- a/bokeh/document.py
+++ b/bokeh/document.py
@@ -96,12 +96,17 @@ class Document(object):
         self._session_context = None
         self._modules = []
         self._template_variables = {}
+        print("init document")
 
-    def __del__(self):
+    def delete_modules(self):
+        print("delete_modules "+str(self))
         for module in self._modules:
             if module.__name__ in sys.modules:
                 del sys.modules[module.__name__]
 
+    def __del__(self):
+        print("destroy doc"+str(self))
+
     @property
     def roots(self):
         ''' A list of all the root models in this Document.
diff --git a/bokeh/server/application_context.py b/bokeh/server/application_context.py
index a3992ec..f3f23a5 100644
--- a/bokeh/server/application_context.py
+++ b/bokeh/server/application_context.py
@@ -193,6 +193,7 @@ class ApplicationContext(object):
 
     @gen.coroutine
     def _discard_session(self, session, should_discard):
+        print('discarding session')
         if session.connection_count > 0:
             raise RuntimeError("Should not be discarding a session with open connections")
         log.debug("Discarding session %r last in use %r milliseconds ago", session.id, session.milliseconds_since_last_unsubscribe)
@@ -202,6 +203,7 @@ class ApplicationContext(object):
         # session.destroy() wants the document lock so it can shut down the document
         # callbacks.
         def do_discard():
+            print('doing discard')
             # while we yielded for the document lock, the discard-worthiness of the
             # session may have changed.
             # However, since we have the document lock, our own lock will cause the
diff --git a/bokeh/server/session.py b/bokeh/server/session.py
index 4ecf24b..a5df9d4 100644
--- a/bokeh/server/session.py
+++ b/bokeh/server/session.py
@@ -111,6 +111,8 @@ class ServerSession(object):
         return self._expiration_blocked_count
 
     def destroy(self):
+        print('session.destroy')
+        self._document.delete_modules()
         self._destroyed = True
         self._document.remove_on_change(self)
         self._callbacks.remove_all_callbacks()
```
It removes the relevant `sys.modules[]` and resolves the memory leak. The document objects also get GC'd after a while. Seems you are right, it was a circular reference that could not have been cleaned up.

If you want, I can create a PR, but I think adding tests would go too far for my current understanding of bokeh.

That's great! If you will make an initial PR, I can add tests to it. I'd like you to get the credit for fixing this, and also it lightens my burden a bit too so that I can continue to focus on other issues, which is certainly appreciated!


Resolving patch:
diff --git a/bokeh/document.py b/bokeh/document.py
--- a/bokeh/document.py
+++ b/bokeh/document.py
@@ -97,7 +97,9 @@ def __init__(self, **kwargs):
         self._modules = []
         self._template_variables = {}
 
-    def __del__(self):
+    def delete_modules(self):
+        ''' Clean up sys.modules after the session is destroyed.
+        '''
         for module in self._modules:
             if module.__name__ in sys.modules:
                 del sys.modules[module.__name__]
diff --git a/bokeh/server/session.py b/bokeh/server/session.py
--- a/bokeh/server/session.py
+++ b/bokeh/server/session.py
@@ -112,6 +112,7 @@ def expiration_blocked_count(self):
 
     def destroy(self):
         self._destroyed = True
+        self._document.delete_modules()
         self._document.remove_on_change(self)
         self._callbacks.remove_all_callbacks()
 


Test patch:
diff --git a/bokeh/server/tests/test_session.py b/bokeh/server/tests/test_session.py
new file mode 100644
--- /dev/null
+++ b/bokeh/server/tests/test_session.py
@@ -0,0 +1,41 @@
+import mock
+
+from bokeh.document import Document
+
+import bokeh.server.session as bss
+
+def test_creation():
+    d = Document()
+    s = bss.ServerSession('some-id', d, 'ioloop')
+    assert s.id == 'some-id'
+    assert s.document == d
+    assert s.destroyed is False
+    assert s.expiration_requested is False
+    assert s.expiration_blocked == 0
+
+def test_subscribe():
+    d = Document()
+    s = bss.ServerSession('some-id', d, 'ioloop')
+    assert s.connection_count == 0
+    s.subscribe('connection1')
+    assert s.connection_count == 1
+    s.subscribe('connection2')
+    assert s.connection_count == 2
+    s.unsubscribe('connection1')
+    assert s.connection_count == 1
+    s.unsubscribe('connection2')
+    assert s.connection_count == 0
+
+def test_destroy():
+    d = Document()
+    s = bss.ServerSession('some-id', d, 'ioloop')
+    with mock.patch('bokeh.document.Document.delete_modules') as docdm:
+        s.destroy()
+        assert s.destroyed
+        docdm.assert_called_once()
+
+    s = bss.ServerSession('some-id', d, 'ioloop')
+    with mock.patch('bokeh.document.Document.remove_on_change') as docroc:
+        s.destroy()
+        assert s.destroyed
+        docroc.assert_called_with(s)
diff --git a/bokeh/tests/test_document.py b/bokeh/tests/test_document.py
--- a/bokeh/tests/test_document.py
+++ b/bokeh/tests/test_document.py
@@ -37,6 +37,20 @@ def test_default_template_vars(self):
         assert not d.roots
         assert d.template_variables == {}
 
+    def test_delete_modules(self):
+        d = document.Document()
+        assert not d.roots
+        class FakeMod(object):
+            __name__ = 'junkjunkjunk'
+        mod = FakeMod()
+        import sys
+        assert 'junkjunkjunk' not in sys.modules
+        sys.modules['junkjunkjunk'] = mod
+        d._modules.append(mod)
+        assert 'junkjunkjunk' in sys.modules
+        d.delete_modules()
+        assert 'junkjunkjunk' not in sys.modules
+
     def test_add_roots(self):
         d = document.Document()
         assert not d.roots