Repository: bokeh/bokeh

Issue:
Tornado 5.0 compatibility
Note that the next verion of Tornado, 5.0, will include breaking changes which break the existing Bokeh test suite.


Discussion:
Is there a description of the changes somewhere?
Well the majority of failures (though not all I don't think) seem to be this:
```
TypeError: initialize() got an unexpected keyword argument 'io_loop'
```
So let's start there. @bdarnell do you have any comments or guidance? Are there any migration notes available?
Well, it looks like this change to `HTTPServer`:
```
.. versionchanged:: 5.0
   The ``io_loop`` argument has been removed.
```
I'm not sure what the appropriate change to make is, though.
I think that whenever you explicitly provide an `io_loop=` keyword the thing to do is to *not* provide that keyword, and instead ensure that you create the object in the right event loop.  Given that Bokeh doesn't do particularly odd things I suspect that this is always the case, and that you can remove these keywords without concern.
Unfortunately it doesn't seem to be as simple as that. Doing that causes several tests to hang completely. 
FWIW `bokeh serve` app invocations seem to work, but `tornado_embed.py`, e.g. does not:
```
❯ python tornado_embed.py
Traceback (most recent call last):
  File "tornado_embed.py", line 47, in <module>
    server = Server({'/bkapp': modify_doc}, num_procs=4, extra_patterns=[('/', IndexHandler)])
  File "/Users/bryan/work/bokeh/bokeh/server/server.py", line 398, in __init__
    http_server.start(opts.num_procs)
  File "/Users/bryan/anaconda/lib/python3.6/site-packages/tornado-5.0.dev1-py3.6-macosx-10.7-x86_64.egg/tornado/tcpserver.py", line 222, in start
    process.fork_processes(num_processes)
  File "/Users/bryan/anaconda/lib/python3.6/site-packages/tornado-5.0.dev1-py3.6-macosx-10.7-x86_64.egg/tornado/process.py", line 130, in fork_processes
    raise RuntimeError("Cannot run in multiple processes: IOLoop instance "
RuntimeError: Cannot run in multiple processes: IOLoop instance has already been initialized. You cannot call IOLoop.instance() before calling start_processes()
```
As for the tests we do manage io_loops explicitly in a way I only vaguely understand. Is there an expected timeframe for 5.0? I think in the short term may only be able to update the dependencies to `<5.0` for bokeh
There's no strict timeline for Tornado 5.0 yet; I want to work with projects like bokeh to make sure the new version will work for you before it's released. 

As @mrocklin says, the `io_loop` arguments have been removed because we're now stricter about everything running on the proper thread, and these arguments mainly give you a way to mess that up (and implementing them consistently with asyncio is tricky). Can you point to the specific tests that are failing, or where that explicit IOLoop management happens? The places that I looked at all seemed to be doing the right thing (mainly calling `make_current()`). I tried running the tests and after wrestling with dependencies all of the "unit and not selenium" tests are passing. 

The `tornado_embed.py` problem looks like a regression - I think I may have broken this entry point to multi-process mode. 
@bdarnell First, please let me say thank you for your help with this. 

First here is a diff I'm starting with
```
❯ git diff
diff --git a/bokeh/server/server.py b/bokeh/server/server.py
index ba5243730..2e80103df 100644
--- a/bokeh/server/server.py
+++ b/bokeh/server/server.py
@@ -393,10 +393,7 @@ class Server(BaseServer):
         try:
             tornado_app = BokehTornado(applications, extra_websocket_origins=extra_websocket_origins, prefix=self.prefix, **kwargs)

-            if io_loop is not None:
-                http_server = HTTPServer(tornado_app, io_loop=io_loop, **http_server_kwargs)
-            else:
-                http_server = HTTPServer(tornado_app, **http_server_kwargs)
+            http_server = HTTPServer(tornado_app, **http_server_kwargs)

             http_server.start(opts.num_procs)
             http_server.add_sockets(sockets)
diff --git a/bokeh/server/tests/test_server.py b/bokeh/server/tests/test_server.py
index ac930aeaa..d851d8430 100644
--- a/bokeh/server/tests/test_server.py
+++ b/bokeh/server/tests/test_server.py
@@ -143,16 +143,14 @@ def test__lifecycle_hooks():
         def check_done():
             if len(handler.hooks) == 4:
                 server.io_loop.stop()
-        server_load_checker = PeriodicCallback(check_done, 1,
-                                               io_loop=server.io_loop)
+        server_load_checker = PeriodicCallback(check_done, 1)
         server_load_checker.start()
         server.io_loop.start()
         server_load_checker.stop()

         # now we create a session
         client_session = pull_session(session_id='test__lifecycle_hooks',
-                                      url=url(server),
-                                      io_loop=server.io_loop)
+                                      url=url(server))
         client_doc = client_session.document
         assert len(client_doc.roots) == 1

diff --git a/bokeh/server/tornado.py b/bokeh/server/tornado.py
index 153e64644..10903b9b7 100644
--- a/bokeh/server/tornado.py
+++ b/bokeh/server/tornado.py
@@ -232,15 +232,13 @@ class BokehTornado(TornadoApplication):
         self._clients = set()

         self._stats_job = PeriodicCallback(self._log_stats,
-                                           self._stats_log_frequency_milliseconds,
-                                           io_loop=self._loop)
+                                           self._stats_log_frequency_milliseconds)

         self._cleanup_job = PeriodicCallback(self._cleanup_sessions,
-                                             self._check_unused_sessions_milliseconds,
-                                             io_loop=self._loop)
+                                             self._check_unused_sessions_milliseconds)

         if self._keep_alive_milliseconds > 0:
-            self._ping_job = PeriodicCallback(self._keep_alive, self._keep_alive_milliseconds, io_loop=self._loop)
+            self._ping_job = PeriodicCallback(self._keep_alive, self._keep_alive_milliseconds)
         else:
             self._ping_job = None
```
Then if I run:
```
py.test -m "not (examples or integration or js or quality)" bokeh/server bokeh/command/subcommands/tests/test_serve.py
```
I see these two failures
```
_________________________________________________________ TestCallbackGroup.test_periodic_runs _________________________________________________________

self = <bokeh.server.tests.test_callbacks.TestCallbackGroup testMethod=test_periodic_runs>

    def test_periodic_runs(self):
        with (LoopAndGroup()) as ctx:
            func = _make_invocation_counter(ctx.io_loop, stop_after=5)
            self.assertEqual(0, len(ctx.group._periodic_callbacks))
            ctx.group.add_periodic_callback(func, period_milliseconds=1)
            self.assertEqual(1, len(ctx.group._periodic_callbacks))
>       self.assertEqual(5, func.count())
E       AssertionError: 5 != 0

bokeh/server/tests/test_callbacks.py:74: AssertionError
----------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------
Keyboard interrupt
----------------------------------------------------------------- Captured stderr call -----------------------------------------------------------------
DEBUG:asyncio:Using selector: KqueueSelector
------------------------------------------------------------------ Captured log call -------------------------------------------------------------------
selector_events.py          65 DEBUG    Using selector: KqueueSelector
_______________________________________________ TestCallbackGroup.test_same_callback_as_all_three_types ________________________________________________

self = <bokeh.server.tests.test_callbacks.TestCallbackGroup testMethod=test_same_callback_as_all_three_types>

    def test_same_callback_as_all_three_types(self):
        with (LoopAndGroup()) as ctx:
            func = _make_invocation_counter(ctx.io_loop, stop_after=5)
            # we want the timeout and next_tick to run before the periodic
            ctx.group.add_periodic_callback(func, period_milliseconds=2)
            ctx.group.add_timeout_callback(func, timeout_milliseconds=1)
            ctx.group.add_next_tick_callback(func)
>       self.assertEqual(5, func.count())
E       AssertionError: 5 != 2

bokeh/server/tests/test_callbacks.py:129: AssertionError
```

But those tests hang completely. This is only if I hit ctrl-c to stop them. Then `test_server.py` hangs entirely and hitting ctrl-c exits the entire test run so I don't know yet what might be failing there. 
As for explicit management I think this is about it:

https://github.com/bokeh/bokeh/blob/master/bokeh/server/tests/utils.py#L59
Jus as a heads up, `ipython notebeook` does not work either:
```
File "/Users/bryan/anaconda/lib/python3.6/site-packages/notebook/notebookapp.py", line 43, in <module>
    ioloop.install()
  File "/Users/bryan/anaconda/lib/python3.6/site-packages/zmq/eventloop/ioloop.py", line 215, in install
    ioloop.IOLoop.configure(ZMQIOLoop)
  File "/Users/bryan/anaconda/lib/python3.6/site-packages/tornado-5.0.dev1-py3.6-macosx-10.7-x86_64.egg/tornado/ioloop.py", line 182, in configure
    "only AsyncIOLoop is allowed when asyncio is available")
RuntimeError: only AsyncIOLoop is allowed when asyncio is available
```
Reported in: https://github.com/jupyter/notebook/issues/3023
Fixed in: https://github.com/jupyter/notebook/pull/3034


OK, when I said all the tests were passing, I had fat-fingered my PYTHONPATH tweaks to run against tornado master, so it was still using 4.5. Now I'm able to reproduce the behavior from https://github.com/bokeh/bokeh/issues/7308#issuecomment-351767751

Here's a diff that gets all of those tests passing but one:

```diff
diff --git a/bokeh/server/server.py b/bokeh/server/server.py
index ba5243730..2e80103df 100644
--- a/bokeh/server/server.py
+++ b/bokeh/server/server.py
@@ -393,10 +393,7 @@ class Server(BaseServer):
         try:
             tornado_app = BokehTornado(applications, extra_websocket_origins=extra_websocket_origins, prefix=self.prefix, **kwargs)
 
-            if io_loop is not None:
-                http_server = HTTPServer(tornado_app, io_loop=io_loop, **http_server_kwargs)
-            else:
-                http_server = HTTPServer(tornado_app, **http_server_kwargs)
+            http_server = HTTPServer(tornado_app, **http_server_kwargs)
 
             http_server.start(opts.num_procs)
             http_server.add_sockets(sockets)
diff --git a/bokeh/server/tests/test_callbacks.py b/bokeh/server/tests/test_callbacks.py
index c08bf752f..05f07f645 100644
--- a/bokeh/server/tests/test_callbacks.py
+++ b/bokeh/server/tests/test_callbacks.py
@@ -31,6 +31,12 @@ def run(loop):
 class LoopAndGroup(object):
     def __init__(self, quit_after=None):
         self.io_loop = IOLoop()
+        try:
+            import asyncio
+        except ImportError:
+            pass
+        else:
+            asyncio.set_event_loop(self.io_loop.asyncio_loop)
         self.group = _CallbackGroup(self.io_loop)
 
         if quit_after is not None:
diff --git a/bokeh/server/tests/test_server.py b/bokeh/server/tests/test_server.py
index ac930aeaa..15b6a9d20 100644
--- a/bokeh/server/tests/test_server.py
+++ b/bokeh/server/tests/test_server.py
@@ -143,8 +143,7 @@ def test__lifecycle_hooks():
         def check_done():
             if len(handler.hooks) == 4:
                 server.io_loop.stop()
-        server_load_checker = PeriodicCallback(check_done, 1,
-                                               io_loop=server.io_loop)
+        server_load_checker = PeriodicCallback(check_done, 1)
         server_load_checker.start()
         server.io_loop.start()
         server_load_checker.stop()
diff --git a/bokeh/server/tests/utils.py b/bokeh/server/tests/utils.py
index b43c2e540..417f306d5 100644
--- a/bokeh/server/tests/utils.py
+++ b/bokeh/server/tests/utils.py
@@ -41,7 +41,7 @@ def websocket_open(io_loop, url, origin=None):
     request = HTTPRequest(url)
     if origin is not None:
         request.headers['Origin'] = origin
-    websocket_connect(request, callback=handle_connection, io_loop=io_loop)
+    websocket_connect(request, callback=handle_connection)
 
     io_loop.start()
 
@@ -60,6 +60,12 @@ class ManagedServerLoop(object):
     def __init__(self, application, **server_kwargs):
         loop = IOLoop()
         loop.make_current()
+        try:
+            import asyncio
+        except ImportError:
+            pass
+        else:
+            asyncio.set_event_loop(loop.asyncio_loop)
         server_kwargs['io_loop'] = loop
         self._server = Server(application, **server_kwargs)
     def __exit__(self, type, value, traceback):
diff --git a/bokeh/server/tornado.py b/bokeh/server/tornado.py
index 153e64644..10903b9b7 100644
--- a/bokeh/server/tornado.py
+++ b/bokeh/server/tornado.py
@@ -232,15 +232,13 @@ class BokehTornado(TornadoApplication):
         self._clients = set()
 
         self._stats_job = PeriodicCallback(self._log_stats,
-                                           self._stats_log_frequency_milliseconds,
-                                           io_loop=self._loop)
+                                           self._stats_log_frequency_milliseconds)
 
         self._cleanup_job = PeriodicCallback(self._cleanup_sessions,
-                                             self._check_unused_sessions_milliseconds,
-                                             io_loop=self._loop)
+                                             self._check_unused_sessions_milliseconds)
 
         if self._keep_alive_milliseconds > 0:
-            self._ping_job = PeriodicCallback(self._keep_alive, self._keep_alive_milliseconds, io_loop=self._loop)
+            self._ping_job = PeriodicCallback(self._keep_alive, self._keep_alive_milliseconds)
         else:
             self._ping_job = None
```

I'm going to change `IOLoop.make_current()` to call `asyncio.set_event_loop` for you; clients shouldn't have to deal with that. You might need to add a `make_current()` call in `LoopAndGroup`, though. 

Note that I had to reintroduce the `io_loop` passed to `pull_session`. Passing `None` there doesn't work as expected due to this special case in `ClientConnection`:

https://github.com/bokeh/bokeh/blob/90c3f6c756b4452862978f85dcb41525c254ddc3/bokeh/client/connection.py#L73-L76

The remaining failure is `test__lifecycle_hooks`, where the order of two hooks was reversed:

```
===================================================== FAILURES ======================================================
_______________________________________________ test__lifecycle_hooks _______________________________________________

    def test__lifecycle_hooks():
        application = Application()
        handler = HookTestHandler()
        application.add(handler)
        with ManagedServerLoop(application, check_unused_sessions_milliseconds=30) as server:
            # wait for server callbacks to run before we mix in the
            # session, this keeps the test deterministic
            def check_done():
                if len(handler.hooks) == 4:
                    server.io_loop.stop()
            server_load_checker = PeriodicCallback(check_done, 1)
            server_load_checker.start()
            server.io_loop.start()
            server_load_checker.stop()
    
            # now we create a session
            client_session = pull_session(session_id='test__lifecycle_hooks',
                                          url=url(server),
                                          io_loop=server.io_loop)
            client_doc = client_session.document
            assert len(client_doc.roots) == 1
    
            server_session = server.get_session('/', client_session.id)
            server_doc = server_session.document
            assert len(server_doc.roots) == 1
    
            client_session.close()
            # expire the session quickly rather than after the
            # usual timeout
            server_session.request_expiration()
    
            def on_done():
                server.io_loop.stop()
    
            server.io_loop.call_later(0.1, on_done)
    
            server.io_loop.start()
    
>       assert handler.hooks == ["server_loaded",
                                 "next_tick_server",
                                 "timeout_server",
                                 "periodic_server",
                                 "session_created",
                                 "next_tick_session",
                                 "modify",
                                 "timeout_session",
                                 "periodic_session",
                                 "session_destroyed",
                                 "server_unloaded"]
E       assert ['server_load...session', ...] == ['server_loade...session', ...]
E         At index 6 diff: 'timeout_session' != 'modify'
E         Full diff:
E         ['server_loaded',
E         'next_tick_server',
E         'timeout_server',
E         'periodic_server',
E         'session_created',
E         'next_tick_session',
E         +  'modify',
E         'timeout_session',
E         -  'modify',
E         'periodic_session',
E         'session_destroyed',
E         'server_unloaded']

bokeh/server/tests/test_server.py:174: AssertionError
```

This is a known consequence of the move to asyncio: the timing of callbacks in some situations has changed. If this is significant for you I can look into this further, but otherwise I would suggest relaxing this test. 

Resolving patch:
diff --git a/bokeh/server/server.py b/bokeh/server/server.py
--- a/bokeh/server/server.py
+++ b/bokeh/server/server.py
@@ -393,10 +393,7 @@ def __init__(self, applications, io_loop=None, http_server_kwargs=None, **kwargs
         try:
             tornado_app = BokehTornado(applications, extra_websocket_origins=extra_websocket_origins, prefix=self.prefix, **kwargs)
 
-            if io_loop is not None:
-                http_server = HTTPServer(tornado_app, io_loop=io_loop, **http_server_kwargs)
-            else:
-                http_server = HTTPServer(tornado_app, **http_server_kwargs)
+            http_server = HTTPServer(tornado_app, **http_server_kwargs)
 
             http_server.start(opts.num_procs)
             http_server.add_sockets(sockets)
diff --git a/bokeh/server/tornado.py b/bokeh/server/tornado.py
--- a/bokeh/server/tornado.py
+++ b/bokeh/server/tornado.py
@@ -232,15 +232,13 @@ def initialize(self, io_loop):
         self._clients = set()
 
         self._stats_job = PeriodicCallback(self._log_stats,
-                                           self._stats_log_frequency_milliseconds,
-                                           io_loop=self._loop)
+                                           self._stats_log_frequency_milliseconds)
 
         self._cleanup_job = PeriodicCallback(self._cleanup_sessions,
-                                             self._check_unused_sessions_milliseconds,
-                                             io_loop=self._loop)
+                                             self._check_unused_sessions_milliseconds)
 
         if self._keep_alive_milliseconds > 0:
-            self._ping_job = PeriodicCallback(self._keep_alive, self._keep_alive_milliseconds, io_loop=self._loop)
+            self._ping_job = PeriodicCallback(self._keep_alive, self._keep_alive_milliseconds)
         else:
             self._ping_job = None
 


Test patch:
diff --git a/bokeh/server/tests/test_callbacks.py b/bokeh/server/tests/test_callbacks.py
--- a/bokeh/server/tests/test_callbacks.py
+++ b/bokeh/server/tests/test_callbacks.py
@@ -31,6 +31,7 @@ def run(loop):
 class LoopAndGroup(object):
     def __init__(self, quit_after=None):
         self.io_loop = IOLoop()
+        self.io_loop.make_current()
         self.group = _CallbackGroup(self.io_loop)
 
         if quit_after is not None:
diff --git a/bokeh/server/tests/test_server.py b/bokeh/server/tests/test_server.py
--- a/bokeh/server/tests/test_server.py
+++ b/bokeh/server/tests/test_server.py
@@ -143,8 +143,7 @@ def test__lifecycle_hooks():
         def check_done():
             if len(handler.hooks) == 4:
                 server.io_loop.stop()
-        server_load_checker = PeriodicCallback(check_done, 1,
-                                               io_loop=server.io_loop)
+        server_load_checker = PeriodicCallback(check_done, 1)
         server_load_checker.start()
         server.io_loop.start()
         server_load_checker.stop()
diff --git a/bokeh/server/tests/utils.py b/bokeh/server/tests/utils.py
--- a/bokeh/server/tests/utils.py
+++ b/bokeh/server/tests/utils.py
@@ -41,7 +41,7 @@ def handle_connection(future):
     request = HTTPRequest(url)
     if origin is not None:
         request.headers['Origin'] = origin
-    websocket_connect(request, callback=handle_connection, io_loop=io_loop)
+    websocket_connect(request, callback=handle_connection)
 
     io_loop.start()