Repository: bokeh/bokeh

Issue:
Bokeh Server is incompatible with tornado=4.5
There's a currently undiagnosed issue where Bokeh Server applications are incompatible with the newest tornado v4.5 release.

The server appears to start normally, but doesn't receive any requests or open any sessions when queried.

To reproduce:
* Install Bokeh v0.12.5 via conda or pip
* Run a Bokeh Server app example via ``bokeh serve app.py``
(below is an example app)
* Try (and fail) to open an application session at http://localhost:5006/line_on_off in a browser or via cURL. You'll see that any request just hangs.

contents of app.py
```python
""" Example demonstrating turning lines on and off - with bokeh server

"""

import numpy as np

from bokeh.io import curdoc
from bokeh.layouts import row
from bokeh.palettes import Viridis3
from bokeh.plotting import figure
from bokeh.models import CheckboxGroup

p = figure()
props = dict(line_width=4, line_alpha=0.7)
x = np.linspace(0, 4 * np.pi, 100)
l0 = p.line(x, np.sin(x), color=Viridis3[0], legend="Line 0", **props)
l1 = p.line(x, 4 * np.cos(x), color=Viridis3[1], legend="Line 1", **props)
l2 = p.line(x, np.tan(x), color=Viridis3[2], legend="Line 2", **props)

checkbox = CheckboxGroup(labels=["Line 0", "Line 1", "Line 2"], active=[0, 1, 2], width=100)

def update(attr, old, new):
    l0.visible = 0 in checkbox.active
    l1.visible = 1 in checkbox.active
    l2.visible = 2 in checkbox.active

checkbox.on_change('active', update)

layout = row(checkbox, p)
curdoc().add_root(layout)
```

Platform: MacOS and Ubuntu 14.04
Bokeh = 0.12.5
Tornado = 4.5



Discussion:
The culprit is the change to the handling of `yield None`: this used to raise `BadYieldError`, now it's equivalent to `yield gen.moment`. There are two places in Bokeh that catch BadYieldError; both of these need to be updated with explicit checks for None. 
Working up a PR now.
@bdarnell THANKS was there a deprecation and if so, when did it start? i.e. will updated code work for tornado >= 4.3 ? Or will version checks be needed?

No, there was no advance notice of the change. I assumed that taking something that never worked (i.e. it always raised an exception) and making it work wouldn't break anybody; I wasn't considering the fact that someone might be catching this exception and relying on it at runtime. 

You won't need a version check; looking for None will work for all versions of Tornado. 

```
@gen.coroutine
def yield_for_all_futures(result):
    """ Converts result into a Future by collapsing any futures inside result.

    If result is a Future we yield until it's done, then if the value inside
    the Future is another Future we yield until it's done as well, and so on.
    """
    while True:
        if result is None:
            break
        try:
            future = gen.convert_yielded(result)
        except gen.BadYieldError:
            # result is not a yieldable thing, we are done
            break
        else:
            result = yield future
    raise gen.Return(result)
```

This function is dangerous, BTW: lots of things other than Futures are accepted by `convert_yielded` (and have been ever since it was introduced). In particular, an empty list (or empty dict) is accepted by `convert_yielded` and resolves into another empty list, resulting in an infinite loop. It's better to only peel off one layer of `Future` at a time, and only once per maybe-coroutine function call. 
@bdarnell unfortunately this is not the issue or at least not all of it. With this diff:
```
diff --git a/bokeh/util/tornado.py b/bokeh/util/tornado.py
index 45834a28b..1a939a397 100644
--- a/bokeh/util/tornado.py
+++ b/bokeh/util/tornado.py
@@ -18,6 +18,9 @@ def yield_for_all_futures(result):
     while True:
         try:
             future = gen.convert_yielded(result)
+            if future is None:
+                import pdb; pdb.set_trace()
+                raise gen.BadYieldError
         except gen.BadYieldError:
             # result is not a yieldable thing, we are done
             break
@@ -58,6 +61,9 @@ class _AsyncPeriodic(object):
             result = self._func()
             try:
                 callback_future = gen.convert_yielded(result)
+                if callback_future is None:
+                    import pdb; pdb.set_trace()
+                    raise gen.BadYieldError
             except gen.BadYieldError:
                 # result is not a yieldable thing
                 return sleep_future
``` 
an app still does not connect or load, but the pdb breakpoints are never hit, so it's not even getting to these points in the code. 
I think it's because ``gen.convert_yielded(None)`` returns a Future. You may have to check if ``result`` is None or not

```
    while result is not None:
        future = gen.convert_yielded(result)
        result = yield future
    raise gen.Return(result)
```
Sure enough, this diff works:
```
diff --git a/bokeh/util/tornado.py b/bokeh/util/tornado.py
index 45834a28b..4f305a14e 100644
--- a/bokeh/util/tornado.py
+++ b/bokeh/util/tornado.py
@@ -17,7 +17,11 @@ def yield_for_all_futures(result):
     """
     while True:
         try:
+            if result is None:
+                raise gen.BadYieldError
             future = gen.convert_yielded(result)
+            if future is None:
+                raise gen.BadYieldError
         except gen.BadYieldError:
             # result is not a yieldable thing, we are done
             break
@@ -57,7 +61,11 @@ class _AsyncPeriodic(object):
             sleep_future = self.sleep()
             result = self._func()
             try:
+                if result is None:
+                    raise gen.BadYieldError
                 callback_future = gen.convert_yielded(result)
+                if callback_future is None:
+                    raise gen.BadYieldError
             except gen.BadYieldError:
                 # result is not a yieldable thing
                 return sleep_future
```

The second check in each pair is unnecessary - `convert_yielded` will never return None. 
@bdarnell first I really appreciate your input. I tested an app with the following diff against 4.4.2 and 4.5 and it seems to work great, and if I understand your earlier comment, is also safer. Do you agree?
```
diff --git a/bokeh/util/tornado.py b/bokeh/util/tornado.py
index 45834a28b..03a366098 100644
--- a/bokeh/util/tornado.py
+++ b/bokeh/util/tornado.py
@@ -16,13 +16,11 @@ def yield_for_all_futures(result):
     the Future is another Future we yield until it's done as well, and so on.
     """
     while True:
-        try:
-            future = gen.convert_yielded(result)
-        except gen.BadYieldError:
-            # result is not a yieldable thing, we are done
+        # result is not a yieldable thing, we are done
+        if not isinstance(result, gen.Future):
             break
-        else:
-            result = yield future
+        future = gen.convert_yielded(result)
+        result = yield future
     raise gen.Return(result)

 class _AsyncPeriodic(object):
@@ -56,13 +54,10 @@ class _AsyncPeriodic(object):
             # the period.
             sleep_future = self.sleep()
             result = self._func()
-            try:
-                callback_future = gen.convert_yielded(result)
-            except gen.BadYieldError:
-                # result is not a yieldable thing
+            if not isinstance(result, gen.Future):
                 return sleep_future
-            else:
-                return gen.multi([sleep_future, callback_future])
+            callback_future = gen.convert_yielded(result)
+            return gen.multi([sleep_future, callback_future])
         def on_done(future):
             if not self._stopped:
                 self._loop.add_future(invoke(), on_done)
```

Python and JS tests also pass with this diff. 
Also seems to work with py2.7 and tornado 4.3
Checking for `isinstance(result, gen.Future)` is too narrow - it won't work with python 3.5 `async def` native coroutines (among other things, but native coroutines are the most important). I think you could use `inspect.isawaitable` on python 3.5+ and `backports_abc.isawaitable` on older versions of python, although that wouldn't be 100% compatible with what you were doing before. I'm not sure whether that would be a real problem or not. The solution that is most compatible with what you had before is the one from my comment above: https://github.com/bokeh/bokeh/issues/6152#issuecomment-294649755

Resolving patch:
diff --git a/bokeh/util/tornado.py b/bokeh/util/tornado.py
--- a/bokeh/util/tornado.py
+++ b/bokeh/util/tornado.py
@@ -16,6 +16,12 @@ def yield_for_all_futures(result):
     the Future is another Future we yield until it's done as well, and so on.
     """
     while True:
+
+        # This is needed for Tornado >= 4.5 where convert_yielded will no
+        # longer raise BadYieldError on None
+        if result is None:
+            break
+
         try:
             future = gen.convert_yielded(result)
         except gen.BadYieldError:
@@ -23,6 +29,7 @@ def yield_for_all_futures(result):
             break
         else:
             result = yield future
+
     raise gen.Return(result)
 
 class _AsyncPeriodic(object):
@@ -56,6 +63,12 @@ def invoke():
             # the period.
             sleep_future = self.sleep()
             result = self._func()
+
+            # This is needed for Tornado >= 4.5 where convert_yielded will no
+            # longer raise BadYieldError on None
+            if result is None:
+                return sleep_future
+
             try:
                 callback_future = gen.convert_yielded(result)
             except gen.BadYieldError:


Test patch: