Repository: bokeh/bokeh

Issue:
Issue with rapid page reloads with Tornado 5
Seeing this on rapid page reloads:
```
2018-03-12 09:05:56,504 200 GET /gapminder (::1) 134.13ms
2018-03-12 09:05:56,506 Sending pull-doc-reply from session 'c13X3Xet3szgUeNWFMzrL86pmFq77b8q04pGAUD8WUFJ'
2018-03-12 09:05:56,513 Failed sending message as connection was closed
2018-03-12 09:05:56,514 WebSocket connection closed: code=None, reason=None
2018-03-12 09:05:56,604 Future exception was never retrieved
future: <Future finished exception=WebSocketClosedError()>
Traceback (most recent call last):
  File "/Users/bryanv/anaconda/envs/15dev1/lib/python3.6/site-packages/tornado/websocket.py", line 808, in wrapper
    yield fut
  File "/Users/bryanv/anaconda/envs/15dev1/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
tornado.iostream.StreamClosedError: Stream is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/bryanv/anaconda/envs/15dev1/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "/Users/bryanv/anaconda/envs/15dev1/lib/python3.6/site-packages/tornado/websocket.py", line 810, in wrapper
    raise WebSocketClosedError()
tornado.websocket.WebSocketClosedError
```

Seemed to happen between `0.12.14` and `0.12.15dev1`


Discussion:
Was hoping it might just be a Tornado 5 related issue but it seems to be any Tornado version. Will do a more careful bisection later today. 
Actually upon further inspection this seems like it might only happen with Tornado 5 after all. @bdarnell  do you have any ideas or tips for debugging?

To reproduce it is sufficient to run e.g. `bokeh serve --show gapminder` then hit apple-R (or whatever else) to reload the page rapidly. 
The type of exception raised by `write_message` changed in 5.0 - it is now always `WebSocketClosedError`, while it used to be a mix of `WebSocketClosedError` and `StreamClosedError` (mostly the latter). Are you catching `StreamClosedError` but not `WebSocketClosedError`? 

These "Future exception was never retrieved" messages also got more common in 5.0 with the move to asyncio. You may need to use `yield` or `await` when calling `write_message` to avoid this. This isn't ideal, but I'm not sure what to do about it. We jump through some hoops to silence these messages in other places but it looks like we missed WSH.write_message. 
We only catch `WebSocketClosedError`. Looking at our WS handler subclass, I see this:

```
   @gen.coroutine
    def write_message(self, message, binary=False, locked=True):
        def write_message_unlocked():
            future = super(WSHandler, self).write_message(message, binary)
            # don't yield this future or we're blocking on ourselves!
            raise gen.Return(future)
        if locked:
            with (yield self.write_lock.acquire()):
                write_message_unlocked()
        else:
            write_message_unlocked()
```
And changing to this seems to fix things:
```
    @gen.coroutine
    def write_message(self, message, binary=False, locked=True):
        if locked:
            with (yield self.write_lock.acquire()):
                yield super(WSHandler, self).write_message(message, binary)
        else:
            yield super(WSHandler, self).write_message(message, binary)

    def on_close(self):
```

I'm not sure about the previous comment (I assume from @havocp) not to yield or else we block ourselves, tho. 
Well, with that change and Tornado 4, I see this on too-fast reloads: 
```
Handler or its work threw an exception: StreamClosedError('Stream is closed',): Message 'PULL-DOC-REQ' (revision 1)
Traceback (most recent call last):
  File "/Users/bryanv/work/bokeh/bokeh/server/views/ws.py", line 150, in on_message
    yield self._schedule(work)
  File "/Users/bryanv/anaconda/lib/python3.6/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/Users/bryanv/anaconda/lib/python3.6/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/Users/bryanv/anaconda/lib/python3.6/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "/Users/bryanv/work/bokeh/bokeh/server/views/ws.py", line 226, in _schedule
    yield self.send_message(work)
  File "/Users/bryanv/anaconda/lib/python3.6/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/Users/bryanv/anaconda/lib/python3.6/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/Users/bryanv/anaconda/lib/python3.6/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "/Users/bryanv/work/bokeh/bokeh/server/views/ws.py", line 176, in send_message
    yield message.send(self)
  File "/Users/bryanv/anaconda/lib/python3.6/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/Users/bryanv/anaconda/lib/python3.6/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/Users/bryanv/anaconda/lib/python3.6/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "/Users/bryanv/work/bokeh/bokeh/protocol/message.py", line 235, in send
    yield conn.write_message(self.metadata_json, locked=False)
  File "/Users/bryanv/anaconda/lib/python3.6/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/Users/bryanv/anaconda/lib/python3.6/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/Users/bryanv/anaconda/lib/python3.6/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "/Users/bryanv/work/bokeh/bokeh/server/views/ws.py", line 192, in write_message
    yield super(WSHandler, self).write_message(message, binary)
  File "/Users/bryanv/anaconda/lib/python3.6/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/Users/bryanv/anaconda/lib/python3.6/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
tornado.iostream.StreamClosedError: Stream is closed
```

I'll try to see if there is a place to catch both exceptions. 
This diff seems to function with both Tornado 4 and 5: 
```
diff --git a/bokeh/server/views/ws.py b/bokeh/server/views/ws.py
index f524962cc..c643d9b0a 100644
--- a/bokeh/server/views/ws.py
+++ b/bokeh/server/views/ws.py
@@ -11,7 +11,7 @@ import codecs
 from six.moves.urllib.parse import urlparse

 from tornado import gen, locks
-from tornado.websocket import WebSocketHandler, WebSocketClosedError
+from tornado.websocket import StreamClosedError, WebSocketHandler, WebSocketClosedError

 from ..protocol_handler import ProtocolHandler
 from ...protocol import Protocol
@@ -174,7 +174,7 @@ class WSHandler(WebSocketHandler):
         '''
         try:
             yield message.send(self)
-        except WebSocketClosedError:
+        except (WebSocketClosedError, StreamClosedError):
             # on_close() is / will be called anyway
             log.warn("Failed sending message as connection was closed")
         raise gen.Return(None)
@@ -185,15 +185,11 @@ class WSHandler(WebSocketHandler):
         write lock before writing.

         '''
-        def write_message_unlocked():
-            future = super(WSHandler, self).write_message(message, binary)
-            # don't yield this future or we're blocking on ourselves!
-            raise gen.Return(future)
         if locked:
             with (yield self.write_lock.acquire()):
-                write_message_unlocked()
+                yield super(WSHandler, self).write_message(message, binary)
         else:
-            write_message_unlocked()
+            yield super(WSHandler, self).write_message(message, binary)

     def on_close(self):
         ''' Clean up when the connection is closed.
```

Comments welcome!
That looks good to me. It's not clear to me what purpose `write_message_unlocked` was serving or what that comment was about. 
@bdarnell thanks as always for the help and feedback. I think `write_message_unlocked` was just to encapsulate the two lines of code when previously raising `gen.Return(future)`. Are there any considerations at to be aware of at all doing a `yield` vs raising `gen.Return`? I'll work up a PR today, hoping there's a way to simulate this kind if disconnect so I can put it under test. 
> Are there any considerations at to be aware of at all doing a yield vs raising gen.Return?

There was no yield in `write_message_unlocked`, so the `raise gen.Return(future)` was exactly equivalent to `return future` and the whole thing was equivalent to `super().write_message`. The change is just that now you're waiting on the Future and before you were simply discarding it. 

Resolving patch:
diff --git a/bokeh/server/views/ws.py b/bokeh/server/views/ws.py
--- a/bokeh/server/views/ws.py
+++ b/bokeh/server/views/ws.py
@@ -11,7 +11,7 @@
 from six.moves.urllib.parse import urlparse
 
 from tornado import gen, locks
-from tornado.websocket import WebSocketHandler, WebSocketClosedError
+from tornado.websocket import StreamClosedError, WebSocketHandler, WebSocketClosedError
 
 from ..protocol_handler import ProtocolHandler
 from ...protocol import Protocol
@@ -41,6 +41,19 @@ def initialize(self, application_context, bokeh_websocket_path):
         pass
 
     def check_origin(self, origin):
+        ''' Implement a check_origin policy for Tornado to call.
+
+        The suplied origin will be compared to the Bokeh server whitelist. If the
+        origin is not allow, an error will be logged and ``False`` will be returned.
+
+        Args:
+            origin (str) :
+                The URL of the connection origin
+
+        Returns:
+            bool, True if the connection is allowed, False otherwise
+
+        '''
         from ..util import check_whitelist
         parsed_origin = urlparse(origin)
         origin_host = parsed_origin.netloc.lower()
@@ -59,6 +72,9 @@ def check_origin(self, origin):
     def open(self):
         ''' Initialize a connection to a client.
 
+        Returns:
+            None
+
         '''
         log.info('WebSocket connection opened')
 
@@ -87,11 +103,31 @@ def on_fully_opened(future):
                 log.debug("Failed to fully open connection %r", e)
 
         future = self._async_open(session_id, proto_version)
-        self.application.io_loop.add_future(future,
-                                            on_fully_opened)
+        self.application.io_loop.add_future(future, on_fully_opened)
 
     @gen.coroutine
     def _async_open(self, session_id, proto_version):
+        ''' Perform the specific steps needed to open a connection to a Bokeh session
+
+        Sepcifically, this method coordinates:
+
+        * Getting a session for a session ID (creating a new one if needed)
+        * Creating a protocol receiver and hander
+        * Opening a new ServerConnection and sending it an ACK
+
+        Args:
+            session_id (str) :
+                A session ID to for a session to connect to
+
+                If no session exists with the given ID, a new session is made
+
+            proto_version (str):
+                The protocol version requested by the connecting client.
+
+        Returns:
+            None
+
+        '''
         try:
             yield self.application_context.create_session_if_needed(session_id, self.request)
             session = self.application_context.get_session(session_id)
@@ -120,20 +156,19 @@ def _async_open(self, session_id, proto_version):
     def on_message(self, fragment):
         ''' Process an individual wire protocol fragment.
 
-            The websocket RFC specifies opcodes for distinguishing
-            text frames from binary frames. Tornado passes us either
-            a text or binary string depending on that opcode, we have
-            to look at the type of the fragment to see what we got.
+        The websocket RFC specifies opcodes for distinguishing text frames
+        from binary frames. Tornado passes us either a text or binary string
+        depending on that opcode, we have to look at the type of the fragment
+        to see what we got.
 
         Args:
             fragment (unicode or bytes) : wire fragment to process
 
         '''
 
-        # We shouldn't throw exceptions from on_message because
-        # the caller is just Tornado and it doesn't know what to
-        # do with them other than report them as an unhandled
-        # Future
+        # We shouldn't throw exceptions from on_message because the caller is
+        # just Tornado and it doesn't know what to do with them other than
+        # report them as an unhandled Future
 
         try:
             message = yield self._receive(fragment)
@@ -174,7 +209,7 @@ def send_message(self, message):
         '''
         try:
             yield message.send(self)
-        except WebSocketClosedError:
+        except (WebSocketClosedError, StreamClosedError): # Tornado 4.x may raise StreamClosedError
             # on_close() is / will be called anyway
             log.warn("Failed sending message as connection was closed")
         raise gen.Return(None)
@@ -185,22 +220,17 @@ def write_message(self, message, binary=False, locked=True):
         write lock before writing.
 
         '''
-        def write_message_unlocked():
-            future = super(WSHandler, self).write_message(message, binary)
-            # don't yield this future or we're blocking on ourselves!
-            raise gen.Return(future)
         if locked:
             with (yield self.write_lock.acquire()):
-                write_message_unlocked()
+                yield super(WSHandler, self).write_message(message, binary)
         else:
-            write_message_unlocked()
+            yield super(WSHandler, self).write_message(message, binary)
 
     def on_close(self):
         ''' Clean up when the connection is closed.
 
         '''
-        log.info('WebSocket connection closed: code=%s, reason=%r',
-                 self.close_code, self.close_reason)
+        log.info('WebSocket connection closed: code=%s, reason=%r', self.close_code, self.close_reason)
         if self.connection is not None:
             self.application.client_lost(self.connection)
 


Test patch:
diff --git a/bokeh/server/views/tests/__init__.py b/bokeh/server/views/tests/__init__.py
new file mode 100644
diff --git a/bokeh/server/views/tests/test_ws.py b/bokeh/server/views/tests/test_ws.py
new file mode 100644
--- /dev/null
+++ b/bokeh/server/views/tests/test_ws.py
@@ -0,0 +1,24 @@
+import logging
+
+import pytest
+from tornado.websocket import StreamClosedError, WebSocketClosedError
+
+from bokeh.server.views.ws import WSHandler
+
+from bokeh.util.logconfig import basicConfig
+
+# needed for caplog tests to function
+basicConfig()
+
+@pytest.mark.parametrize('exc', [StreamClosedError, WebSocketClosedError])
+def test_send_message_raises(caplog, exc):
+    class ExcMessage(object):
+        def send(self, handler):
+            raise exc()
+    assert len(caplog.records) == 0
+    with caplog.at_level(logging.WARN):
+        # fake self not great but much easier than setting up a real view
+        ret = WSHandler.send_message("self", ExcMessage())
+        assert len(caplog.records) == 1
+        assert caplog.text.endswith("Failed sending message as connection was closed\n")
+        assert ret.result() is None
diff --git a/tests/examples/test_examples.py b/tests/examples/test_examples.py
--- a/tests/examples/test_examples.py
+++ b/tests/examples/test_examples.py
@@ -1,6 +1,7 @@
 from __future__ import absolute_import, print_function
 
 import os
+from os.path import basename
 import time
 import pytest
 import subprocess
@@ -76,7 +77,7 @@ def test_server_examples(server_example, example, report, bokeh_server):
     for session_callback in list(doc._session_callbacks.keys()):
         doc._remove_session_callback(session_callback)
 
-    session_id = "session_id"
+    session_id = basename(example.path)
     push_session(doc, session_id=session_id)
 
     if example.no_js: