Repository: tornadoweb/tornado

Issue:
asyncio Futures can hang if created from the wrong thread
This snippet: https://gist.github.com/pitrou/d0ed381729ef9045491263774311e9f0 worked before changeset ac13ee5f64cd3bceb1628dbcbbfc77445358038f, but fails afterwards. The reason is `asyncio` sets the `._loop` attribute on its Future objects when said future is created, depending on the current thread's loop. If you create an asyncio Future from a thread and use it from another one, it will never awaken the proper event loop. 

This seems to be biting us hard and is also very hard to debug, unfortunately. @mrocklin



Discussion:
It surprises me that we create Tornado futures on threads that are not running the IOLoop.
Not explicitly so, but we create a `locks.Event` object in our `distributed.Future` constructor, for example. And `locks.Event.__init__` creates a Tornado/asyncio Future for itself.
Interestingly, asyncio implements a protection against that, but it only works if you use `await`. This snippet: https://gist.github.com/pitrou/bd32ec5ac13bacc617b044e366fefe86
raises the following exception:

```
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/home/antoine/cpython/default/Lib/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/home/antoine/cpython/default/Lib/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "aiofutthreads.py", line 14, in run
    loop.run_until_complete(coro(loop))
  File "/home/antoine/cpython/default/Lib/asyncio/base_events.py", line 466, in run_until_complete
    return future.result()
  File "aiofutthreads.py", line 9, in coro
    await fut
RuntimeError: Task <Task pending coro=<coro() running at aiofutthreads.py:9> cb=[_run_until_complete_cb() at /home/antoine/cpython/default/Lib/asyncio/base_events.py:175]> got Future <Future pending> attached to a different loop
```

The discussion on the upstream bug report (https://bugs.python.org/issue31960) made it clear that it's not possible for asyncio to warn in our situation.
This is unfortunate (especially the difficulty of debugging it), but I'm not sure what we can do about it and still achieve our (my) goals for asyncio integration. Tornado 5.0 will unfortunately be disruptive to people using IOLoops on non-main threads (which is a fairly exotic use case). 

This is actually a case where asyncio's restrictive event loop policy could come in handy. If instead of replacing the main-thread-only policy with an any-thread policy, you replaced it with a policy that only created an event loop for your designated event loop thread, you'd get an error when the Future tries to access the wrong thread's event loop.

We could also consider some diagnostics in the python 2 version of tornado.concurrent.Future to help give a warning about this case so there's not a difference between python 2 and 3. 
> This is unfortunate (especially the difficulty of debugging it), but I'm not sure what we can do about it and still achieve our (my) goals for asyncio integration. 

In the Event case, one solution would be to change the implementation and create a separate future for each waiting coroutine, as asyncio does (see https://github.com/python/cpython/blob/master/Lib/asyncio/locks.py#L257-L274). Actually, this is probably the only correct way to do it now that futures can be cancelled.
Oh, that's a good point. We need to audit everything for shared futures now.

Resolving patch:
diff --git a/tornado/concurrent.py b/tornado/concurrent.py
--- a/tornado/concurrent.py
+++ b/tornado/concurrent.py
@@ -337,11 +337,12 @@ def _check_done(self):
 
     def _set_done(self):
         self._done = True
-        from tornado.ioloop import IOLoop
-        loop = IOLoop.current()
-        for cb in self._callbacks:
-            loop.add_callback(cb, self)
-        self._callbacks = None
+        if self._callbacks:
+            from tornado.ioloop import IOLoop
+            loop = IOLoop.current()
+            for cb in self._callbacks:
+                loop.add_callback(cb, self)
+            self._callbacks = None
 
     # On Python 3.3 or older, objects with a destructor part of a reference
     # cycle are never destroyed. It's no longer the case on Python 3.4 thanks to
diff --git a/tornado/locks.py b/tornado/locks.py
--- a/tornado/locks.py
+++ b/tornado/locks.py
@@ -15,6 +15,7 @@
 from __future__ import absolute_import, division, print_function
 
 import collections
+from concurrent.futures import CancelledError
 
 from tornado import gen, ioloop
 from tornado.concurrent import Future, future_set_result_unless_cancelled
@@ -196,7 +197,8 @@ def runner():
         Done
     """
     def __init__(self):
-        self._future = Future()
+        self._value = False
+        self._waiters = set()
 
     def __repr__(self):
         return '<%s %s>' % (
@@ -204,23 +206,26 @@ def __repr__(self):
 
     def is_set(self):
         """Return ``True`` if the internal flag is true."""
-        return self._future.done()
+        return self._value
 
     def set(self):
         """Set the internal flag to ``True``. All waiters are awakened.
 
         Calling `.wait` once the flag is set will not block.
         """
-        if not self._future.done():
-            self._future.set_result(None)
+        if not self._value:
+            self._value = True
+
+            for fut in self._waiters:
+                if not fut.done():
+                    fut.set_result(None)
 
     def clear(self):
         """Reset the internal flag to ``False``.
 
         Calls to `.wait` will block until `.set` is called.
         """
-        if self._future.done():
-            self._future = Future()
+        self._value = False
 
     def wait(self, timeout=None):
         """Block until the internal flag is true.
@@ -228,10 +233,21 @@ def wait(self, timeout=None):
         Returns a Future, which raises `tornado.util.TimeoutError` after a
         timeout.
         """
+        fut = Future()
+        if self._value:
+            fut.set_result(None)
+            return fut
+        self._waiters.add(fut)
+        fut.add_done_callback(lambda fut: self._waiters.remove(fut))
         if timeout is None:
-            return self._future
+            return fut
         else:
-            return gen.with_timeout(timeout, self._future)
+            timeout_fut = gen.with_timeout(timeout, fut, quiet_exceptions=(CancelledError,))
+            # This is a slightly clumsy workaround for the fact that
+            # gen.with_timeout doesn't cancel its futures. Cancelling
+            # fut will remove it from the waiters list.
+            timeout_fut.add_done_callback(lambda tf: fut.cancel() if not fut.done() else None)
+            return timeout_fut
 
 
 class _ReleasingContextManager(object):
diff --git a/tornado/websocket.py b/tornado/websocket.py
--- a/tornado/websocket.py
+++ b/tornado/websocket.py
@@ -237,6 +237,7 @@ def write_message(self, message, binary=False):
         is allowed.
 
         If the connection is already closed, raises `WebSocketClosedError`.
+        Returns a `.Future` which can be used for flow control.
 
         .. versionchanged:: 3.2
            `WebSocketClosedError` was added (previously a closed connection
@@ -244,6 +245,10 @@ def write_message(self, message, binary=False):
 
         .. versionchanged:: 4.3
            Returns a `.Future` which can be used for flow control.
+
+        .. versionchanged:: 5.0
+           Consistently raises `WebSocketClosedError`. Previously could
+           sometimes raise `.StreamClosedError`.
         """
         if self.ws_connection is None:
             raise WebSocketClosedError()
@@ -788,7 +793,23 @@ def write_message(self, message, binary=False):
         if self._compressor:
             message = self._compressor.compress(message)
             flags |= self.RSV1
-        return self._write_frame(True, opcode, message, flags=flags)
+        # For historical reasons, write methods in Tornado operate in a semi-synchronous
+        # mode in which awaiting the Future they return is optional (But errors can
+        # still be raised). This requires us to go through an awkward dance here
+        # to transform the errors that may be returned while presenting the same
+        # semi-synchronous interface.
+        try:
+            fut = self._write_frame(True, opcode, message, flags=flags)
+        except StreamClosedError:
+            raise WebSocketClosedError()
+
+        @gen.coroutine
+        def wrapper():
+            try:
+                yield fut
+            except StreamClosedError:
+                raise WebSocketClosedError()
+        return wrapper()
 
     def write_ping(self, data):
         """Send ping frame."""
@@ -1144,8 +1165,16 @@ def headers_received(self, start_line, headers):
         future_set_result_unless_cancelled(self.connect_future, self)
 
     def write_message(self, message, binary=False):
-        """Sends a message to the WebSocket server."""
-        return self.protocol.write_message(message, binary)
+        """Sends a message to the WebSocket server.
+
+        If the stream is closed, raises `WebSocketClosedError`.
+        Returns a `.Future` which can be used for flow control.
+
+        .. versionchanged:: 5.0
+           Exception raised on a closed stream changed from `.StreamClosedError`
+           to `WebSocketClosedError`.
+        """
+        return self.protocol.write_message(message, binary=binary)
 
     def read_message(self, callback=None):
         """Reads a message from the WebSocket server.


Test patch:
diff --git a/maint/test/cython/tox.ini b/maint/test/cython/tox.ini
--- a/maint/test/cython/tox.ini
+++ b/maint/test/cython/tox.ini
@@ -1,6 +1,6 @@
 [tox]
 # This currently segfaults on pypy.
-envlist = py27,py34,py35,py36
+envlist = py27,py35,py36
 
 [testenv]
 deps =
@@ -13,6 +13,5 @@ commands = python -m unittest cythonapp_test
 # defaults for the others.
 basepython =
            py27: python2.7
-           py34: python3.4
            py35: python3.5
            py36: python3.6
diff --git a/tornado/test/asyncio_test.py b/tornado/test/asyncio_test.py
--- a/tornado/test/asyncio_test.py
+++ b/tornado/test/asyncio_test.py
@@ -47,7 +47,8 @@ def test_asyncio_future(self):
         if hasattr(asyncio, 'ensure_future'):
             ensure_future = asyncio.ensure_future
         else:
-            ensure_future = asyncio.async
+            # async is a reserved word in Python 3.7
+            ensure_future = getattr(asyncio, 'async')
 
         x = yield ensure_future(
             asyncio.get_event_loop().run_in_executor(None, lambda: 42))
diff --git a/tornado/test/gen_test.py b/tornado/test/gen_test.py
--- a/tornado/test/gen_test.py
+++ b/tornado/test/gen_test.py
@@ -1069,6 +1069,20 @@ def f():
         actual = repr(coro)
         self.assertIn(expected, actual)
 
+    @unittest.skipIf(asyncio is None, "asyncio module not present")
+    @gen_test
+    def test_asyncio_gather(self):
+        # This demonstrates that tornado coroutines can be understood
+        # by asyncio (This failed prior to Tornado 5.0).
+        @gen.coroutine
+        def f():
+            yield gen.moment
+            raise gen.Return(1)
+
+        ret = yield asyncio.gather(f(), f())
+        self.assertEqual(ret, [1, 1])
+        self.finished = True
+
 
 class GenSequenceHandler(RequestHandler):
     @asynchronous
diff --git a/tornado/test/template_test.py b/tornado/test/template_test.py
--- a/tornado/test/template_test.py
+++ b/tornado/test/template_test.py
@@ -6,8 +6,8 @@
 
 from tornado.escape import utf8, native_str, to_unicode
 from tornado.template import Template, DictLoader, ParseError, Loader
-from tornado.test.util import unittest, is_coverage_running
-from tornado.util import ObjectDict, unicode_type, PY3
+from tornado.test.util import unittest
+from tornado.util import ObjectDict, unicode_type
 
 
 class TemplateTest(unittest.TestCase):
@@ -175,11 +175,6 @@ def test_no_inherit_future(self):
         self.assertEqual(template.generate(), '0')
 
     def test_non_ascii_name(self):
-        if PY3 and is_coverage_running():
-            try:
-                os.fsencode(u"t\u00e9st.html")
-            except UnicodeEncodeError:
-                self.skipTest("coverage tries to access unencodable filename")
         loader = DictLoader({u"t\u00e9st.html": "hello"})
         self.assertEqual(loader.load(u"t\u00e9st.html").generate(), b"hello")
 
diff --git a/tornado/test/util.py b/tornado/test/util.py
--- a/tornado/test/util.py
+++ b/tornado/test/util.py
@@ -97,24 +97,6 @@ def exec_test(caller_globals, caller_locals, s):
     return local_namespace
 
 
-def is_coverage_running():
-    """Return whether coverage is currently running.
-    """
-    if 'coverage' not in sys.modules:
-        return False
-    tracer = sys.gettrace()
-    if tracer is None:
-        return False
-    try:
-        mod = tracer.__module__
-    except AttributeError:
-        try:
-            mod = tracer.__class__.__module__
-        except AttributeError:
-            return False
-    return mod.startswith('coverage')
-
-
 def subTest(test, *args, **kwargs):
     """Compatibility shim for unittest.TestCase.subTest.
 
diff --git a/tornado/test/websocket_test.py b/tornado/test/websocket_test.py
--- a/tornado/test/websocket_test.py
+++ b/tornado/test/websocket_test.py
@@ -7,7 +7,6 @@
 from tornado.concurrent import Future
 from tornado import gen
 from tornado.httpclient import HTTPError, HTTPRequest
-from tornado.iostream import StreamClosedError
 from tornado.log import gen_log, app_log
 from tornado.template import DictLoader
 from tornado.testing import AsyncHTTPTestCase, gen_test, bind_unused_port, ExpectLog
@@ -25,7 +24,7 @@
     traceback.print_exc()
     raise
 
-from tornado.websocket import WebSocketHandler, websocket_connect, WebSocketError
+from tornado.websocket import WebSocketHandler, websocket_connect, WebSocketError, WebSocketClosedError
 
 try:
     from tornado import speedups
@@ -50,10 +49,11 @@ def on_close(self):
 
 
 class EchoHandler(TestWebSocketHandler):
+    @gen.coroutine
     def on_message(self, message):
         try:
-            self.write_message(message, isinstance(message, bytes))
-        except StreamClosedError:
+            yield self.write_message(message, isinstance(message, bytes))
+        except WebSocketClosedError:
             pass
 
 
@@ -342,7 +342,7 @@ def test_write_after_close(self):
         ws = yield self.ws_connect('/close_reason')
         msg = yield ws.read_message()
         self.assertIs(msg, None)
-        with self.assertRaises(StreamClosedError):
+        with self.assertRaises(WebSocketClosedError):
             ws.write_message('hello')
 
     @gen_test