Repository: tornadoweb/tornado

Issue:
KeyError when closing IOLoop
This started showing up in Dask's test suite recently:

```python-traceback
distributed/utils_test.py:144: in pristine_loop
    loop.close(all_fds=True)
../../Software/anaconda/envs/test-environment/lib/python3.6/site-packages/tornado/platform/asyncio.py:223: in close
    super(AsyncIOLoop, self).close(all_fds=all_fds)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <tornado.platform.asyncio.AsyncIOLoop object at 0x7f5751d46eb8>, all_fds = True

    def close(self, all_fds=False):
        self.closing = True
        for fd in list(self.handlers):
            fileobj, handler_func = self.handlers[fd]
            self.remove_handler(fd)
            if all_fds:
                self.close_fd(fileobj)
        self.asyncio_loop.close()
>       del IOLoop._ioloop_for_asyncio[self.asyncio_loop]
E       KeyError: <_UnixSelectorEventLoop running=False closed=True debug=False>
```

This is likely due to some change in upstream dependencies.  It looks like Tornado hasn't had a release during the time when this arose, so it's likely something else.  Still, I thought I'd raise the issue.


Discussion:
That line is new in Tornado 5.0.2, so this is almost certainly related to that upgrade (this was added to improve compatibility with `pytest-asyncio`, although my actual testing of that combination was limited).

This is either about mismatched calls to `initialize()` and `close()` or some sort of situation in which you're explicitly closing both the `asyncio` loop and the tornado loop but in an order that the code isn't expecting. I suspect the `del` line here needs to be wrapped in an `if not self.asyncio_loop.is_closed()` or something like that. 
The same error occurred in this CI build: https://travis-ci.org/tornadoweb/tornado/jobs/381213596

It's in `test_add_callback_while_closing`. This test does use threads (in a slightly dodgy way), but it doesn't have mismatched initialize/close calls as speculated in my previous comment. 

Resolving patch:
diff --git a/tornado/platform/asyncio.py b/tornado/platform/asyncio.py
--- a/tornado/platform/asyncio.py
+++ b/tornado/platform/asyncio.py
@@ -62,8 +62,13 @@ def close(self, all_fds=False):
             self.remove_handler(fd)
             if all_fds:
                 self.close_fd(fileobj)
-        self.asyncio_loop.close()
+        # Remove the mapping before closing the asyncio loop. If this
+        # happened in the other order, we could race against another
+        # initialize() call which would see the closed asyncio loop,
+        # assume it was closed from the asyncio side, and do this
+        # cleanup for us, leading to a KeyError.
         del IOLoop._ioloop_for_asyncio[self.asyncio_loop]
+        self.asyncio_loop.close()
 
     def add_handler(self, fd, handler, events):
         fd, fileobj = self.split_fd(fd)


Test patch:
diff --git a/tornado/test/ioloop_test.py b/tornado/test/ioloop_test.py
--- a/tornado/test/ioloop_test.py
+++ b/tornado/test/ioloop_test.py
@@ -458,6 +458,16 @@ def handle_read(fd, events):
             client.close()
             server.close()
 
+    @gen_test
+    def test_init_close_race(self):
+        # Regression test for #2367
+        def f():
+            for i in range(10):
+                loop = IOLoop()
+                loop.close()
+
+        yield gen.multi([self.io_loop.run_in_executor(None, f) for i in range(2)])
+
 
 # Deliberately not a subclass of AsyncTestCase so the IOLoop isn't
 # automatically set as current.
diff --git a/tornado/test/simple_httpclient_test.py b/tornado/test/simple_httpclient_test.py
--- a/tornado/test/simple_httpclient_test.py
+++ b/tornado/test/simple_httpclient_test.py
@@ -21,7 +21,7 @@
 from tornado.log import gen_log
 from tornado.concurrent import Future
 from tornado.netutil import Resolver, bind_sockets
-from tornado.simple_httpclient import SimpleAsyncHTTPClient, HTTPStreamClosedError
+from tornado.simple_httpclient import SimpleAsyncHTTPClient, HTTPStreamClosedError, HTTPTimeoutError
 from tornado.test.httpclient_test import ChunkHandler, CountdownHandler, HelloWorldHandler, RedirectHandler  # noqa: E501
 from tornado.test import httpclient_test
 from tornado.testing import (AsyncHTTPTestCase, AsyncHTTPSTestCase, AsyncTestCase,
@@ -673,10 +673,11 @@ def test_port_mapping(self):
 
 class ResolveTimeoutTestCase(AsyncHTTPTestCase):
     def setUp(self):
-        # Dummy Resolver subclass that never invokes its callback.
+        # Dummy Resolver subclass that never finishes.
         class BadResolver(Resolver):
+            @gen.coroutine
             def resolve(self, *args, **kwargs):
-                pass
+                yield Event().wait()
 
         super(ResolveTimeoutTestCase, self).setUp()
         self.http_client = SimpleAsyncHTTPClient(
@@ -686,7 +687,7 @@ def get_app(self):
         return Application([url("/hello", HelloWorldHandler), ])
 
     def test_resolve_timeout(self):
-        with self.assertRaises(TypeError):
+        with self.assertRaises(HTTPTimeoutError):
             self.fetch('/hello', connect_timeout=0.1, raise_error=True)