Repository: tornadoweb/tornado

Issue:
Check _events and _handlers before start()
```
Traceback (most recent call last):
    File "/usr/local/lib/python2.7/site-packages/tornado/ioloop.py", line 887, in start
     fd_obj, handler_func = self._handlers[fd]
     KeyError: 16
```

https://github.com/tornadoweb/tornado/blob/master/tornado/ioloop.py#L883

I ran into this issue when testing using multiple processes and threads.  We sandbox our test processes using multiprocessing and were getting this cryptic error.

I found a blog post @toumorokoshi wrote going over his own experiences with this issue: http://y.tsutsumi.io/keyerror-in-self_handlers-a-journey-deep-into-tornados-internals.html

At the very least it would be nice to have some documentation suggesting that you run IOLoop.clear_instance() and IOLoop.clear_current() after a fork()


Discussion:
This also appears to be related to https://github.com/tornadoweb/tornado/pull/1214 which was closed as there was not enough information to reproduce at the time.

This ticket is also a duplicate of https://github.com/tornadoweb/tornado/issues/1103
What do you mean by "checking" these fields before `start()`? If you're forking with a live IOLoop, I don't think there is any way we can make this work. (we *might* be able to provide better diagnostic messages, but I'm not sure)

> At the very least it would be nice to have some documentation suggesting that you run IOLoop.clear_instance() and IOLoop.clear_current() after a fork()

`tornado.process.fork_processes` has a safeguard for this (it disallows forking once an IOLoop has been created). Bare `os.fork()` is not recommended in Tornado. 
Our use case is sandboxing tornado in a multiprocessing.Process so we can run tests against a simulated environment.

A diagnostic message would be fine.  Something that tells the user "you are not using the ioloop correctly".
When you said 'check _events and _handlers before start()`, did you have something in mind for that check? I'm not seeing how those attributes could be used to detect this kind of improper use. 

I think we could store a process id in `IOLoop__init__` and verify that we're in the same process in `start()` (and maybe a thread id, although this would potentially break some things that are working now). 
The pid check makes sense.  Perhaps add a logging.warn(...) if the check fails?  That way no existing functionality is broken.
I would argue to make the documentation around this caveat a bit more pronounced. I had a lot of trouble finding a section that mentions this issue in the documentation:

http://www.tornadoweb.org/en/stable/guide/running.html?highlight=fork#processes-and-ports

Maybe a comment or two closer to the initial guide? just a warning that says "do not call start or run_sync on an IOLoop before forking" would have probably helped.

Regarding fork_processes: I ran into my situation because I was using gunicorn. I prefer it due to it's ability to better manage cycling and shutdowns. 

Resolving patch:
diff --git a/tornado/ioloop.py b/tornado/ioloop.py
--- a/tornado/ioloop.py
+++ b/tornado/ioloop.py
@@ -121,6 +121,16 @@ def connection_ready(sock, fd, events):
     current instance. If ``make_current=False``, the new `IOLoop` will
     not try to become current.
 
+    In general, an `IOLoop` cannot survive a fork or be shared across
+    processes in any way. When multiple processes are being used, each
+    process should create its own `IOLoop`, which also implies that
+    any objects which depend on the `IOLoop` (such as
+    `.AsyncHTTPClient`) must also be created in the child processes.
+    As a guideline, anything that starts processes (including the
+    `tornado.process` and `multiprocessing` modules) should do so as
+    early as possible, ideally the first thing the application does
+    after loading its configuration in ``main()``.
+
     .. versionchanged:: 4.2
        Added the ``make_current`` keyword argument to the `IOLoop`
        constructor.
@@ -701,6 +711,7 @@ def initialize(self, impl, time_func=None, **kwargs):
         self._stopped = False
         self._closing = False
         self._thread_ident = None
+        self._pid = os.getpid()
         self._blocking_signal_threshold = None
         self._timeout_counter = itertools.count()
 
@@ -753,6 +764,8 @@ def set_blocking_signal_threshold(self, seconds, action):
     def start(self):
         if self._running:
             raise RuntimeError("IOLoop is already running")
+        if os.getpid() != self._pid:
+            raise RuntimeError("Cannot share PollIOLoops across processes")
         self._setup_logging()
         if self._stopped:
             self._stopped = False


Test patch: