Repository: tornadoweb/tornado

Issue:
Runner objects can get GCed unexpectedly
Hi,

This is probably a little unusual and I am not sure I understand the whole object graph, but I believe I have reduced my production issue to this test case.

If a coroutine is waiting on a future, and the only hard references to that future are entirely inside the body of the inner generator of the coroutine, a garbage collection run can sweep away the in-progress coroutine and its runner object, leave the coroutine's future unresolved forever, and in some cases leave the application in a stuck state.

Ran on Python 3.4 with Tornado 4.3.

``` python
from tornado.ioloop import IOLoop
from tornado.gen import Future, coroutine
import weakref
import gc

wr_fut = None

def count_runners():
    runners = len([
        x for x in gc.get_objects()
        if x.__class__.__name__ == "Runner"
    ])
    print("There are",runners,"Runners")

def resolve_fut():
    count_runners()
    gc.collect(2)
    count_runners()
    fut = wr_fut()
    if not fut:
        print('failure! The app is going to hang forever now')
    else:
        fut.set_result(1)

@coroutine
def main():
    global wr_fut
    my_fut = Future()
    wr_fut = weakref.ref(my_fut)
    IOLoop.current().add_callback(resolve_fut)
    yield my_fut
    print('success!')

IOLoop.current().run_sync(main)
```

This program hangs forever despite using run_sync. That's because the future returned by the coroutine in main() still exists, it just no longer has a Runner object to actually make the future do anything useful. This is _very_ surprising behavior. I would expect the generator/coroutine/Runner/frame objects to be alive for as long as {{run_sync}} is waiting.

Also, note that explicit call to `gc.collect(2)` ; this bug actually requires the garbage collector to run for the bug to trigger, which made this bug _very_ difficult to track down.

I have managed hack tornado into making it so that the Runner cannot die while its Future is alive which causes my test case to work:

``` python
# gen.py line 1241 tornado 4.3
                except (StopIteration, Return) as e:
                    future.set_result(_value_from_stopiteration(e))
                except Exception:
                    future.set_exc_info(sys.exc_info())
                else:
                    # this is the original code below:
                    #Runner(result, future, yielded)
                    # this hack makes the Runner not die and my test case work
                    future._the_runner = Runner(result, future, yielded)
                try:
                    return future
```

So the real-world scenario: I have a proprietary networking library that is using weakrefs for event callbacks and a wrapper library that converts these events into futures. Because I have gone out of my way to avoid creating garbage collection cycles (so that my network client can be GCed when it is no longer in use, unlike a lot of async clients available to tornado), it appears I have discovered this issue.

What is the most appropriate bugfix for this? I personally feel that Runner objects should have their lifecycle tied directly to the future they're driving so that Runners cannot die while their future is alive. However, an argument could also be made that Runner should have a `__del__` method that marks the future as canceled (or `.set_exception(RuntimeError("GCed"))`) instead so that developers can more clearly see that they hit this failure mode.

Please don't tell me the answer is "don't use weakrefs"! I believe it may be possible to create a test case that presents the same problem without using weakrefs but it would have to abuse certain behaviors of sockets, filenos and IOLoop's fd handlers to more closely emulate the actual workings of my network library.

Runner objects can get GCed unexpectedly
Hi,

This is probably a little unusual and I am not sure I understand the whole object graph, but I believe I have reduced my production issue to this test case.

If a coroutine is waiting on a future, and the only hard references to that future are entirely inside the body of the inner generator of the coroutine, a garbage collection run can sweep away the in-progress coroutine and its runner object, leave the coroutine's future unresolved forever, and in some cases leave the application in a stuck state.

Ran on Python 3.4 with Tornado 4.3.

``` python
from tornado.ioloop import IOLoop
from tornado.gen import Future, coroutine
import weakref
import gc

wr_fut = None

def count_runners():
    runners = len([
        x for x in gc.get_objects()
        if x.__class__.__name__ == "Runner"
    ])
    print("There are",runners,"Runners")

def resolve_fut():
    count_runners()
    gc.collect(2)
    count_runners()
    fut = wr_fut()
    if not fut:
        print('failure! The app is going to hang forever now')
    else:
        fut.set_result(1)

@coroutine
def main():
    global wr_fut
    my_fut = Future()
    wr_fut = weakref.ref(my_fut)
    IOLoop.current().add_callback(resolve_fut)
    yield my_fut
    print('success!')

IOLoop.current().run_sync(main)
```

This program hangs forever despite using run_sync. That's because the future returned by the coroutine in main() still exists, it just no longer has a Runner object to actually make the future do anything useful. This is _very_ surprising behavior. I would expect the generator/coroutine/Runner/frame objects to be alive for as long as {{run_sync}} is waiting.

Also, note that explicit call to `gc.collect(2)` ; this bug actually requires the garbage collector to run for the bug to trigger, which made this bug _very_ difficult to track down.

I have managed hack tornado into making it so that the Runner cannot die while its Future is alive which causes my test case to work:

``` python
# gen.py line 1241 tornado 4.3
                except (StopIteration, Return) as e:
                    future.set_result(_value_from_stopiteration(e))
                except Exception:
                    future.set_exc_info(sys.exc_info())
                else:
                    # this is the original code below:
                    #Runner(result, future, yielded)
                    # this hack makes the Runner not die and my test case work
                    future._the_runner = Runner(result, future, yielded)
                try:
                    return future
```

So the real-world scenario: I have a proprietary networking library that is using weakrefs for event callbacks and a wrapper library that converts these events into futures. Because I have gone out of my way to avoid creating garbage collection cycles (so that my network client can be GCed when it is no longer in use, unlike a lot of async clients available to tornado), it appears I have discovered this issue.

What is the most appropriate bugfix for this? I personally feel that Runner objects should have their lifecycle tied directly to the future they're driving so that Runners cannot die while their future is alive. However, an argument could also be made that Runner should have a `__del__` method that marks the future as canceled (or `.set_exception(RuntimeError("GCed"))`) instead so that developers can more clearly see that they hit this failure mode.

Please don't tell me the answer is "don't use weakrefs"! I believe it may be possible to create a test case that presents the same problem without using weakrefs but it would have to abuse certain behaviors of sockets, filenos and IOLoop's fd handlers to more closely emulate the actual workings of my network library.



Discussion:
Apologies, but _my_ answer is, "Don't use weakrefs." After all, a weakref's purpose is precisely to allow the object tree it references to be garbage-collected. It's not Tornado's responsibility to sabotage code that uses weakrefs for this purpose. If you can somehow arrange for the weakref to be converted to a strong ref before the future is yielded, like:

```
fut = wr_fut()
```

... then that is the right and proper solution.

To me, the fact that the only thing that keeps the `Runner` alive is the `Future` object yielded inside the coroutine is very surprising. In this situation, the object graph of hard references flows in almost the exact opposite way as it would if main were not a coroutine: Instead of my executing code context and stack frames keeping `main`'s body alive, a stand-in result object from the complete opposite part of the system keeps it alive. I would expect that the future that represents the result of the coroutine to be what keeps the `Runner` object alive.

Instead of `Runner` instances floating in space like they do now, maybe they should be inserted into a WeakKeyDictionary inside the IOLoop where the key is the `Runner`'s `Future` object and value is the `Runner`. That way the `Runner` of the future will always survive as long as the actual future does. I believe the maintenance overhead of keeping the dictionary in a good state is a small price, and a very small price to avoid surprises like this one.

@AaronOpfer Your expectations are perfectly reasonable; the inverted object graph is surprising. It's burned me too; see #1429 for more analysis. 

So far we've handled this on a case-by-case basis (e.g. in WaitIterator). Maybe a more general solution is called for if application code can run into this. However, it's important that the link that is added from the Future to the Runner is also removed when it is no longer needed (so we don't move large parts of the object graph from the fast refcounting path to the slow cyclic GC). I'm not sure if WeakKeyDictionary has the desired properties or not. It might be better to put an attribute directly on the Future, as long as we can clear it when the Runner finishes.

I believe [`WeakKeyDictionary`](https://docs.python.org/3/library/weakref.html#weakref.WeakKeyDictionary) is tailor made for this situation:

> Entries in the dictionary will be discarded when there is no longer a strong reference to the key. [WeakKeyDictionary] can be used to associate additional data with an object owned by other parts of an application without adding attributes to those objects.

Furthermore, since the `Runner` itself knows when the future is going to be resolved, it could manually clean itself out of the `WeakKeyDictionary` at that time. This would be a good idea in case something is holding onto the Future for a long time for some reason (maybe with yield [...] syntax or something).

So that fixes both cases: when the future is GCed we clean the runner and when the Runner is done we also clear the runner.

Are Tornado Developers interested in developing a fix for this? I could also develop a fix. Just trying to prevent duplication of work.

I won't have time to look at this in the near future, so I don't think you'll be duplicating work if you want to pick it up. 

Apologies, but _my_ answer is, "Don't use weakrefs." After all, a weakref's purpose is precisely to allow the object tree it references to be garbage-collected. It's not Tornado's responsibility to sabotage code that uses weakrefs for this purpose. If you can somehow arrange for the weakref to be converted to a strong ref before the future is yielded, like:

```
fut = wr_fut()
```

... then that is the right and proper solution.

To me, the fact that the only thing that keeps the `Runner` alive is the `Future` object yielded inside the coroutine is very surprising. In this situation, the object graph of hard references flows in almost the exact opposite way as it would if main were not a coroutine: Instead of my executing code context and stack frames keeping `main`'s body alive, a stand-in result object from the complete opposite part of the system keeps it alive. I would expect that the future that represents the result of the coroutine to be what keeps the `Runner` object alive.

Instead of `Runner` instances floating in space like they do now, maybe they should be inserted into a WeakKeyDictionary inside the IOLoop where the key is the `Runner`'s `Future` object and value is the `Runner`. That way the `Runner` of the future will always survive as long as the actual future does. I believe the maintenance overhead of keeping the dictionary in a good state is a small price, and a very small price to avoid surprises like this one.

@AaronOpfer Your expectations are perfectly reasonable; the inverted object graph is surprising. It's burned me too; see #1429 for more analysis. 

So far we've handled this on a case-by-case basis (e.g. in WaitIterator). Maybe a more general solution is called for if application code can run into this. However, it's important that the link that is added from the Future to the Runner is also removed when it is no longer needed (so we don't move large parts of the object graph from the fast refcounting path to the slow cyclic GC). I'm not sure if WeakKeyDictionary has the desired properties or not. It might be better to put an attribute directly on the Future, as long as we can clear it when the Runner finishes.

I believe [`WeakKeyDictionary`](https://docs.python.org/3/library/weakref.html#weakref.WeakKeyDictionary) is tailor made for this situation:

> Entries in the dictionary will be discarded when there is no longer a strong reference to the key. [WeakKeyDictionary] can be used to associate additional data with an object owned by other parts of an application without adding attributes to those objects.

Furthermore, since the `Runner` itself knows when the future is going to be resolved, it could manually clean itself out of the `WeakKeyDictionary` at that time. This would be a good idea in case something is holding onto the Future for a long time for some reason (maybe with yield [...] syntax or something).

So that fixes both cases: when the future is GCed we clean the runner and when the Runner is done we also clear the runner.

Are Tornado Developers interested in developing a fix for this? I could also develop a fix. Just trying to prevent duplication of work.

I won't have time to look at this in the near future, so I don't think you'll be duplicating work if you want to pick it up. 


Resolving patch:
diff --git a/tornado/gen.py b/tornado/gen.py
--- a/tornado/gen.py
+++ b/tornado/gen.py
@@ -83,6 +83,7 @@ def get(self):
 import sys
 import textwrap
 import types
+import weakref
 
 from tornado.concurrent import Future, TracebackFuture, is_future, chain_future
 from tornado.ioloop import IOLoop
@@ -244,6 +245,24 @@ def coroutine(func, replace_callback=True):
     """
     return _make_coroutine_wrapper(func, replace_callback=True)
 
+# Ties lifetime of runners to their result futures. Github Issue #1769
+# Generators, like any object in Python, must be strong referenced
+# in order to not be cleaned up by the garbage collector. When using
+# coroutines, the Runner object is what strong-refs the inner
+# generator. However, the only item that strong-reffed the Runner
+# was the last Future that the inner generator yielded (via the
+# Future's internal done_callback list). Usually this is enough, but
+# it is also possible for this Future to not have any strong references
+# other than other objects referenced by the Runner object (usually
+# when using other callback patterns and/or weakrefs). In this
+# situation, if a garbage collection ran, a cycle would be detected and
+# Runner objects could be destroyed along with their inner generators
+# and everything in their local scope.
+# This map provides strong references to Runner objects as long as
+# their result future objects also have strong references (typically
+# from the parent coroutine's Runner). This keeps the coroutine's
+# Runner alive.
+_futures_to_runners = weakref.WeakKeyDictionary()
 
 def _make_coroutine_wrapper(func, replace_callback):
     """The inner workings of ``@gen.coroutine`` and ``@gen.engine``.
@@ -294,7 +313,7 @@ def wrapper(*args, **kwargs):
                 except Exception:
                     future.set_exc_info(sys.exc_info())
                 else:
-                    Runner(result, future, yielded)
+                    _futures_to_runners[future] = Runner(result, future, yielded)
                 try:
                     return future
                 finally:


Test patch:
diff --git a/tornado/test/gen_test.py b/tornado/test/gen_test.py
--- a/tornado/test/gen_test.py
+++ b/tornado/test/gen_test.py
@@ -1,5 +1,6 @@
 from __future__ import absolute_import, division, print_function, with_statement
 
+import gc
 import contextlib
 import datetime
 import functools
@@ -25,7 +26,6 @@
 except ImportError:
     futures = None
 
-
 class GenEngineTest(AsyncTestCase):
     def setUp(self):
         super(GenEngineTest, self).setUp()
@@ -1368,5 +1368,28 @@ def test_no_ref(self):
                                gen.WaitIterator(gen.sleep(0)).next())
 
 
+class RunnerGCTest(AsyncTestCase):
+    """Github issue 1769: Runner objects can get GCed unexpectedly"""
+    @gen_test
+    def test_gc(self):
+        """Runners shouldn't GC if future is alive"""
+        # Create the weakref
+        weakref_scope = [None]
+        def callback():
+            gc.collect(2)
+            weakref_scope[0]().set_result(123)
+
+        @gen.coroutine
+        def tester():
+            fut = Future()
+            weakref_scope[0] = weakref.ref(fut)
+            self.io_loop.add_callback(callback)
+            yield fut
+
+        yield gen.with_timeout(
+            datetime.timedelta(seconds=0.2),
+            tester()
+        )
+
 if __name__ == '__main__':
     unittest.main()