Repository: twisted/twisted

Issue:
There is a persistent reference cycle in deferLater().
|[<img alt="JCoder3's avatar" src="https://avatars.githubusercontent.com/u/46210275?s=50" width="50" height="50">](https://github.com/JCoder3)| @JCoder3 reported|
|-|-|
|Trac ID|trac#10310|
|Type|defect|
|Created|2022-02-16 06:26:36Z|

A TCP server port is used to listen for connections.

When port.stopListening() (in twisted\internet\tcp.py) is called, there is memory leak.

It's due to a cyclic reference in deferLater() (in twisted\internet\task.py).

It could be solved by enforcing
**deferred._canceller = None** 
where it is returned.

Maybe it's better to have a method for clearing deferred._canceller so as to break the reference loop.

Attachments:

* [cyclic_reference.JPG](https://raw.githubusercontent.com/twisted/twistedmatrix.com-trac-attachments/trunk/ticket/5c2/5c202c1f703b4df85bd3f0123eb1c4038ade3e14/6ad32adb0c7bac520c28ce51c5eb60b7c7501d25.JPG) (80191 bytes) - added by JCoder3 on 2022-02-16 06:29:09Z - the reference cycle

<details><summary>Searchable metadata</summary>

```
trac-id__10310 10310
type__defect defect
reporter__JCoder3 JCoder3
priority__normal normal
milestone__None None
branch__ 
branch_author__ 
status__new new
resolution__None None
component__core core
keywords__None None
time__1644992796628301 1644992796628301
changetime__1646083036271745 1646083036271745
version__None None
owner__JCoder3 JCoder3
cc__TBBle
```
</details>



Discussion:
|[<img alt="exarkun's avatar" src="https://avatars.githubusercontent.com/u/254565?s=50" width="50" height="50">](https://github.com/exarkun)|@exarkun set owner to @JCoder3|
|-|-|

|[<img alt="exarkun's avatar" src="https://avatars.githubusercontent.com/u/254565?s=50" width="50" height="50">](https://github.com/exarkun)<a name="note_2"></a>|@exarkun commented|
|-|-|

A reference cycle is not a memory leak.  Python has a garbage collector that can collect reference cycles.

Is there some reason it cannot collect this cycle?

|[<img alt="JCoder3's avatar" src="https://avatars.githubusercontent.com/u/46210275?s=50" width="50" height="50">](https://github.com/JCoder3)<a name="note_1"></a>|@JCoder3 commented|
|-|-|

Or set _canceller to be None after callback() is called, as there is no need to cancel deferred after it is fired.

In deferred.cancel(), it will invoke DelayedCall.cancel() which will raise error.AlreadyCalled after d.callback is triggered by the timer/delay. 
Because in Port.loseconnection(), twisted\internet\tcp.py,

self.deferred = deferLater( self.reactor, **0**, self.connectionLost, connDone) .
|[<img alt="TBBle's avatar" src="https://avatars.githubusercontent.com/u/138397?s=50" width="50" height="50">](https://github.com/TBBle)<a name="note_4"></a>|@TBBle commented|
|-|-|

The use-case where we discovered this is in a game engine, where the Python automatic GC is disabled because we can't take the uncontrolled framerate hit when it runs at arbitrary points of time.

It's run manually at known safe-points, but we have a codebase policy that circular references should be broken before releasing them, and the GC is used only as a fail-safe and issue-reporter, as depending on load and usage patterns, those 'safe points' may well be minutes or more apart.

So yes, this is a circular reference with no external reference, not a 'leak' in the general Python sense, but "leak" is used as shorthand for such cases in this codebase.

Sorry for the terminology confusion.

As @JCoder3 mentioned, we have to fix it by breaking the cycle in an internal member of the Deferred.

This could be fixed in Twisted by having the `DelayedCall`'s `func`, `args`, and `kw` cleared [after calling](https://github.com/twisted/twisted/blob/twisted-22.1.0/src/twisted/internet/base.py#L990-L991), as is [already done on cancel](https://github.com/twisted/twisted/blob/twisted-22.1.0/src/twisted/internet/base.py#L145).
|[<img alt="exarkun's avatar" src="https://avatars.githubusercontent.com/u/254565?s=50" width="50" height="50">](https://github.com/exarkun)<a name="note_6"></a>|@exarkun commented|
|-|-|

Thanks for the clarification.  It should be straightforward to fix this specific cycle in Twisted and the suggestion to have `DelayedCall` clear those attributes seems reasonable.  Breaking the cycle by clearing the canceller is more difficult because a `Deferred` doesn't ever know that it is done (until it is garbage collected).  Even though `Deferred.callback` has been called, there may be asynchronous Deferreds in the chain or application code might call `addCallback` at a later time.

It should be possible to add a unit test that demonstrates a `DelayedCall` does not participate in a reference cycle with a `Deferred`'s canceller after the `DelayedCall` has fired.  This doesn't prove it is impossible to create a reference cycle with Deferred, DelayedCall, or deferLater (and in general Twisted doesn't promise to construct no persistent reference cycles), but at least it will prevent this particular cycle in the future.

|[<img alt="TBBle's avatar" src="https://avatars.githubusercontent.com/u/138397?s=50" width="50" height="50">](https://github.com/TBBle)<a name="note_7"></a>|@TBBle commented|
|-|-|

Indeed, the fix should be in `DelayedCall` because it is never reused after calling or cancelling, while `Deferred` offers no such guarantee.

Clearing the canceller, as mentioned by @JCoder3, is how we work around the problem in our code-base, because we know in this case that the `Deferred` is done-with at this point, but the `Deferred` itself could not know that.

Resolving patch:
diff --git a/src/twisted/internet/defer.py b/src/twisted/internet/defer.py
--- a/src/twisted/internet/defer.py
+++ b/src/twisted/internet/defer.py
@@ -797,6 +797,11 @@ def _startRunCallbacks(self, result: object) -> None:
                 self._debugInfo = DebugInfo()
             self._debugInfo.invoker = traceback.format_stack()[:-2]
         self.called = True
+
+        # Clear the canceller to avoid any circular references. This is safe to
+        # do as the canceller does not get called after the deferred has fired
+        self._canceller = None
+
         self.result = result
         self._runCallbacks()
 


Test patch:
diff --git a/src/twisted/test/test_defer.py b/src/twisted/test/test_defer.py
--- a/src/twisted/test/test_defer.py
+++ b/src/twisted/test/test_defer.py
@@ -12,7 +12,9 @@
 import re
 import traceback
 import types
+import unittest as pyunit
 import warnings
+import weakref
 from asyncio import (
     AbstractEventLoop,
     CancelledError,
@@ -58,6 +60,7 @@
 )
 from twisted.internet.task import Clock
 from twisted.python import log
+from twisted.python.compat import _PYPY
 from twisted.python.failure import Failure
 from twisted.trial import unittest
 
@@ -1678,6 +1681,109 @@ def test_fromCoroutineRequiresCoroutine(self) -> None:
         for thing in thingsThatAreNotCoroutines:
             self.assertRaises(defer.NotACoroutineError, Deferred.fromCoroutine, thing)
 
+    @pyunit.skipIf(_PYPY, "GC works differently on PyPy.")
+    def test_canceller_circular_reference_callback(self) -> None:
+        """
+        A circular reference between a `Deferred` and its canceller
+        is broken when the deferred is called.
+        """
+
+        # Create a canceller and weak reference to track if its been freed.
+        canceller = DummyCanceller()
+        weakCanceller = weakref.ref(canceller)
+
+        # Create a Deferred with the above canceller, adding a reference to the
+        # deferred to the canceller to create the circular reference.
+        deferred: Deferred[Any] = Deferred(canceller)
+        canceller.deferred = deferred
+        weakDeferred = weakref.ref(deferred)
+
+        deferred.callback(None)
+
+        # We manually remove the reference from the local
+        # function scope, to avoid interfering with the test.
+        del deferred
+        del canceller
+
+        # Once all local references have been dropped, the canceller should have
+        # been freed.
+        self.assertIsNone(weakCanceller())
+        self.assertIsNone(weakDeferred())
+
+    @pyunit.skipIf(_PYPY, "GC works differently on PyPy.")
+    def test_canceller_circular_reference_errback(self) -> None:
+        """
+        A circular reference between a `Deferred` and its canceller
+        is broken when the deferred fails.
+        """
+
+        # Create a canceller and weak reference to track if its been freed.
+        canceller = DummyCanceller()
+        weakCanceller = weakref.ref(canceller)
+
+        # Create a Deferred with the above canceller, adding a reference to the
+        # deferred to the canceller to create the circular reference.
+        deferred: Deferred[Any] = Deferred(canceller)
+        canceller.deferred = deferred
+        weakDeferred = weakref.ref(deferred)
+
+        failure = Failure(Exception("The test demands failures."))
+        deferred.errback(failure)
+
+        # We need to get the failure out of the deferred otherwise the test will
+        # fail due to unhandled error.
+        self.failureResultOf(deferred)
+
+        del deferred
+        del canceller
+
+        # Once all local references have been dropped, the canceller should have
+        # been freed.
+        self.assertIsNone(weakCanceller())
+        self.assertIsNone(weakDeferred())
+
+    @pyunit.skipIf(_PYPY, "GC works differently on PyPy.")
+    def test_canceller_circular_reference_non_final(self) -> None:
+        """
+        A circular reference between a `Deferred` and its canceller is broken
+        when the deferred is called, even if another deferred gets added to its
+        chain.
+        """
+
+        # Create a canceller and weak reference to track if its been freed.
+        canceller = DummyCanceller()
+        weakCanceller = weakref.ref(canceller)
+
+        # Create a Deferred with the above canceller, adding a reference to the
+        # deferred to the canceller to create the circular reference.
+        deferred: Deferred[Any] = Deferred(canceller)
+        canceller.deferred = deferred
+
+        deferred.addCallback(lambda _: Deferred())
+        deferred.callback(None)
+
+        del deferred
+        del canceller
+
+        # Once all local references have been dropped, the canceller should have
+        # been freed.
+        self.assertIsNone(weakCanceller())
+
+
+class DummyCanceller:
+    """
+    A callable that does nothing.
+    It is intended to be used just to get an object reference
+    to support GC related tests.
+    """
+
+    deferred: Optional[Deferred[Any]] = None
+
+    def __call__(self, deferred: Deferred[Any]) -> None:  # pragma: no cover
+        """
+        This is not expected to be called as part of the current test suite.
+        """
+
 
 def _setupRaceState(numDeferreds: int) -> tuple[list[int], list[Deferred[object]]]:
     """
@@ -3550,7 +3656,6 @@ def test_fromFutureDeferredCancelled(self) -> None:
 
 
 class CoroutineContextVarsTests(unittest.TestCase):
-
     if contextvars is None:
         skip = "contextvars is not available"  # type: ignore[unreachable]
 
@@ -3582,7 +3687,6 @@ def yieldingDeferred() -> Generator[Deferred[Any], Any, None]:
         # context is 1 when the function is defined
         @defer.inlineCallbacks
         def testFunction() -> Generator[Deferred[Any], Any, None]:
-
             # Expected to be 2
             self.assertEqual(var.get(), 2)
 
@@ -3747,7 +3851,6 @@ async def asyncFuncAwaitingDeferred() -> None:
 
         # context is 1 when the function is defined
         async def testFunction() -> bool:
-
             # Expected to be 2
             self.assertEqual(var.get(), 2)