Repository: python/cpython
Pull request: gh-118272: Clear generator frame's locals when the generator is closed
Fixes #118272.

Co-authored-by: Thomas Grainger <tagrain@gmail.com>

<!-- gh-issue-number: gh-118272 -->
* Issue: gh-118272
<!-- /gh-issue-number -->


Problem statement:
Generator .close does not release resources
# Bug report

### Bug description:

Sending a generator `.close()` in Python 3.12 does not release resources used by local variables as it used to do in earlier versions of Python. Wrapping the generator content in a `try: ... except GeneratorExit: pass` releases the resources as expected.

The following shows the difference in space usage between wrapping the code with `try`-`except` and not.

```python
def generator_using_a_lot_of_space():
    a_big_set = set(range(10_000_000))
    yield 42

g = generator_using_a_lot_of_space()
input('A <press enter>')  # 13.4 MB
print(next(g))
input('B <press enter>')  # 576.6 MB
g.close()
input('C <press enter>')  # 576.6 MB  <== space usage used to drop to 13-14 MB

def generator_using_a_lot_of_space():
    a_big_set = set(range(10_000_000))
    try: 
        yield 42
    except GeneratorExit:
        pass
    
g = generator_using_a_lot_of_space()
input('D <press enter>')  # 14.6 MB
print(next(g))
input('E <press enter>')  # 575.6 MB
g.close()
input('F <press enter>')  # 14.6 MB  <== drop in space usage as expected
```

The above space measurements were done with 
Python 3.12.3 (tags/v3.12.3:f6650f9, Apr  9 2024, 14:05:25) [MSC v.1938 64 bit (AMD64)] on win32
and looking at the space usage reported by the Windows 11 Task Manager.

### CPython versions tested on:

3.12

### Operating systems tested on:

Windows

<!-- gh-linked-prs -->
### Linked PRs
* gh-118277
* gh-118451
* gh-118478
<!-- /gh-linked-prs -->

Hints from issue discussion:
bisected to eb4774d2b7f5d50cd3cb8dc5abce9f94ce54197e using:
```python
import gc
import weakref

class Foo:
    pass

f = Foo()
f_wr = weakref.ref(f)

def genfn():
    a = f
    yield


g = genfn()
next(g)
del f
g.close()
gc.collect()
print(f_wr)
```

outputs:
```
<weakref at 0x7fd58a1917b0; to 'Foo' at 0x7fd58ab12cf0>
```
should output:
```
<weakref at 0x7ad6faf19d50; dead>
```

passes on 3.11, fails on 3.12 and 3.13a6

Commit message: gh-118272: Clear generator frame's locals when the generator is closed

Patch to review:
diff --git a/Include/internal/pycore_frame.h b/Include/internal/pycore_frame.h
index f913928f38bd05..37ae5ae850389b 100644
--- a/Include/internal/pycore_frame.h
+++ b/Include/internal/pycore_frame.h
@@ -227,6 +227,9 @@ _PyFrame_GetFrameObject(_PyInterpreterFrame *frame)
     return _PyFrame_MakeAndSetFrameObject(frame);
 }
 
+void
+_PyFrame_ClearLocals(_PyInterpreterFrame *frame);
+
 /* Clears all references in the frame.
  * If take is non-zero, then the _PyInterpreterFrame frame
  * may be transferred to the frame object it references
diff --git a/Lib/test/test_generators.py b/Lib/test/test_generators.py
index d48f0d47ba1962..429988524386e1 100644
--- a/Lib/test/test_generators.py
+++ b/Lib/test/test_generators.py
@@ -532,6 +532,26 @@ def f():
         with self.assertRaises(RuntimeError):
             gen.close()
 
+    def test_close_releases_frame_locals(self):
+        # See gh-118272
+
+        class Foo:
+            pass
+
+        f = Foo()
+        f_wr = weakref.ref(f)
+
+        def genfn():
+            a = f
+            yield
+
+        g = genfn()
+        next(g)
+        del f
+        g.close()
+        gc.collect()
+        self.assertIsNone(f_wr())
+
 
 class GeneratorThrowTest(unittest.TestCase):
 
diff --git a/Misc/NEWS.d/next/Core and Builtins/2024-04-25-12-55-47.gh-issue-118272.5ptjk_.rst b/Misc/NEWS.d/next/Core and Builtins/2024-04-25-12-55-47.gh-issue-118272.5ptjk_.rst
new file mode 100644
index 00000000000000..32043440fd0365
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2024-04-25-12-55-47.gh-issue-118272.5ptjk_.rst	
@@ -0,0 +1,2 @@
+Fix bug where ``generator.close`` does not free the generator frame's
+locals.
diff --git a/Objects/genobject.c b/Objects/genobject.c
index 5d7da49cfca166..04736a04562e14 100644
--- a/Objects/genobject.c
+++ b/Objects/genobject.c
@@ -380,6 +380,7 @@ gen_close(PyGenObject *gen, PyObject *args)
             // RESUME after YIELD_VALUE and exception depth is 1
             assert((oparg & RESUME_OPARG_LOCATION_MASK) != RESUME_AT_FUNC_START);
             gen->gi_frame_state = FRAME_COMPLETED;
+            _PyFrame_ClearLocals((_PyInterpreterFrame *)gen->gi_iframe);
             Py_RETURN_NONE;
         }
     }
diff --git a/Python/frame.c b/Python/frame.c
index db9d13359a23ca..ec390e7426ad47 100644
--- a/Python/frame.c
+++ b/Python/frame.c
@@ -94,6 +94,17 @@ take_ownership(PyFrameObject *f, _PyInterpreterFrame *frame)
     }
 }
 
+void
+_PyFrame_ClearLocals(_PyInterpreterFrame *frame)
+{
+    assert(frame->stacktop >= 0);
+    for (int i = 0; i < frame->stacktop; i++) {
+        Py_XDECREF(frame->localsplus[i]);
+    }
+    frame->stacktop = 0;
+    Py_CLEAR(frame->f_locals);
+}
+
 void
 _PyFrame_ClearExceptCode(_PyInterpreterFrame *frame)
 {
@@ -114,11 +125,7 @@ _PyFrame_ClearExceptCode(_PyInterpreterFrame *frame)
         }
         Py_DECREF(f);
     }
-    assert(frame->stacktop >= 0);
-    for (int i = 0; i < frame->stacktop; i++) {
-        Py_XDECREF(frame->localsplus[i]);
-    }
-    Py_XDECREF(frame->f_locals);
+    _PyFrame_ClearLocals(frame);
     Py_DECREF(frame->f_funcobj);
 }
 


Code review comments:
diff_hunk: @@ -532,6 +532,26 @@ def f():
         with self.assertRaises(RuntimeError):
             gen.close()
 
+    def test_close_releases_frame_locals(self):
+        # See gh-118272
+
+        class Foo:
+            pass
+
+        f = Foo()
+        f_wr = weakref.ref(f)
+
+        def genfn():
+            a = f
+            yield
+
+        g = genfn()
+        next(g)
+        del f
+        g.close()
+        gc.collect()
path: Lib/test/test_generators.py
text: @user1:
```suggestion
        support.gc_collect()
```