Repository: numpy/numpy

Issue:
BUG?: 1.14 printing requires gc run to clean up all references to printed array
Dear NumPy devs, one of our NumPy integration doctests in Cython has been failing on travis for a couple of weeks. I checked the changelog of CPython and Cython from the time when it started, but neither showed anything remotely related. However, the failing travis nodes use NumPy 1.14, and since there were PyPy related changes in that release, I suspect that this could be the cause. I only see the failures in Py3.5+, and they started in the CPython-dev integration test runs for py3.6-dev and py3.7-dev on travis, however those are built (NumPy is pre-installed in them). Here's an example of a [failing run](https://travis-ci.org/cython/cython/builds/342548198).

The now failing [test](https://github.com/cython/cython/blob/2e1066dcfc1c4ccbfaa0babf3cbe9f58640431fd/tests/memoryview/numpy_memoryview.pyx#L415-L423) is simply this:
```
    cdef int[:, :] array = create_array((4, 5), mode="c", use_callback=True)
    print(np.asarray(array)[::2, ::2])
```
Meaning, there is a buffer provider (not NumPy), we take a Cython memory view from it, and then wrap that with `numpy.asarray()` to print it. The difference is when the buffer provider will be deallocated. Previously, it was cleaned up immediately at function exit (and we print a confirmation in a cleanup callback when that happens). With the new NumPy version, it is no longer cleaned up immediately, but only at the next GC run. I added a conditional GC call to the test to trigger that and keep the test from failing (we didn't need a GC run before), but I thought I'd at least bring this change to your attention, in case it's unintentional.




Discussion:
Sounds like a bug if it is numpy's fault, since there should be no reason to require a gc run, but I don't see why this should happen (not sure about all pypy changes, but those that I remember should be pretty unrelated).
There is a view, an array and the memoryview. The view should reference the array (or maybe the memoryview, but don't think it does here) and the array also the memoryview (stored in `array.base`).

If CPython has something to do with it, it would have to be something about changes in the buffer/memoryview interface. I expect the same goes for NumPy here.

Or.... there is something going on about the new printing functionality causing some more complex referencing. That seems strange, but I think it might be a real thing to look into. The print itself is required? I.e. if you only print a single item from the array or so it is fine?
If it is the printing, not sure it is a real bug, but could still be nice to remove at least if it is not hard.
Yeah, it is the printing. Printing temporarily increases the reference count by one.
So yeah, I guess its not really a bug, but still might be nice to get rid of if it is easy.
My guess would be it's the recursion detection, but it's not clear to my why that would hold onto a reference.
> Printing temporarily increases the reference count by one.

Can you give a repro? I can't seem to make this happen on master

sorry:

```
#!/usr/bin/env python

import numpy as np
import sys
import gc

gc.disable()

arr = np.arange(2000).reshape(5, 2, -1)

view = arr[::2, ::2]

print(sys.getrefcount(view))
print(view)
print(view)
print(sys.getrefcount(view))
gc.collect()
print(sys.getrefcount(view))
```
prints out 2 and then 4. Was using 1.14.0 on py 3.6.4, so there is some chance that it got fixed in master, didn't really excpect much changes since 1.14.0....
4 and then 2 of course ;).
Yeah, still in master.
It has something to do with the new use of a recursive closure in `_formatArray`.

Here's the code structure which triggers it, where `func` plays the role of `_formatArray`, and `closure` plays the role of `recurse` in `arrayprint.py` :
```python
import sys, gc

def func(a):
    def closure(i):
        if i == 0:
            return a
        return closure(i-1)
    return "val"

gc.disable()
a = np.arange(2)
print(sys.getrefcount(a))
r = func(a)
r = func(a)
print(sys.getrefcount(a))
gc.collect()
print(sys.getrefcount(a))
```

Interestingly, it only happens if the closure is recursive, and the closure does not even need to be called.
Yeah, I guess this is the expected behavior:

https://bugs.python.org/issue4921

So we just have to reorganize this code to avoid a recursive closure.
Can't you just delete the closure function when done with it to solve the cycle manually?
Oh, that's much simpler. I'll submit a PR with just that.

Resolving patch:
diff --git a/numpy/core/arrayprint.py b/numpy/core/arrayprint.py
--- a/numpy/core/arrayprint.py
+++ b/numpy/core/arrayprint.py
@@ -772,7 +772,8 @@ def recurser(index, hanging_indent, curr_width):
                     s += hanging_indent + summary_insert + line_sep
 
             for i in range(trailing_items, 1, -1):
-                nested = recurser(index + (-i,), next_hanging_indent, next_width)
+                nested = recurser(index + (-i,), next_hanging_indent,
+                                  next_width)
                 s += hanging_indent + nested + line_sep
 
             nested = recurser(index + (-1,), next_hanging_indent, next_width)
@@ -782,12 +783,16 @@ def recurser(index, hanging_indent, curr_width):
         s = '[' + s[len(hanging_indent):] + ']'
         return s
 
-    # invoke the recursive part with an initial index and prefix
-    return recurser(
-        index=(),
-        hanging_indent=next_line_prefix,
-        curr_width=line_width)
-
+    try:
+        # invoke the recursive part with an initial index and prefix
+        return recurser(index=(),
+                        hanging_indent=next_line_prefix,
+                        curr_width=line_width)
+    finally:
+        # recursive closures have a cyclic reference to themselves, which
+        # requires gc to collect (gh-10620). To avoid this problem, for
+        # performance and PyPy friendliness, we break the cycle:
+        recurser = None
 
 def _none_or_positive_arg(x, name):
     if x is None:
diff --git a/numpy/core/shape_base.py b/numpy/core/shape_base.py
--- a/numpy/core/shape_base.py
+++ b/numpy/core/shape_base.py
@@ -446,7 +446,13 @@ def block_recursion(arrays, depth=0):
             # type(arrays) is not list
             return atleast_nd(arrays, result_ndim)
 
-    return block_recursion(arrays)
+    try:
+        return block_recursion(arrays)
+    finally:
+        # recursive closures have a cyclic reference to themselves, which
+        # requires gc to collect (gh-10620). To avoid this problem, for
+        # performance and PyPy friendliness, we break the cycle:
+        block_recursion = None
 
 
 def block(arrays):
diff --git a/numpy/lib/npyio.py b/numpy/lib/npyio.py
--- a/numpy/lib/npyio.py
+++ b/numpy/lib/npyio.py
@@ -1109,6 +1109,11 @@ def tobytes_first(x, conv):
     finally:
         if fown:
             fh.close()
+        # recursive closures have a cyclic reference to themselves, which
+        # requires gc to collect (gh-10620). To avoid this problem, for
+        # performance and PyPy friendliness, we break the cycle:
+        flatten_dtype_internal = None
+        pack_items = None
 
     if X is None:
         X = np.array([], dtype)


Test patch:
diff --git a/numpy/core/tests/test_arrayprint.py b/numpy/core/tests/test_arrayprint.py
--- a/numpy/core/tests/test_arrayprint.py
+++ b/numpy/core/tests/test_arrayprint.py
@@ -1,7 +1,7 @@
 # -*- coding: utf-8 -*-
 from __future__ import division, absolute_import, print_function
 
-import sys
+import sys, gc
 
 import numpy as np
 from numpy.testing import (
@@ -355,6 +355,18 @@ def test_wide_element(self):
             "[ 'xxxxx']"
         )
 
+    def test_refcount(self):
+        # make sure we do not hold references to the array due to a recursive
+        # closure (gh-10620)
+        gc.disable()
+        a = np.arange(2)
+        r1 = sys.getrefcount(a)
+        np.array2string(a)
+        np.array2string(a)
+        r2 = sys.getrefcount(a)
+        gc.collect()
+        gc.enable()
+        assert_(r1 == r2)
 
 class TestPrintOptions(object):
     """Test getting and setting global print options."""