Repository: numpy/numpy

Issue:
BUG?: 1.14 printing requires gc run to clean up all references to printed array
Dear NumPy devs, one of our NumPy integration doctests in Cython has been failing on travis for a couple of weeks. I checked the changelog of CPython and Cython from the time when it started, but neither showed anything remotely related. However, the failing travis nodes use NumPy 1.14, and since there were PyPy related changes in that release, I suspect that this could be the cause. I only see the failures in Py3.5+, and they started in the CPython-dev integration test runs for py3.6-dev and py3.7-dev on travis, however those are built (NumPy is pre-installed in them). Here's an example of a [failing run](https://travis-ci.org/cython/cython/builds/342548198).

The now failing [test](https://github.com/cython/cython/blob/2e1066dcfc1c4ccbfaa0babf3cbe9f58640431fd/tests/memoryview/numpy_memoryview.pyx#L415-L423) is simply this:
```
    cdef int[:, :] array = create_array((4, 5), mode="c", use_callback=True)
    print(np.asarray(array)[::2, ::2])
```
Meaning, there is a buffer provider (not NumPy), we take a Cython memory view from it, and then wrap that with `numpy.asarray()` to print it. The difference is when the buffer provider will be deallocated. Previously, it was cleaned up immediately at function exit (and we print a confirmation in a cleanup callback when that happens). With the new NumPy version, it is no longer cleaned up immediately, but only at the next GC run. I added a conditional GC call to the test to trigger that and keep the test from failing (we didn't need a GC run before), but I thought I'd at least bring this change to your attention, in case it's unintentional.




Discussion:
Sounds like a bug if it is numpy's fault, since there should be no reason to require a gc run, but I don't see why this should happen (not sure about all pypy changes, but those that I remember should be pretty unrelated).
There is a view, an array and the memoryview. The view should reference the array (or maybe the memoryview, but don't think it does here) and the array also the memoryview (stored in `array.base`).

If CPython has something to do with it, it would have to be something about changes in the buffer/memoryview interface. I expect the same goes for NumPy here.

Or.... there is something going on about the new printing functionality causing some more complex referencing. That seems strange, but I think it might be a real thing to look into. The print itself is required? I.e. if you only print a single item from the array or so it is fine?
If it is the printing, not sure it is a real bug, but could still be nice to remove at least if it is not hard.
Yeah, it is the printing. Printing temporarily increases the reference count by one.
So yeah, I guess its not really a bug, but still might be nice to get rid of if it is easy.
My guess would be it's the recursion detection, but it's not clear to my why that would hold onto a reference.
> Printing temporarily increases the reference count by one.

Can you give a repro? I can't seem to make this happen on master

sorry:

```
#!/usr/bin/env python

import numpy as np
import sys
import gc

gc.disable()

arr = np.arange(2000).reshape(5, 2, -1)

view = arr[::2, ::2]

print(sys.getrefcount(view))
print(view)
print(view)
print(sys.getrefcount(view))
gc.collect()
print(sys.getrefcount(view))
```
prints out 2 and then 4. Was using 1.14.0 on py 3.6.4, so there is some chance that it got fixed in master, didn't really excpect much changes since 1.14.0....
4 and then 2 of course ;).
Yeah, still in master.
It has something to do with the new use of a recursive closure in `_formatArray`.

Here's the code structure which triggers it, where `func` plays the role of `_formatArray`, and `closure` plays the role of `recurse` in `arrayprint.py` :
```python
import sys, gc

def func(a):
    def closure(i):
        if i == 0:
            return a
        return closure(i-1)
    return "val"

gc.disable()
a = np.arange(2)
print(sys.getrefcount(a))
r = func(a)
r = func(a)
print(sys.getrefcount(a))
gc.collect()
print(sys.getrefcount(a))
```

Interestingly, it only happens if the closure is recursive, and the closure does not even need to be called.
Yeah, I guess this is the expected behavior:

https://bugs.python.org/issue4921

So we just have to reorganize this code to avoid a recursive closure.
Can't you just delete the closure function when done with it to solve the cycle manually?
Oh, that's much simpler. I'll submit a PR with just that.
After a quick grep I found other recursive closures in numpy: [in np.block](https://github.com/numpy/numpy/blob/4105af42c38539e8bf854c49134fe221d86ec79d/numpy/core/shape_base.py#L442),and  [a couple in np.loadtxt](https://github.com/numpy/numpy/blob/4105af42c38539e8bf854c49134fe221d86ec79d/numpy/lib/npyio.py#L965). Those examples don't seem too harmful since they only use a few python scalars as closure variables.
Hah, also, python2 does not allow me to delete the closure function:
```python
[1]: def func(a):
...:     def closure(i):
...:         if i == 0:
...:             return a
...:         return closure(i-1)
...:     del closure
...:     return "val"
SyntaxError: can not delete variable 'closure' referenced in nested scope
```
Python3 allows it though.
Does setting `closure = None` work, if `del closure` doesn't? That should be enough to break the reference cycle
Or `a = None`. Seems more explicit to reset the reference to the array than to try to delete some inner function.
leaving the inner function still leaves a reference cycle though, which we still want to clean up if possible. Setting `a = None` is addressing the symptom, not the cause.
Both `a = None` and `closure = None`  seem to work. I agree only `closure = None` seems to really remove the cycle, though.

I don't very much like adding these extra `= None` lines though, they seem a little too magical. Maybe we should go with @eric-wieser's idea of a callable class which has the recursive part as a member function, or just leave it as a separate helper function like it is in #10621 right now, and avoid the recursive closures altogether.

Actually, recursive closures seem like something PyPy could be made to account for more nicely. PyPy probably already implements its own version of closures and the co_cellvars table, so it seems there is opportunity to detect and remove the cycle when the closure goes out of scope.... maybe it gets complicated though.

Oh, I guess the problem with my last paragraph is the case where you return the (recursive) closure function. I.e it survives past the scope of the conaining function.
> they seem a little too magical

I think adding `# break the reference cycle - bpo-XXXXX` explains most of the magic, and I think that might be preferable to passing parameters around in an opaque tuple as you do in #10621.

I went for a recursive closure in the first place because it was less verbose than defining a helper class, and seems like a reasonably useful pattern in general for a function with a recursive implementation that needs to do some initial work.

Even if we fix `_formatArray` with your proposed fix, we should break the reference cycles manually in the other places we use this pattern.


True, perhaps the `= None` trick is the more elegant one in the end. 

I'm still searching for alternate solutions. This is an interesting thread I'm still reading through, which shows a couple other cute solutions: https://mail.python.org/pipermail/python-dev/2009-December/094439.html

(also shows we may want to put the `=None` in a finally block)
So after reading that thread, I think the best is to set the closure to `None` in a `finally` block. I've updated #10621 accordingly, fixing the 4 recursive closures I found.

Resolving patch:
diff --git a/numpy/core/arrayprint.py b/numpy/core/arrayprint.py
--- a/numpy/core/arrayprint.py
+++ b/numpy/core/arrayprint.py
@@ -642,122 +642,124 @@ def _extendLine(s, line, word, line_width, next_line_prefix, legacy):
     line += word
     return s, line
 
-def _recursive_fmt(param, index, indent, curr_width):
-    """
-    Helper function for _formatArray, to recursively print array elements.
-
-    "param" are the values that are invariant under recursion, including
-    the array to be printed itself. index, indent and curr_width
-    are updated during recursion.
-    """
-    # unpack parameters
-    a, format_function, separator, edge_items, summary_insert, legacy = param
 
-    axis = len(index)
-    axes_left = a.ndim - axis
+def _formatArray(a, format_function, line_width, next_line_prefix,
+                 separator, edge_items, summary_insert, legacy):
+    """formatArray is designed for two modes of operation:
 
-    if axes_left == 0:
-        return format_function(a[index])
+    1. Full output
 
-    # when recursing, add a space to align with the [ added, and reduce the
-    # length of the line by 1
-    next_indent = indent + ' '
-    if legacy == '1.13':
-        next_width = curr_width
-    else:
-        next_width = curr_width - len(']')
+    2. Summarized output
 
-    a_len = a.shape[axis]
-    show_summary = summary_insert and 2*edge_items < a_len
-    if show_summary:
-        leading_items = edge_items
-        trailing_items = edge_items
-    else:
-        leading_items = 0
-        trailing_items = a_len
+    """
+    def recurser(index, hanging_indent, curr_width):
+        """
+        By using this local function, we don't need to recurse with all the
+        arguments. Since this function is not created recursively, the cost is
+        not significant
+        """
+        axis = len(index)
+        axes_left = a.ndim - axis
 
-    # stringify the array with the hanging indent on the first line too
-    s = ''
+        if axes_left == 0:
+            return format_function(a[index])
 
-    # last axis (rows) - wrap elements if they would not fit on one line
-    if axes_left == 1:
-        # the length up until the beginning of the separator / bracket
+        # when recursing, add a space to align with the [ added, and reduce the
+        # length of the line by 1
+        next_hanging_indent = hanging_indent + ' '
         if legacy == '1.13':
-            elem_width = curr_width - len(separator.rstrip())
+            next_width = curr_width
         else:
-            elem_width = curr_width - max(len(separator.rstrip()), len(']'))
-
-        line = indent
-        for i in range(leading_items):
-            word = _recursive_fmt(param, index + (i,), next_indent, next_width)
-            s, line = _extendLine(s, line, word, elem_width, indent, legacy)
-            line += separator
+            next_width = curr_width - len(']')
 
+        a_len = a.shape[axis]
+        show_summary = summary_insert and 2*edge_items < a_len
         if show_summary:
-            s, line = _extendLine(
-                s, line, summary_insert, elem_width, indent, legacy)
-            if legacy == '1.13':
-                line += ", "
-            else:
-                line += separator
-
-        for i in range(trailing_items, 1, -1):
-            word = _recursive_fmt(param, index + (-i,), next_indent, next_width)
-            s, line = _extendLine(s, line, word, elem_width, indent, legacy)
-            line += separator
-
-        if legacy == '1.13':
-            # width of the separator is not considered on 1.13
-            elem_width = curr_width
-        word =_recursive_fmt(param, index + (-1,), next_indent, next_width)
-        s, line = _extendLine(
-            s, line, word, elem_width, indent, legacy)
-
-        s += line
+            leading_items = edge_items
+            trailing_items = edge_items
+        else:
+            leading_items = 0
+            trailing_items = a_len
 
-    # other axes - insert newlines between rows
-    else:
+        # stringify the array with the hanging indent on the first line too
         s = ''
-        line_sep = separator.rstrip() + '\n'*(axes_left - 1)
-
-        for i in range(leading_items):
-            nested =_recursive_fmt(param, index + (i,), next_indent, next_width)
-            s += indent + nested + line_sep
 
-        if show_summary:
+        # last axis (rows) - wrap elements if they would not fit on one line
+        if axes_left == 1:
+            # the length up until the beginning of the separator / bracket
             if legacy == '1.13':
-                # trailing space, fixed nbr of newlines, and fixed separator
-                s += indent + summary_insert + ", \n"
+                elem_width = curr_width - len(separator.rstrip())
             else:
-                s += indent + summary_insert + line_sep
+                elem_width = curr_width - max(len(separator.rstrip()), len(']'))
 
-        for i in range(trailing_items, 1, -1):
-            nested = _recursive_fmt(param, index + (-i,), next_indent,
-                                    next_width)
-            s += indent + nested + line_sep
+            line = hanging_indent
+            for i in range(leading_items):
+                word = recurser(index + (i,), next_hanging_indent, next_width)
+                s, line = _extendLine(
+                    s, line, word, elem_width, hanging_indent, legacy)
+                line += separator
 
-        nested =_recursive_fmt(param, index + (-1,), next_indent, next_width)
-        s += indent + nested
+            if show_summary:
+                s, line = _extendLine(
+                    s, line, summary_insert, elem_width, hanging_indent, legacy)
+                if legacy == '1.13':
+                    line += ", "
+                else:
+                    line += separator
+
+            for i in range(trailing_items, 1, -1):
+                word = recurser(index + (-i,), next_hanging_indent, next_width)
+                s, line = _extendLine(
+                    s, line, word, elem_width, hanging_indent, legacy)
+                line += separator
 
-    # remove the hanging indent, and wrap in []
-    s = '[' + s[len(indent):] + ']'
-    return s
+            if legacy == '1.13':
+                # width of the seperator is not considered on 1.13
+                elem_width = curr_width
+            word = recurser(index + (-1,), next_hanging_indent, next_width)
+            s, line = _extendLine(
+                s, line, word, elem_width, hanging_indent, legacy)
 
-def _formatArray(a, format_function, line_width, next_line_prefix,
-                 separator, edge_items, summary_insert, legacy):
-    """_formatArray is designed for two modes of operation:
+            s += line
 
-    1. Full output
+        # other axes - insert newlines between rows
+        else:
+            s = ''
+            line_sep = separator.rstrip() + '\n'*(axes_left - 1)
 
-    2. Summarized output
+            for i in range(leading_items):
+                nested = recurser(index + (i,), next_hanging_indent, next_width)
+                s += hanging_indent + nested + line_sep
 
-    """
+            if show_summary:
+                if legacy == '1.13':
+                    # trailing space, fixed nbr of newlines, and fixed separator
+                    s += hanging_indent + summary_insert + ", \n"
+                else:
+                    s += hanging_indent + summary_insert + line_sep
+
+            for i in range(trailing_items, 1, -1):
+                nested = recurser(index + (-i,), next_hanging_indent,
+                                  next_width)
+                s += hanging_indent + nested + line_sep
 
-    # invoke the recursive part with an initial index and prefix
-    param = a, format_function, separator, edge_items, summary_insert, legacy
-    return _recursive_fmt(param, index=(), indent=next_line_prefix,
-                          curr_width=line_width)
+            nested = recurser(index + (-1,), next_hanging_indent, next_width)
+            s += hanging_indent + nested
 
+        # remove the hanging indent, and wrap in []
+        s = '[' + s[len(hanging_indent):] + ']'
+        return s
+
+    try:
+        # invoke the recursive part with an initial index and prefix
+        return recurser(index=(),
+                        hanging_indent=next_line_prefix,
+                        curr_width=line_width)
+    finally:
+        # recursive closures have a cyclic reference to themselves, which
+        # requires gc to collect (gh-10620). To avoid this problem, for
+        # performance and PyPy friendliness, we break the cycle:
+        recurser = None
 
 def _none_or_positive_arg(x, name):
     if x is None:
diff --git a/numpy/core/shape_base.py b/numpy/core/shape_base.py
--- a/numpy/core/shape_base.py
+++ b/numpy/core/shape_base.py
@@ -446,7 +446,13 @@ def block_recursion(arrays, depth=0):
             # type(arrays) is not list
             return atleast_nd(arrays, result_ndim)
 
-    return block_recursion(arrays)
+    try:
+        return block_recursion(arrays)
+    finally:
+        # recursive closures have a cyclic reference to themselves, which
+        # requires gc to collect (gh-10620). To avoid this problem, for
+        # performance and PyPy friendliness, we break the cycle:
+        block_recursion = None
 
 
 def block(arrays):
diff --git a/numpy/lib/npyio.py b/numpy/lib/npyio.py
--- a/numpy/lib/npyio.py
+++ b/numpy/lib/npyio.py
@@ -1101,6 +1101,11 @@ def tobytes_first(x, conv):
     finally:
         if fown:
             fh.close()
+        # recursive closures have a cyclic reference to themselves, which
+        # requires gc to collect (gh-10620). To avoid this problem, for
+        # performance and PyPy friendliness, we break the cycle:
+        flatten_dtype_internal = None
+        pack_items = None
 
     if X is None:
         X = np.array([], dtype)


Test patch: