Repository: numpy/numpy

Issue:
BUG?: 1.14 printing requires gc run to clean up all references to printed array
Dear NumPy devs, one of our NumPy integration doctests in Cython has been failing on travis for a couple of weeks. I checked the changelog of CPython and Cython from the time when it started, but neither showed anything remotely related. However, the failing travis nodes use NumPy 1.14, and since there were PyPy related changes in that release, I suspect that this could be the cause. I only see the failures in Py3.5+, and they started in the CPython-dev integration test runs for py3.6-dev and py3.7-dev on travis, however those are built (NumPy is pre-installed in them). Here's an example of a [failing run](https://travis-ci.org/cython/cython/builds/342548198).

The now failing [test](https://github.com/cython/cython/blob/2e1066dcfc1c4ccbfaa0babf3cbe9f58640431fd/tests/memoryview/numpy_memoryview.pyx#L415-L423) is simply this:
```
    cdef int[:, :] array = create_array((4, 5), mode="c", use_callback=True)
    print(np.asarray(array)[::2, ::2])
```
Meaning, there is a buffer provider (not NumPy), we take a Cython memory view from it, and then wrap that with `numpy.asarray()` to print it. The difference is when the buffer provider will be deallocated. Previously, it was cleaned up immediately at function exit (and we print a confirmation in a cleanup callback when that happens). With the new NumPy version, it is no longer cleaned up immediately, but only at the next GC run. I added a conditional GC call to the test to trigger that and keep the test from failing (we didn't need a GC run before), but I thought I'd at least bring this change to your attention, in case it's unintentional.




Discussion:
Sounds like a bug if it is numpy's fault, since there should be no reason to require a gc run, but I don't see why this should happen (not sure about all pypy changes, but those that I remember should be pretty unrelated).
There is a view, an array and the memoryview. The view should reference the array (or maybe the memoryview, but don't think it does here) and the array also the memoryview (stored in `array.base`).

If CPython has something to do with it, it would have to be something about changes in the buffer/memoryview interface. I expect the same goes for NumPy here.

Or.... there is something going on about the new printing functionality causing some more complex referencing. That seems strange, but I think it might be a real thing to look into. The print itself is required? I.e. if you only print a single item from the array or so it is fine?
If it is the printing, not sure it is a real bug, but could still be nice to remove at least if it is not hard.
Yeah, it is the printing. Printing temporarily increases the reference count by one.
So yeah, I guess its not really a bug, but still might be nice to get rid of if it is easy.
My guess would be it's the recursion detection, but it's not clear to my why that would hold onto a reference.
> Printing temporarily increases the reference count by one.

Can you give a repro? I can't seem to make this happen on master

sorry:

```
#!/usr/bin/env python

import numpy as np
import sys
import gc

gc.disable()

arr = np.arange(2000).reshape(5, 2, -1)

view = arr[::2, ::2]

print(sys.getrefcount(view))
print(view)
print(view)
print(sys.getrefcount(view))
gc.collect()
print(sys.getrefcount(view))
```
prints out 2 and then 4. Was using 1.14.0 on py 3.6.4, so there is some chance that it got fixed in master, didn't really excpect much changes since 1.14.0....
4 and then 2 of course ;).
Yeah, still in master.
It has something to do with the new use of a recursive closure in `_formatArray`.

Here's the code structure which triggers it, where `func` plays the role of `_formatArray`, and `closure` plays the role of `recurse` in `arrayprint.py` :
```python
import sys, gc

def func(a):
    def closure(i):
        if i == 0:
            return a
        return closure(i-1)
    return "val"

gc.disable()
a = np.arange(2)
print(sys.getrefcount(a))
r = func(a)
r = func(a)
print(sys.getrefcount(a))
gc.collect()
print(sys.getrefcount(a))
```

Interestingly, it only happens if the closure is recursive, and the closure does not even need to be called.
Yeah, I guess this is the expected behavior:

https://bugs.python.org/issue4921

So we just have to reorganize this code to avoid a recursive closure.
Can't you just delete the closure function when done with it to solve the cycle manually?
Oh, that's much simpler. I'll submit a PR with just that.

Resolving patch:
diff --git a/numpy/core/arrayprint.py b/numpy/core/arrayprint.py
--- a/numpy/core/arrayprint.py
+++ b/numpy/core/arrayprint.py
@@ -642,118 +642,121 @@ def _extendLine(s, line, word, line_width, next_line_prefix, legacy):
     line += word
     return s, line
 
+def _recursive_fmt(param, index, indent, curr_width):
+    """
+    Helper function for _formatArray, to recursively print array elements.
 
-def _formatArray(a, format_function, line_width, next_line_prefix,
-                 separator, edge_items, summary_insert, legacy):
-    """formatArray is designed for two modes of operation:
+    "param" are the values that are invariant under recursion, including
+    the array to be printed itself. index, indent and curr_width
+    are updated during recursion.
+    """
+    # unpack parameters
+    a, format_function, separator, edge_items, summary_insert, legacy = param
 
-    1. Full output
+    axis = len(index)
+    axes_left = a.ndim - axis
 
-    2. Summarized output
+    if axes_left == 0:
+        return format_function(a[index])
 
-    """
-    def recurser(index, hanging_indent, curr_width):
-        """
-        By using this local function, we don't need to recurse with all the
-        arguments. Since this function is not created recursively, the cost is
-        not significant
-        """
-        axis = len(index)
-        axes_left = a.ndim - axis
+    # when recursing, add a space to align with the [ added, and reduce the
+    # length of the line by 1
+    next_indent = indent + ' '
+    if legacy == '1.13':
+        next_width = curr_width
+    else:
+        next_width = curr_width - len(']')
 
-        if axes_left == 0:
-            return format_function(a[index])
+    a_len = a.shape[axis]
+    show_summary = summary_insert and 2*edge_items < a_len
+    if show_summary:
+        leading_items = edge_items
+        trailing_items = edge_items
+    else:
+        leading_items = 0
+        trailing_items = a_len
 
-        # when recursing, add a space to align with the [ added, and reduce the
-        # length of the line by 1
-        next_hanging_indent = hanging_indent + ' '
-        if legacy == '1.13':
-            next_width = curr_width
-        else:
-            next_width = curr_width - len(']')
+    # stringify the array with the hanging indent on the first line too
+    s = ''
 
-        a_len = a.shape[axis]
-        show_summary = summary_insert and 2*edge_items < a_len
-        if show_summary:
-            leading_items = edge_items
-            trailing_items = edge_items
+    # last axis (rows) - wrap elements if they would not fit on one line
+    if axes_left == 1:
+        # the length up until the beginning of the separator / bracket
+        if legacy == '1.13':
+            elem_width = curr_width - len(separator.rstrip())
         else:
-            leading_items = 0
-            trailing_items = a_len
+            elem_width = curr_width - max(len(separator.rstrip()), len(']'))
 
-        # stringify the array with the hanging indent on the first line too
-        s = ''
+        line = indent
+        for i in range(leading_items):
+            word = _recursive_fmt(param, index + (i,), next_indent, next_width)
+            s, line = _extendLine(s, line, word, elem_width, indent, legacy)
+            line += separator
 
-        # last axis (rows) - wrap elements if they would not fit on one line
-        if axes_left == 1:
-            # the length up until the beginning of the separator / bracket
+        if show_summary:
+            s, line = _extendLine(
+                s, line, summary_insert, elem_width, indent, legacy)
             if legacy == '1.13':
-                elem_width = curr_width - len(separator.rstrip())
+                line += ", "
             else:
-                elem_width = curr_width - max(len(separator.rstrip()), len(']'))
-
-            line = hanging_indent
-            for i in range(leading_items):
-                word = recurser(index + (i,), next_hanging_indent, next_width)
-                s, line = _extendLine(
-                    s, line, word, elem_width, hanging_indent, legacy)
                 line += separator
 
-            if show_summary:
-                s, line = _extendLine(
-                    s, line, summary_insert, elem_width, hanging_indent, legacy)
-                if legacy == '1.13':
-                    line += ", "
-                else:
-                    line += separator
-
-            for i in range(trailing_items, 1, -1):
-                word = recurser(index + (-i,), next_hanging_indent, next_width)
-                s, line = _extendLine(
-                    s, line, word, elem_width, hanging_indent, legacy)
-                line += separator
+        for i in range(trailing_items, 1, -1):
+            word = _recursive_fmt(param, index + (-i,), next_indent, next_width)
+            s, line = _extendLine(s, line, word, elem_width, indent, legacy)
+            line += separator
+
+        if legacy == '1.13':
+            # width of the separator is not considered on 1.13
+            elem_width = curr_width
+        word =_recursive_fmt(param, index + (-1,), next_indent, next_width)
+        s, line = _extendLine(
+            s, line, word, elem_width, indent, legacy)
+
+        s += line
+
+    # other axes - insert newlines between rows
+    else:
+        s = ''
+        line_sep = separator.rstrip() + '\n'*(axes_left - 1)
+
+        for i in range(leading_items):
+            nested =_recursive_fmt(param, index + (i,), next_indent, next_width)
+            s += indent + nested + line_sep
 
+        if show_summary:
             if legacy == '1.13':
-                # width of the seperator is not considered on 1.13
-                elem_width = curr_width
-            word = recurser(index + (-1,), next_hanging_indent, next_width)
-            s, line = _extendLine(
-                s, line, word, elem_width, hanging_indent, legacy)
+                # trailing space, fixed nbr of newlines, and fixed separator
+                s += indent + summary_insert + ", \n"
+            else:
+                s += indent + summary_insert + line_sep
 
-            s += line
+        for i in range(trailing_items, 1, -1):
+            nested = _recursive_fmt(param, index + (-i,), next_indent,
+                                    next_width)
+            s += indent + nested + line_sep
 
-        # other axes - insert newlines between rows
-        else:
-            s = ''
-            line_sep = separator.rstrip() + '\n'*(axes_left - 1)
+        nested =_recursive_fmt(param, index + (-1,), next_indent, next_width)
+        s += indent + nested
 
-            for i in range(leading_items):
-                nested = recurser(index + (i,), next_hanging_indent, next_width)
-                s += hanging_indent + nested + line_sep
+    # remove the hanging indent, and wrap in []
+    s = '[' + s[len(indent):] + ']'
+    return s
 
-            if show_summary:
-                if legacy == '1.13':
-                    # trailing space, fixed nbr of newlines, and fixed separator
-                    s += hanging_indent + summary_insert + ", \n"
-                else:
-                    s += hanging_indent + summary_insert + line_sep
+def _formatArray(a, format_function, line_width, next_line_prefix,
+                 separator, edge_items, summary_insert, legacy):
+    """_formatArray is designed for two modes of operation:
 
-            for i in range(trailing_items, 1, -1):
-                nested = recurser(index + (-i,), next_hanging_indent, next_width)
-                s += hanging_indent + nested + line_sep
+    1. Full output
 
-            nested = recurser(index + (-1,), next_hanging_indent, next_width)
-            s += hanging_indent + nested
+    2. Summarized output
 
-        # remove the hanging indent, and wrap in []
-        s = '[' + s[len(hanging_indent):] + ']'
-        return s
+    """
 
     # invoke the recursive part with an initial index and prefix
-    return recurser(
-        index=(),
-        hanging_indent=next_line_prefix,
-        curr_width=line_width)
+    param = a, format_function, separator, edge_items, summary_insert, legacy
+    return _recursive_fmt(param, index=(), indent=next_line_prefix,
+                          curr_width=line_width)
 
 
 def _none_or_positive_arg(x, name):


Test patch:
diff --git a/numpy/core/tests/test_arrayprint.py b/numpy/core/tests/test_arrayprint.py
--- a/numpy/core/tests/test_arrayprint.py
+++ b/numpy/core/tests/test_arrayprint.py
@@ -1,7 +1,7 @@
 # -*- coding: utf-8 -*-
 from __future__ import division, absolute_import, print_function
 
-import sys
+import sys, gc
 
 import numpy as np
 from numpy.testing import (
@@ -355,6 +355,18 @@ def test_wide_element(self):
             "[ 'xxxxx']"
         )
 
+    def test_refcount(self):
+        # make sure we do not hold references to the array due to a recursive
+        # closure (gh-10620)
+        gc.disable()
+        a = np.arange(2)
+        r1 = sys.getrefcount(a)
+        np.array2string(a)
+        np.array2string(a)
+        r2 = sys.getrefcount(a)
+        gc.collect()
+        gc.enable()
+        assert_(r1 == r2)
 
 class TestPrintOptions(object):
     """Test getting and setting global print options."""