Repository: numpy/numpy

Issue:
BUG: close() method of NpzFile returned by np.load triggers ResourceWarning
_EDIT: I had missed a key point in the docstring of np.load, so my original description of the issue was wrong._

When `np.load` is given an npz file, it returns an instance of `NpzFile`.  To close the file held by the `NpzFile` instance, the `close` method must be called.  However, this results in a `ResourceWarning`.

For example:

```
$ python3 -Wdefault
Python 3.3.2 (default, Jun 14 2013, 12:12:22) 
[GCC 4.6.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> xy = np.load('xy.npz')
>>> xy.close()
/home/warren/local_py332/lib/python3.3/site-packages/numpy/lib/npyio.py:219: ResourceWarning: unclosed file <_io.BufferedReader name='xy.npz'>
  self.zip.close()
```

And for the record:

```
>>> np.__version__
'1.8.0.dev-6dd20ff'
```

This ResourceWarning shows up in the numpy and scipy test suites. 



Discussion:
A work-around is to close the file held in the `zip` attribute before calling the `close()` method of the `NpzFile` instance:

```
$ python3 -q -Wdefault
>>> import numpy as np
>>> xy = np.load('xy.npz')
>>> xy.zip.fp.close()
>>> xy.close()   # No ResourceWarning.
>>>
```

That closure is part of zip.close(), which in turn should be called by xy.close(). Hmm...

I think this fixes it:

```
diff --git a/numpy/lib/npyio.py b/numpy/lib/npyio.py
index ba26fc2..8b6f55d 100644
--- a/numpy/lib/npyio.py
+++ b/numpy/lib/npyio.py
@@ -379,7 +379,7 @@ def load(file, mmap_mode=None):
         fid.seek(-N, 1) # back-up
         if magic.startswith(_ZIP_PREFIX):  # zip-file (assume .npz)
             own_fid = False
-            return NpzFile(fid, own_fid=own_fid)
+            return NpzFile(fid, own_fid=True)
         elif magic == format.MAGIC_PREFIX: # .npy file
             if mmap_mode:
                 return format.open_memmap(file, mode=mmap_mode)
```

I haven't read the surrounding code so this is a wild guess, but are you
sure it shouldn't instead be, delete the own_fid = false line?

On Fri, Jun 21, 2013 at 11:14 PM, Charles Harris
notifications@github.comwrote:

> I think this fixes it:
> 
> diff --git a/numpy/lib/npyio.py b/numpy/lib/npyio.py
> index ba26fc2..8b6f55d 100644
> --- a/numpy/lib/npyio.py
> +++ b/numpy/lib/npyio.py
> @@ -379,7 +379,7 @@ def load(file, mmap_mode=None):
>          fid.seek(-N, 1) # back-up
>          if magic.startswith(_ZIP_PREFIX):  # zip-file (assume .npz)
>              own_fid = False
> -            return NpzFile(fid, own_fid=own_fid)
> -            return NpzFile(fid, own_fid=True)
>        elif magic == format.MAGIC_PREFIX: # .npy file
>            if mmap_mode:
>                return format.open_memmap(file, mode=mmap_mode)
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/numpy/numpy/issues/3457#issuecomment-19841196
> .

No, otherwise the load function will close the file on exit. What the combination does is tranfer ownership from load to the NpzFile. That said, I suspect the load function could be refactored to be less obfuscated.

Oh, I see, there's a finally:. Even so, I think your patch is wrong for the
case where own_fid starts out False?

On Fri, Jun 21, 2013 at 11:46 PM, Charles Harris
notifications@github.comwrote:

> No, otherwise the load function will close the file on exit. What the
> combination does is tranfer ownership from load to the NpzFile. That said,
> I suspect the load function could be refactored to be less obfuscated.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/numpy/numpy/issues/3457#issuecomment-19842683
> .


Resolving patch:
diff --git a/numpy/lib/npyio.py b/numpy/lib/npyio.py
--- a/numpy/lib/npyio.py
+++ b/numpy/lib/npyio.py
@@ -377,15 +377,20 @@ def load(file, mmap_mode=None):
         N = len(format.MAGIC_PREFIX)
         magic = fid.read(N)
         fid.seek(-N, 1) # back-up
-        if magic.startswith(_ZIP_PREFIX):  # zip-file (assume .npz)
+        if magic.startswith(_ZIP_PREFIX):
+            # zip-file (assume .npz)
+            # Transfer file ownership to NpzFile
+            tmp = own_fid
             own_fid = False
-            return NpzFile(fid, own_fid=own_fid)
-        elif magic == format.MAGIC_PREFIX: # .npy file
+            return NpzFile(fid, own_fid=tmp)
+        elif magic == format.MAGIC_PREFIX:
+            # .npy file
             if mmap_mode:
                 return format.open_memmap(file, mode=mmap_mode)
             else:
                 return format.read_array(fid)
-        else:  # Try a pickle
+        else:
+            # Try a pickle
             try:
                 return pickle.load(fid)
             except:


Test patch:
diff --git a/numpy/lib/tests/test_io.py b/numpy/lib/tests/test_io.py
--- a/numpy/lib/tests/test_io.py
+++ b/numpy/lib/tests/test_io.py
@@ -221,7 +221,7 @@ def test_not_closing_opened_fid(self):
 
     def test_closing_fid(self):
         # Test that issue #1517 (too many opened files) remains closed
-        # It might be a "week" test since failed to get triggered on
+        # It might be a "weak" test since failed to get triggered on
         # e.g. Debian sid of 2012 Jul 05 but was reported to
         # trigger the failure on Ubuntu 10.04:
         # http://projects.scipy.org/numpy/ticket/1517#comment:2
@@ -250,6 +250,18 @@ def test_closing_fid(self):
         finally:
             os.remove(tmp)
 
+    def test_closing_zipfile_after_load(self):
+        # Check that zipfile owns file and can close it.
+        # This needs to pass a file name to load for the
+        # test.
+        fd, tmp = mkstemp(suffix='.npz')
+        os.close(fd)
+        np.savez(tmp, lab='place holder')
+        data = np.load(tmp)
+        fp = data.zip.fp
+        data.close()
+        assert_(fp.closed)
+
 
 class TestSaveTxt(TestCase):
     def test_array(self):