Repository: numpy/numpy

Issue:
Saving large array fails with savez_compressed() but works with save()
I got this strange problem on a CentOS box, Python 2.6.6, Numpy 1.9.1:

```
[minhle@node069 ~]$ python
Python 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.version.version
'1.9.1'
>>> a = np.ones((775890380,))
>>> np.savez_compressed('/home/minhle/scratch/test.npz', a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/minhle/.local/lib/python2.6/site-packages/numpy/lib/npyio.py", line 560, in savez_compressed
    _savez(file, args, kwds, True)
  File "/home/minhle/.local/lib/python2.6/site-packages/numpy/lib/npyio.py", line 597, in _savez
    format.write_array(fid, np.asanyarray(val))
  File "/home/minhle/.local/lib/python2.6/site-packages/numpy/lib/format.py", line 562, in write_array
    array.tofile(fp)
IOError: 775890380 requested and 233691638 written
>>> np.save('/home/minhle/scratch/test.npy', a)
>>> b = np.load('/home/minhle/scratch/test.npy')
>>> b[:10]
array([ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.])
>>> quit()
[minhle@node069 ~]$ cat /etc/*-release
CentOS release 6.5 (Final)
Cluster Manager v5.2
slave
LSB_VERSION=base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
CentOS release 6.5 (Final)
CentOS release 6.5 (Final)
```



Discussion:
Got the same error... any solution in sight?

I just got this error too but with `np.savez` as opposed to `np.savez_compressed`.

For future reference, using `np.save` instead of `np.savez[_compressed]` seems to work as @omerlevy's example code indicates.

So, is this an issue with `savez`? Anyone who knows numpy well have any ideas what's happening here?

`savez` needs to store files temporarily on disk, and perhaps you run out of space on `/tmp`.
Try setting `TMPDIR=$HOME/tmp` or so.

@pv Yeah, changing `TMPDIR` did the trick for me. It may also be worth noting that I'm on an NFS filesystem. I'm not sure what sort of effects, if any, that might have. I'm guessing that I'm into subtleties in the way that /tmp is configured.

However, I think numpy should make this issue explicit in the documentation and provide some semblance of an intelligent, approachable error message.

Agreed.  I ran onto this issue on HPC resources too and I agree that it would be good if the error / warning message recommended checking to make sure `TMPDIR` was set properly with adequate space.  

Ran into the same problem (kudos to ngoldbaum@IRC for pointing me here).  I understand that for temp storage TEMPDIR is usually the logical location. BUT when dealing with large datasets, there is a reason why we might want to savez to a different location and /tmp would never be big enough.  So why not to follow any other downloader's behavior and use targetfilename + '_tempSMTHRANDOM' suffix instead.  This has higher possibility to succeed since would use the target location/partition which should have adequate amount of storage.  Also error message should include location of that temp file to give users sensible feedback


Resolving patch:
diff --git a/numpy/lib/npyio.py b/numpy/lib/npyio.py
--- a/numpy/lib/npyio.py
+++ b/numpy/lib/npyio.py
@@ -627,7 +627,11 @@ def _savez(file, args, kwds, compress, allow_pickle=True, pickle_kwargs=None):
     zipf = zipfile_factory(file, mode="w", compression=compression)
 
     # Stage arrays in a temporary file on disk, before writing to zip.
-    fd, tmpfile = tempfile.mkstemp(suffix='-numpy.npy')
+
+    # Since target file might be big enough to exceed capacity of a global
+    # temporary directory, create temp file side-by-side with the target file.
+    file_dir, file_prefix = os.path.split(file) if _is_string_like(file) else (None, 'tmp')
+    fd, tmpfile = tempfile.mkstemp(prefix=file_prefix, dir=file_dir, suffix='-numpy.npy')
     os.close(fd)
     try:
         for key, val in namedict.items():
@@ -640,6 +644,8 @@ def _savez(file, args, kwds, compress, allow_pickle=True, pickle_kwargs=None):
                 fid.close()
                 fid = None
                 zipf.write(tmpfile, arcname=fname)
+            except IOError as exc:
+                raise IOError("Failed to write to %s: %s" % (tmpfile, exc))
             finally:
                 if fid:
                     fid.close()


Test patch: