Repository: numpy/numpy

Issue:
savez fails on large array of objects
I'm getting a RuntimeError "File size unexpectedly exceeded ZIP64 limit" when using `savez` to save a few arrays to a file, the largest of which is an array of matrices each with a different number of rows (so that the top level array's dtype is object). As best I can tell, the condition used by numpy internally to decide whether or not to pass `force_zip64=True` to the zipfile open is checking if any of the arrays to be put into the archive has nbytes greater than 2^30. My array of matrices reports nbytes less than 2^30 but in reality the total number of bytes exceeds this (it's about 1.7GB). I'm using python 3.6 and numpy 1.14.2 on Ubuntu 16.04. 

The following code produces the error:
```python
test_data = np.asarray([np.random.rand(np.random.randint(50,100),4) for i in range(800000)])
np.savez('test', test_data=test_data)
```
whereas changing `np.random.randint(50,100)`->`75` produces no error.


Discussion:
Sounds like it's not taking into account the size of the pickled objects.

I bet we could just set force_zip64 to true unconditionally. What's the
worst case, it wastes a few dozen bytes? (Or we could do this for object
arrays only. I doubt the complexity of special casing different types is
worth it though.)

On Mar 20, 2018 23:36, "Patrick Komiske" <notifications@github.com> wrote:

> I'm getting a RuntimeError "File size unexpectedly exceeded ZIP64 limit"
> when using savez to save a few arrays to a file, the largest of which is
> an array of matrices each with a different number of rows (so that the top
> level array's dtype is object). As best I can tell, the condition used by
> numpy internally to decide whether or not to pass force_zip64=True to the
> zipfile open is checking if any of the arrays to be put into the archive
> has nbytes greater than 2*30. My array of matrices reports nbytes less
> than 2*30 but in reality the total number of bytes exceeds this (it's
> about 1.7GB). I'm using python 3.6 and numpy 1.14.2 on Ubuntu 16.04.
>
> The following code produces the error:
>
> test_data = np.asarray([np.random.rand(np.random.randint(50,100),4) for i in range(800000)])
> np.savez('test', test_data=test_data)
>
> whereas changing np.random.randint(50,100)->75 produces no error.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/numpy/numpy/issues/10776>, or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAlOaI5xWfAyR3gD0sv_GyEW0xirN6tIks5tgfT8gaJpZM4SzEgp>
> .
>

Setting force_zip64 to True unconditionally seems fine. Another alternative would be to allow the user to change this flag.
+1 from me on using it unconditionally
+1 After few months I still just got into this issue. 
Someone want to fix this?
I can try
Using the example and `force_zip64=True` unconditionally, I get a `MemoryError` on python3 at this [line](https://github.com/numpy/numpy/blob/v1.15.4/numpy/lib/format.py#L593)
```
pickle.dump(array, fp, protocol=2, **pickle_kwargs)
```
Increasing the protocol to 3, the test passes. Why do we limit ourselves to saving in protocol 2?
The pre-python3.6 path can write successfully.

Edit: we limit to protocol 2 for python2 back-compatibility
Maybe we should just leave the protocol to `pickle.HIGHEST_PROTOCOL`, or leave the argument out entirely?

Edit: Since that would make the saved file non-python2 readable. So should we postpone the fix to 1.17?
> Maybe we should just leave the protocol to pickle.HIGHEST_PROTOCOL

We might want to wait until we drop Python 2.7.

- Protocol version 3 was added in Python 3.0. It has explicit support for bytes objects and cannot be unpickled by Python 2.x. This is the default protocol, and the recommended protocol when compatibility with other Python 3 versions is required.
- Protocol version 4 was added in Python 3.4. It adds support for very large objects, pickling more kinds of objects, and some data format optimizations. Refer to PEP 3154 for information about improvements brought by protocol 4.


Resolving patch:
diff --git a/numpy/lib/format.py b/numpy/lib/format.py
--- a/numpy/lib/format.py
+++ b/numpy/lib/format.py
@@ -650,14 +650,13 @@ def write_array(fp, array, version=None, allow_pickle=True, pickle_kwargs=None):
 
     if array.dtype.hasobject:
         # We contain Python objects so we cannot write out the data
-        # directly.  Instead, we will pickle it out with version 2 of the
-        # pickle protocol.
+        # directly.  Instead, we will pickle it out
         if not allow_pickle:
             raise ValueError("Object arrays cannot be saved when "
                              "allow_pickle=False")
         if pickle_kwargs is None:
             pickle_kwargs = {}
-        pickle.dump(array, fp, protocol=2, **pickle_kwargs)
+        pickle.dump(array, fp, protocol=3, **pickle_kwargs)
     elif array.flags.f_contiguous and not array.flags.c_contiguous:
         if isfileobj(fp):
             array.T.tofile(fp)
diff --git a/numpy/lib/npyio.py b/numpy/lib/npyio.py
--- a/numpy/lib/npyio.py
+++ b/numpy/lib/npyio.py
@@ -734,8 +734,8 @@ def _savez(file, args, kwds, compress, allow_pickle=True, pickle_kwargs=None):
         for key, val in namedict.items():
             fname = key + '.npy'
             val = np.asanyarray(val)
-            force_zip64 = val.nbytes >= 2**30
-            with zipf.open(fname, 'w', force_zip64=force_zip64) as fid:
+            # always force zip64, gh-10776
+            with zipf.open(fname, 'w', force_zip64=True) as fid:
                 format.write_array(fid, val,
                                    allow_pickle=allow_pickle,
                                    pickle_kwargs=pickle_kwargs)


Test patch:
diff --git a/numpy/lib/tests/test_io.py b/numpy/lib/tests/test_io.py
--- a/numpy/lib/tests/test_io.py
+++ b/numpy/lib/tests/test_io.py
@@ -504,8 +504,6 @@ def test_complex_negative_exponent(self):
              b' (3.142e+00-2.718e+00j)  (3.142e+00-2.718e+00j)\n'])
 
 
-        
-
     def test_custom_writer(self):
 
         class CustomWriter(list):
@@ -574,6 +572,20 @@ def test_unicode_and_bytes_fmt(self, fmt, iotype):
         else:
             assert_equal(s.read(), b"%f\n" % 1.)
 
+    @pytest.mark.skipif(sys.platform=='win32',
+                        reason="large files cause problems")
+    @pytest.mark.slow
+    def test_large_zip(self):
+        # The test takes at least 6GB of memory, writes a file larger than 4GB
+        try:
+            a = 'a' * 6 * 1024 * 1024 * 1024
+            del a
+        except (MemoryError, OverflowError):
+            pytest.skip("Cannot allocate enough memory for test")
+        test_data = np.asarray([np.random.rand(np.random.randint(50,100),4)
+                               for i in range(800000)])
+        with tempdir() as tmpdir:
+            np.savez(os.path.join(tmpdir, 'test.npz'), test_data=test_data)
 
 class LoadTxtBase(object):
     def check_compressed(self, fopen, suffixes):
@@ -2379,7 +2391,7 @@ def test_savez_load(self):
             np.savez(path, lab='place holder')
             with np.load(path) as data:
                 assert_array_equal(data['lab'], 'place holder')
-    
+
     def test_savez_compressed_load(self):
         # Test that pathlib.Path instances can be used with savez.
         with temppath(suffix='.npz') as path: