Repository: scipy/scipy

Issue:
Fix mmio/fromfile on gzip on Python 3 (Trac #1627)
_Original ticket http://projects.scipy.org/scipy/ticket/1627 on 2012-03-22 by @pv, assigned to unknown._

Consider this:

```
Python 3.2.2
>>> from scipy.io import mmread
>>> mmread('illc1033.mtx.gz')
Traceback (most recent call last):
  ...
File ".../scipy/io/mmio.py", line 447, in _parse_body
    flat_data = flat_data.reshape(-1,3)
ValueError: total size of new array must be unchanged
```

with ftp://math.nist.gov/pub/MatrixMarket2/Harwell-Boeing/lsq/illc1033.mtx.gz

The reason is passing a GzipFile handle to `numpy.fromfile`. On Python 3, PyObject_AsFileDescriptor in `fromfile` succeeds on a GzipFile, although it should fail (there is no OS level file handle giving the uncompressed stream). As a consequence, the `fromfile` call in `mmread` apparently ends up reading an compressed data stream, which causes this error.

This should be worked around in `scipy.io.mmio` until Python 3 (and Numpy) are fixed.



Discussion:
I just tried this, and `np.fromfile` does work on the gzip handle:

```
$ python3 
Python 3.2.5 (default, Oct 29 2013, 10:03:36) 
[GCC 4.7.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> import gzip
>>> G = gzip.open("illc1033.mtx.gz", "r")
>>> np.fromfile(G)
array([  1.02866359e-064,   1.50168922e-076,   1.31490163e+294, ...,
         1.15488103e+120,   1.31512507e+171,   6.63818119e-221])
>>> 
>>> from scipy.io import mmread
>>> mmread("illc1033.mtx.gz")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.2/site-packages/scipy/io/mmio.py", line 73, in mmread
    return MMFile().read(source)
  File "/usr/lib64/python3.2/site-packages/scipy/io/mmio.py", line 326, in read
    return self._parse_body(stream)
  File "/usr/lib64/python3.2/site-packages/scipy/io/mmio.py", line 475, in _parse_body
    flat_data = flat_data.reshape(-1,3)
ValueError: total size of new array must be unchanged
```

Did I understand you wrong, @pv? This is on scipy 0.13.0.

`np.fromfile` reads the data from the raw stream and not the uncompressed one, so it doesn't work.
The numbers you get from fromfile are garbage.


Resolving patch:
diff --git a/scipy/io/mmio.py b/scipy/io/mmio.py
--- a/scipy/io/mmio.py
+++ b/scipy/io/mmio.py
@@ -12,6 +12,7 @@
 from __future__ import division, print_function, absolute_import
 
 import os
+import sys
 from numpy import asarray, real, imag, conj, zeros, ndarray, concatenate, \
                   ones, ascontiguousarray, vstack, savetxt, fromfile, fromstring
 from numpy.compat import asbytes, asstr
@@ -451,9 +452,15 @@ def _parse_body(self, stream):
                 return coo_matrix((rows, cols), dtype=dtype)
 
             try:
+                # passing a gzipped file to fromfile/fromstring doesn't work
+                # with Python3
+                if (sys.version_info >= (3, 0) and
+                        isinstance(stream, (gzip.GzipFile, bz2.BZ2File))):
+                    flat_data = fromstring(stream.read(), sep=' ')
                 # fromfile works for normal files
-                flat_data = fromfile(stream, sep=' ')
-            except:
+                else:
+                    flat_data = fromfile(stream, sep=' ')
+            except Exception:
                 # fallback - fromfile fails for some file-like objects
                 flat_data = fromstring(stream.read(), sep=' ')
 


Test patch:
diff --git a/scipy/io/tests/test_mmio.py b/scipy/io/tests/test_mmio.py
--- a/scipy/io/tests/test_mmio.py
+++ b/scipy/io/tests/test_mmio.py
@@ -283,6 +283,56 @@ def test_empty_write_read(self):
         b = mmread(fn).todense()
         assert_array_almost_equal(a,b)
 
+    def test_bzip2_py3(self):
+        # test if fix for #2152 works
+        try:
+            # bz2 module isn't always built when building Python.
+            import bz2
+        except:
+            return
+        I = array([0, 0, 1, 2, 3, 3, 3, 4])
+        J = array([0, 3, 1, 2, 1, 3, 4, 4])
+        V = array([1.0, 6.0, 10.5, 0.015, 250.5, -280.0, 33.32, 12.0])
+
+        b = scipy.sparse.coo_matrix((V,(I,J)),shape=(5,5))
+
+        fn = self.fn
+        mmwrite(fn, b)
+
+        fn_bzip2 = "%s.bz2" % fn
+        with open(fn, 'rb') as f_in:
+            f_out = bz2.BZ2File(fn_bzip2, 'wb')
+            f_out.write(f_in.read())
+            f_out.close()
+
+        a = mmread(fn_bzip2).todense()
+        assert_array_almost_equal(a, b.todense())
+
+    def test_gzip_py3(self):
+        # test if fix for #2152 works
+        try:
+            # gzip module can be missing from Python installation
+            import gzip
+        except:
+            return
+        I = array([0, 0, 1, 2, 3, 3, 3, 4])
+        J = array([0, 3, 1, 2, 1, 3, 4, 4])
+        V = array([1.0, 6.0, 10.5, 0.015, 250.5, -280.0, 33.32, 12.0])
+
+        b = scipy.sparse.coo_matrix((V,(I,J)),shape=(5,5))
+
+        fn = self.fn
+        mmwrite(fn, b)
+
+        fn_gzip = "%s.gz" % fn
+        with open(fn, 'rb') as f_in:
+            f_out = gzip.open(fn_gzip, 'wb')
+            f_out.write(f_in.read())
+            f_out.close()
+
+        a = mmread(fn_gzip).todense()
+        assert_array_almost_equal(a, b.todense())
+
     def test_real_write_read(self):
         I = array([0, 0, 1, 2, 3, 3, 3, 4])
         J = array([0, 3, 1, 2, 1, 3, 4, 4])