Repository: numpy/numpy

Issue:
numpy.lib.format.read_array_header_*  don't return as specified
See [here](https://github.com/numpy/numpy/blob/master/numpy/lib/format.py#L392) and [here](https://github.com/numpy/numpy/blob/master/numpy/lib/format.py#L425). Fix should be adding `return` to these lines. Fix incoming.

read_array_header_1_0 returns None
When trying to read from a .npy file, the np.lib.format-read_array_header_1_0 function returns None instead of the intended triple of shape, fortran, dtype. 

Seems to me that numpy/lib/format.py line 379 needs a "return" to get the tuple back from the call.

Code snippet to reproduce the error:
# Create some junk data:

import numpy as np
filename = 'mp.npy'
data = np.random.random((10,100)).astype(np.float32)
np.save(filename, data)
# The error is caused by:

with open(filename, 'rb') as fhandle:
    major, minor = np.lib.format.read_magic(fhandle)
    shape, fortran, dtype = np.lib.format.read_array_header_1_0(fhandle)



Discussion:

Hmm, same with `read_array_header_2_0`. The functions are not actually used anywhere and are only tested to see if they raise an appropriate error. Want to put together a PR?

Damn, that was a fast turn around

EEK I don’t even know what PR means… Pull Request?

Is there an FAQ for fork/edit/pull request, to guide me through?  Also, how do I run the unit tests to make sure I don’t break anything when I make the change.

Best,
-n

> On Feb 24, 2015, at 3:36 PM, Charles Harris notifications@github.com wrote:
> 
> Hmm, same with read_array_header_2_0. The functions are not actually used anywhere and are only tested to see if they raise an appropriate error. Want to put together a PR?
> 
> —
> Reply to this email directly or view it on GitHub https://github.com/numpy/numpy/issues/5602#issuecomment-75874015.

Take a look at https://github.com/numpy/numpy/blob/master/CONTRIBUTING.md and follow the links. You can run tests from your local repo with `python runtests.py`, start with `python runtests.py --help`. Feel free to ask for help.

Also note that when you submit a pull request, then the tests get run
automatically (in a bunch of different configurations) and the results
listed on the pull request web page. So you'll still probably want to run
the tests yourself to catch silly bugs, but you don't have to be too
paranoid, because you will have that backup check.

On Tue, Feb 24, 2015 at 4:20 PM, Charles Harris notifications@github.com
wrote:

> Take a look at https://github.com/numpy/numpy/blob/master/CONTRIBUTING.md
> and follow the links. You can run tests from your local repo with python
> runtests.py, start with python runtests.py --help. Feel free to ask for
> help.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/numpy/numpy/issues/5602#issuecomment-75880407.

## 

Nathaniel J. Smith -- http://vorpus.org

Thanks for the help  Here is the test I wrote to exercise the issue:

# test_read_array_header_1_0.py

import sys
sys.path.insert(1, '/local1/numpy-dev/lib/python2.7/site-packages')
import numpy as np

def test_read_array_header_1_0():
    filename = 'mp.npy'
    data = np.random.random((10,100)).astype(np.float32)
    np.save(filename, data)

```
with open(filename, 'rb') as fhandle:
    major, minor = np.lib.format.read_magic(fhandle)
    shape, fortran, dtype = np.lib.format.read_array_header_1_0(fhandle)
```

if **name** == "**main**":
    test_read_array_header_1_0()

I am going to update my fork, and then submit a pull request.  Thanks for the help!

OK I see that travis-ci is passing the changes in my pull request: https://travis-ci.org/numpy/numpy/builds/52076121

I probably don't have permissions to push upstream, what is the next step to finish off?

Wait for code review. One of the developers will do the actual merge after review. No one escapes that bottleneck ;) 

I have been writing up some tests in lib/tests/test_format.py, and found that there is no header_data_from_array_2_0 function in format.py.  Is this intentional?  Should I be doing something else to create the tests? 

https://github.com/nicain/numpy/commit/0afaecb139a879efa5c0d8884eb04317729532ac


Resolving patch:
diff --git a/numpy/lib/format.py b/numpy/lib/format.py
--- a/numpy/lib/format.py
+++ b/numpy/lib/format.py
@@ -314,21 +314,19 @@ def _write_array_header(fp, d, version=None):
     header = header + ' '*topad + '\n'
     header = asbytes(_filter_header(header))
 
-    if len(header) >= (256*256) and version == (1, 0):
-        raise ValueError("header does not fit inside %s bytes required by the"
-                         " 1.0 format" % (256*256))
-    if len(header) < (256*256):
-        header_len_str = struct.pack('<H', len(header))
+    hlen = len(header)
+    if hlen < 256*256 and version in (None, (1, 0)):
         version = (1, 0)
-    elif len(header) < (2**32):
-        header_len_str = struct.pack('<I', len(header))
+        header_prefix = magic(1, 0) + struct.pack('<H', hlen)
+    elif hlen < 2**32 and version in (None, (2, 0)):
         version = (2, 0)
+        header_prefix = magic(2, 0) + struct.pack('<I', hlen)
     else:
-        raise ValueError("header does not fit inside 4 GiB required by "
-                         "the 2.0 format")
+        msg = "Header length %s too big for version=%s"
+        msg %= (hlen, version)
+        raise ValueError(msg)
 
-    fp.write(magic(*version))
-    fp.write(header_len_str)
+    fp.write(header_prefix)
     fp.write(header)
     return version
 
@@ -389,7 +387,7 @@ def read_array_header_1_0(fp):
         If the data is invalid.
 
     """
-    _read_array_header(fp, version=(1, 0))
+    return _read_array_header(fp, version=(1, 0))
 
 def read_array_header_2_0(fp):
     """
@@ -422,7 +420,7 @@ def read_array_header_2_0(fp):
         If the data is invalid.
 
     """
-    _read_array_header(fp, version=(2, 0))
+    return _read_array_header(fp, version=(2, 0))
 
 
 def _filter_header(s):


Test patch:
diff --git a/numpy/lib/tests/test_format.py b/numpy/lib/tests/test_format.py
--- a/numpy/lib/tests/test_format.py
+++ b/numpy/lib/tests/test_format.py
@@ -694,6 +694,26 @@ def test_write_version():
     '',
 ])
 
+def test_read_magic():
+    s1 = BytesIO()
+    s2 = BytesIO()
+
+    arr = np.ones((3, 6), dtype=float)
+
+    format.write_array(s1, arr, version=(1, 0))
+    format.write_array(s2, arr, version=(2, 0))
+
+    s1.seek(0)
+    s2.seek(0)
+
+    version1 = format.read_magic(s1)
+    version2 = format.read_magic(s2)
+
+    assert_(version1 == (1, 0))
+    assert_(version2 == (2, 0))
+
+    assert_(s1.tell() == format.MAGIC_LEN)
+    assert_(s2.tell() == format.MAGIC_LEN)
 
 def test_read_magic_bad_magic():
     for magic in malformed_magic:
@@ -724,6 +744,30 @@ def test_large_header():
     assert_raises(ValueError, format.write_array_header_1_0, s, d)
 
 
+def test_read_array_header_1_0():
+    s = BytesIO()
+
+    arr = np.ones((3, 6), dtype=float)
+    format.write_array(s, arr, version=(1, 0))
+
+    s.seek(format.MAGIC_LEN)
+    shape, fortran, dtype = format.read_array_header_1_0(s)
+
+    assert_((shape, fortran, dtype) == ((3, 6), False, float))
+
+
+def test_read_array_header_2_0():
+    s = BytesIO()
+
+    arr = np.ones((3, 6), dtype=float)
+    format.write_array(s, arr, version=(2, 0))
+
+    s.seek(format.MAGIC_LEN)
+    shape, fortran, dtype = format.read_array_header_2_0(s)
+
+    assert_((shape, fortran, dtype) == ((3, 6), False, float))
+
+
 def test_bad_header():
     # header of length less than 2 should fail
     s = BytesIO()