Repository: numpy/numpy

Issue:
numpy npz files with v1 and v2 formats are not interoperable between python 2 and python 3.
The header is a python literal, unfortunately this leaves the numpy file unreadable when it is an array and the literal for array dimensions are Long integers.  e.g. "{'descr': '<i2', 'fortran_order': False, 'shape': (100L, 200L), }    \n"

When loading with an array in the npz file saved with python 2.x and opening with python 3.x we get the following error.

Exception: SyntaxError('invalid syntax', ('<unknown>', 1, 55, "{'descr': '<i2', 'fortran_order': False, 'shape': (100L, 200L), }    \n"

The 100L and 200L are invalid python 3 literals.
A fix is to run a substitution on them, such as this one, for old files.
header = re.sub(b"(\d+)L", b'\1', header)

numpy npz files with v1 and v2 formats are not interoperable between python 2 and python 3.
The header is a python literal, unfortunately this leaves the numpy file unreadable when it is an array and the literal for array dimensions are Long integers.  e.g. "{'descr': '<i2', 'fortran_order': False, 'shape': (100L, 200L), }    \n"

When loading with an array in the npz file saved with python 2.x and opening with python 3.x we get the following error.

Exception: SyntaxError('invalid syntax', ('<unknown>', 1, 55, "{'descr': '<i2', 'fortran_order': False, 'shape': (100L, 200L), }    \n"

The 100L and 200L are invalid python 3 literals.
A fix is to run a substitution on them, such as this one, for old files.
header = re.sub(b"(\d+)L", b'\1', header)



Discussion:
``` python
header = re.sub(b"(\d+)L", b'\1', header)  # Strip the L from long literals of python 2.x source code.
```

Has been tested successfully to allow python 3 open python 2 npz file.
Added in function `_read_array_header` before `d = safe_eval(header)`
In file format.py in numpy/lib

@rkern Thoughts?

It would probably be worth moving to standardized JSON for the header information.

hm too bad this wasn't though of before we did 2.0
the format is still new enough that we might get away with changing it in 1.9.1, but that wouldn't be very nice.

stripping away L would break record keys which are integers followed by L. We'd have to add special logic to where exactly integers go into the header

Hi,
I'm completely new here - but I have used numpy a bit and recently switched to py3.4.
I have anaconda installed.

I have also noticed the issue with incompatible .npz files, and was very happy when I found this seemingly easy work-around.
Tried the listed solution here but could not get it to work.
Added import re after import struct (line 420 in format.py)
Added header re.sub..... before d = safe_eval(header) (line 440 in format.py)
(I know it was in the changed file since I added a print statement in the changed format.py)
I also tried to extract the npz file manually to see if it was possible to at least get to the npy's this way - but that did not work either (header parse error - which I found more than a bit strange when looking into an extracted npy file but maybe I did something wrong)

So now I have given up and am running a copy of a couple of 100's GB data.
Will then write a script for reading the npz in py2.7, saving it in npy, every npz I have has multiple npy's.
After that I will try to recreate the data to be readable in 3.4 the same way, since there are dependencies in the convenience reading I have set up (an easy way ot find the right data in the 100's of GB's), and if I have to go through the hassle of a full translation of the data I don't want to go through the hassle of redoing the complete reading functions because of change of format.

I hope you come up with something simpler than what I'm about to do :-)

well it just doesn't seem to be my day today - I did not have enogh space to safely copy and mess around with the data - so I guess it will have to wait until I have arranged some extra saving space...

I share Julian's concern about using regex to try and catch all and only
integers inside an arbitrary language like this. (I guess strings can't
nest so we aren't quite in the this situation: http://
http://stackoverflow.com/a/1732454stackoverflow.com
http://stackoverflow.com/a/1732454/a/1732454
http://stackoverflow.com/a/1732454 - but it definitely needs some care.)

I think the simplest thing might be to tokenize the input with Python's
built in tokenizer and fix that up before trying to safe_eval.

hm too bad this wasn't though of before we did 2.0
the format is still new enough that we might get away with changing it in
1.9.1, but that wouldn't be very nice.

stripping away L would break record keys which are integers followed by L.
We'd have to add special logic to where exactly integers go into the header

—
Reply to this email directly or view it on GitHub
https://github.com/numpy/numpy/issues/5170#issuecomment-58715359.

Could work. It seems like Python 3's `tokenize` routine will recognize `100L` as a `NUMBER (string='100')` followed by a `NAME (string='L')` It would be straightforward enough to drop such `NAME` tokens when the `end` of the `NUMBER` precisely matches the `start` of the `NAME`.

We don't even need to worry about "precisely matches" unless we want to do
strict validation - number followed by name is never legal python.
Accepting "1 L" is not a big deal.
On 11 Oct 2014 16:08, "Robert Kern" notifications@github.com wrote:

> Could work. It seems like Python 3's tokenize routine will recognize 100L
> as a NUMBER (string='100') followed by a NAME (string='L') It would be
> straightforward enough to drop such NAME tokens when the end of the NUMBER
> precisely matches the start of the NAME.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/numpy/numpy/issues/5170#issuecomment-58752494.

Hi again,

I did go out to buy some more HD space and started copying.
While waiting for the copying and still by the computer I thought I'd give it a go once again....

After viewing the string and only being able to find the capital L in the shape position and no reason for why it ever should be anywhere else - just considering myself) I just went ahead and tried with a simple replace instead of the reg and I got it to work for me at least.
I used: header = header.replace(b'L', b'')

I have tried it on a couple of files and it works, but I have not tried on anything else than .npz files.
I guess there is a risk of breaking something - but it won't be too hard for me to just comment out the line I added in that case.

All in all, I cancelled the copying and am just glad that I did not have to spend the better part of the  weekend to get my old .npz files working again :-)

Don't change the header length. Replace `b'L'` with `b' '`.

oki, will do - thanks.

The re has a slight error.  It needs to be b'\1'. Sorry.

> On Oct 11, 2014, at 3:47 AM, ahed87 notifications@github.com wrote:
> 
> Hi,
> I'm completely new here - but I have used numpy a bit and recently switched to py3.4.
> I have anaconda installed.
> 
> I have also noticed the issue with incompatible .npz files, and was very happy when I found this seemingly easy work-around.
> Tried the listed solution here but could not get it to work.
> Added import re after import struct (line 420 in format.py)
> Added header re.sub..... before d = safe_eval(header) (line 440 in format.py)
> (I know it was in the changed file since I added a print statement in the changed format.py)
> I also tried to extract the npz file manually to see if it was possible to at least get to the npy's this way - but that did not work either (header parse error - which I found more than a bit strange when looking into an extracted npy file but maybe I did something wrong)
> 
> So now I have given up and am running a copy of a couple of 100's GB data.
> Will then write a script for reading the npz in py2.7, saving it in npy, every npz I have has multiple npy's.
> After that I will try to recreate the data to be readable in 3.4 the same way, since there are dependencies in the convenience reading I have set up (an easy way ot find the right data in the 100's of GB's), and if I have to go through the hassle of a full translation of the data I don't want to go through the hassle of redoing the complete reading functions because of change of format.
> 
> I hope you come up with something simpler than what I'm about to do :-)
> 
> well it just doesn't seem to be my day today - I did not have enogh space to safely copy and mess around with the data - so I guess it will have to wait until I have arranged some extra saving space...
> 
> —
> Reply to this email directly or view it on GitHub.

This looks like a fix worth getting into 1.9.1. The removal of the `L` needs to be Python version dependent, correct?

It's probably best to though not strictly necessary. It's cheaper and most robust to test the Python version quickly than to do the processing for no effect.

Python 2.x has int and long types for integers.  Python 3 dropped that difference and the "L" was removed.  Python 3 needs it removed,  I think that 2.x can live without it.  

A number literal should automatically up convert to a long if big enough.  Will bumpy care if it gets a int instead of a long?

> On Oct 11, 2014, at 10:41 AM, Charles Harris notifications@github.com wrote:
> 
> This looks like a fix worth getting into 1.9.1. The removal of the L needs to be Python version dependent, correct?
> 
> —
> Reply to this email directly or view it on GitHub.

No, numpy should convert it again anyway, it would be OK to always remove the L for metadata stuff.

@seberg I'm concerned about the size of long. Is the string form always correctly converted?

Seems so:

```
[~]
|1> from numpy.lib.utils import safe_eval

[~]
|2> import sys

[~]
|3> sys.maxsize
9223372036854775807

[~]
|4> safe_eval('(19223372036854775807, 19223372036854775807)')
(19223372036854775807L, 19223372036854775807L)
```

Hmm, maybe safe_eval can handle the `L` compatibility problem as well.

https://gist.github.com/njsmith/0651d1d3266bf017c9e2 passes its test on
both py2 and py3.

On Sat, Oct 11, 2014 at 7:36 PM, Charles Harris notifications@github.com
wrote:

> Hmm, maybe safe_eval can handle the L compatibility problem as well.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/numpy/numpy/issues/5170#issuecomment-58759669.

## 

Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org

(NB it doesn't actually remove the L on py2, because on py2 "1L" is a
treated as a single token. But who cares.)

On Sat, Oct 11, 2014 at 7:40 PM, Nathaniel Smith njs@pobox.com wrote:

> https://gist.github.com/njsmith/0651d1d3266bf017c9e2 passes its test on
> both py2 and py3.
> 
> On Sat, Oct 11, 2014 at 7:36 PM, Charles Harris notifications@github.com
> wrote:
> 
> > Hmm, maybe safe_eval can handle the L compatibility problem as well.
> > 
> > —
> > Reply to this email directly or view it on GitHub
> > https://github.com/numpy/numpy/issues/5170#issuecomment-58759669.
> 
> ## 
> 
> Nathaniel J. Smith
> Postdoctoral researcher - Informatics - University of Edinburgh
> http://vorpus.org

## 

Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org

@njsmith What I meant is that `safe_eval` should work with the 'L' in python3, which it doesn't currently:

```
>>> np.safe_eval('1000L')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/charris/.local/lib/python3.3/site-packages/numpy/lib/utils.py", line 1136, in safe_eval
    ast = compiler.parse(source, mode="eval")
  File "/usr/lib64/python3.3/ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 1
    1000L
        ^
```

Yes, you just call @njsmith's `fixup_header()` as the first step in `safe_eval()`.

safe_eval can't work with 1L because it is invalid syntax, thus the approach to use the tokenizer to fix the syntax first and then AST parse it
we should probably also just use ast.literal_eval

@juliantaylor The `safe_eval` can be fixed as Robert suggests, making it truly 'safe' and working across python versions, which seems like a useful update.

I doubt it matters, but since these are field shapes if I understand it right... Field shapes cannot actually be large enough to not fit normal ints.

on win64 they probably can

actually how is loading 64 bit requiring data on 32 bit OS handled? will stuff simply overflow or is there a failsafe?

Fields are limited to normal ints, a bit weird, but it is stuck in any case.

Edit: Sorry these are not just fields of course...

@hardkrash 
It did something for me, when I printed it it had replaced the L with a \x01 or similar (don't remember the exact number anymore), but I'm not good enough with re to do what I want (especially when encoding comes into play) - I can just do planless trial and error with re & different encodings - so I try to stay away from it and fell back on the simpler to grasp replace.
Regardless - a huge thanks for letting us know the problem area and workaround!!

b'\1' turns into the character 0x01,  it needs to be the back reference string '\1'. To get that it needs to be b'\1'. By escaping the backslash.

The solution that they are proposing with the cleaned up tokenization seems the best to me.

> On Oct 12, 2014, at 6:47 AM, ahed87 notifications@github.com wrote:
> 
> @hardkrash 
> It did something for me, when I printed it it had replaced the L with a \x01 or similar (don't remember the exact number anymore), but I'm not good enough with re to do what I want (especially when encoding comes into play) - I can just do planless trial and error with re & different encodings - so I try to stay away from it and fell back on the simpler to grasp replace.
> Regardless - a huge thanks for letting us know the problem area and workaround!!
> 
> —
> Reply to this email directly or view it on GitHub.

``` python
header = re.sub(b"(\d+)L", b'\1', header)  # Strip the L from long literals of python 2.x source code.
```

Has been tested successfully to allow python 3 open python 2 npz file.
Added in function `_read_array_header` before `d = safe_eval(header)`
In file format.py in numpy/lib

@rkern Thoughts?

It would probably be worth moving to standardized JSON for the header information.

hm too bad this wasn't though of before we did 2.0
the format is still new enough that we might get away with changing it in 1.9.1, but that wouldn't be very nice.

stripping away L would break record keys which are integers followed by L. We'd have to add special logic to where exactly integers go into the header

Hi,
I'm completely new here - but I have used numpy a bit and recently switched to py3.4.
I have anaconda installed.

I have also noticed the issue with incompatible .npz files, and was very happy when I found this seemingly easy work-around.
Tried the listed solution here but could not get it to work.
Added import re after import struct (line 420 in format.py)
Added header re.sub..... before d = safe_eval(header) (line 440 in format.py)
(I know it was in the changed file since I added a print statement in the changed format.py)
I also tried to extract the npz file manually to see if it was possible to at least get to the npy's this way - but that did not work either (header parse error - which I found more than a bit strange when looking into an extracted npy file but maybe I did something wrong)

So now I have given up and am running a copy of a couple of 100's GB data.
Will then write a script for reading the npz in py2.7, saving it in npy, every npz I have has multiple npy's.
After that I will try to recreate the data to be readable in 3.4 the same way, since there are dependencies in the convenience reading I have set up (an easy way ot find the right data in the 100's of GB's), and if I have to go through the hassle of a full translation of the data I don't want to go through the hassle of redoing the complete reading functions because of change of format.

I hope you come up with something simpler than what I'm about to do :-)

well it just doesn't seem to be my day today - I did not have enogh space to safely copy and mess around with the data - so I guess it will have to wait until I have arranged some extra saving space...

I share Julian's concern about using regex to try and catch all and only
integers inside an arbitrary language like this. (I guess strings can't
nest so we aren't quite in the this situation: http://
http://stackoverflow.com/a/1732454stackoverflow.com
http://stackoverflow.com/a/1732454/a/1732454
http://stackoverflow.com/a/1732454 - but it definitely needs some care.)

I think the simplest thing might be to tokenize the input with Python's
built in tokenizer and fix that up before trying to safe_eval.

hm too bad this wasn't though of before we did 2.0
the format is still new enough that we might get away with changing it in
1.9.1, but that wouldn't be very nice.

stripping away L would break record keys which are integers followed by L.
We'd have to add special logic to where exactly integers go into the header

—
Reply to this email directly or view it on GitHub
https://github.com/numpy/numpy/issues/5170#issuecomment-58715359.

Could work. It seems like Python 3's `tokenize` routine will recognize `100L` as a `NUMBER (string='100')` followed by a `NAME (string='L')` It would be straightforward enough to drop such `NAME` tokens when the `end` of the `NUMBER` precisely matches the `start` of the `NAME`.

We don't even need to worry about "precisely matches" unless we want to do
strict validation - number followed by name is never legal python.
Accepting "1 L" is not a big deal.
On 11 Oct 2014 16:08, "Robert Kern" notifications@github.com wrote:

> Could work. It seems like Python 3's tokenize routine will recognize 100L
> as a NUMBER (string='100') followed by a NAME (string='L') It would be
> straightforward enough to drop such NAME tokens when the end of the NUMBER
> precisely matches the start of the NAME.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/numpy/numpy/issues/5170#issuecomment-58752494.

Hi again,

I did go out to buy some more HD space and started copying.
While waiting for the copying and still by the computer I thought I'd give it a go once again....

After viewing the string and only being able to find the capital L in the shape position and no reason for why it ever should be anywhere else - just considering myself) I just went ahead and tried with a simple replace instead of the reg and I got it to work for me at least.
I used: header = header.replace(b'L', b'')

I have tried it on a couple of files and it works, but I have not tried on anything else than .npz files.
I guess there is a risk of breaking something - but it won't be too hard for me to just comment out the line I added in that case.

All in all, I cancelled the copying and am just glad that I did not have to spend the better part of the  weekend to get my old .npz files working again :-)

Don't change the header length. Replace `b'L'` with `b' '`.

oki, will do - thanks.

The re has a slight error.  It needs to be b'\1'. Sorry.

> On Oct 11, 2014, at 3:47 AM, ahed87 notifications@github.com wrote:
> 
> Hi,
> I'm completely new here - but I have used numpy a bit and recently switched to py3.4.
> I have anaconda installed.
> 
> I have also noticed the issue with incompatible .npz files, and was very happy when I found this seemingly easy work-around.
> Tried the listed solution here but could not get it to work.
> Added import re after import struct (line 420 in format.py)
> Added header re.sub..... before d = safe_eval(header) (line 440 in format.py)
> (I know it was in the changed file since I added a print statement in the changed format.py)
> I also tried to extract the npz file manually to see if it was possible to at least get to the npy's this way - but that did not work either (header parse error - which I found more than a bit strange when looking into an extracted npy file but maybe I did something wrong)
> 
> So now I have given up and am running a copy of a couple of 100's GB data.
> Will then write a script for reading the npz in py2.7, saving it in npy, every npz I have has multiple npy's.
> After that I will try to recreate the data to be readable in 3.4 the same way, since there are dependencies in the convenience reading I have set up (an easy way ot find the right data in the 100's of GB's), and if I have to go through the hassle of a full translation of the data I don't want to go through the hassle of redoing the complete reading functions because of change of format.
> 
> I hope you come up with something simpler than what I'm about to do :-)
> 
> well it just doesn't seem to be my day today - I did not have enogh space to safely copy and mess around with the data - so I guess it will have to wait until I have arranged some extra saving space...
> 
> —
> Reply to this email directly or view it on GitHub.

This looks like a fix worth getting into 1.9.1. The removal of the `L` needs to be Python version dependent, correct?

It's probably best to though not strictly necessary. It's cheaper and most robust to test the Python version quickly than to do the processing for no effect.

Python 2.x has int and long types for integers.  Python 3 dropped that difference and the "L" was removed.  Python 3 needs it removed,  I think that 2.x can live without it.  

A number literal should automatically up convert to a long if big enough.  Will bumpy care if it gets a int instead of a long?

> On Oct 11, 2014, at 10:41 AM, Charles Harris notifications@github.com wrote:
> 
> This looks like a fix worth getting into 1.9.1. The removal of the L needs to be Python version dependent, correct?
> 
> —
> Reply to this email directly or view it on GitHub.

No, numpy should convert it again anyway, it would be OK to always remove the L for metadata stuff.

@seberg I'm concerned about the size of long. Is the string form always correctly converted?

Seems so:

```
[~]
|1> from numpy.lib.utils import safe_eval

[~]
|2> import sys

[~]
|3> sys.maxsize
9223372036854775807

[~]
|4> safe_eval('(19223372036854775807, 19223372036854775807)')
(19223372036854775807L, 19223372036854775807L)
```

Hmm, maybe safe_eval can handle the `L` compatibility problem as well.

https://gist.github.com/njsmith/0651d1d3266bf017c9e2 passes its test on
both py2 and py3.

On Sat, Oct 11, 2014 at 7:36 PM, Charles Harris notifications@github.com
wrote:

> Hmm, maybe safe_eval can handle the L compatibility problem as well.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/numpy/numpy/issues/5170#issuecomment-58759669.

## 

Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org

(NB it doesn't actually remove the L on py2, because on py2 "1L" is a
treated as a single token. But who cares.)

On Sat, Oct 11, 2014 at 7:40 PM, Nathaniel Smith njs@pobox.com wrote:

> https://gist.github.com/njsmith/0651d1d3266bf017c9e2 passes its test on
> both py2 and py3.
> 
> On Sat, Oct 11, 2014 at 7:36 PM, Charles Harris notifications@github.com
> wrote:
> 
> > Hmm, maybe safe_eval can handle the L compatibility problem as well.
> > 
> > —
> > Reply to this email directly or view it on GitHub
> > https://github.com/numpy/numpy/issues/5170#issuecomment-58759669.
> 
> ## 
> 
> Nathaniel J. Smith
> Postdoctoral researcher - Informatics - University of Edinburgh
> http://vorpus.org

## 

Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org

@njsmith What I meant is that `safe_eval` should work with the 'L' in python3, which it doesn't currently:

```
>>> np.safe_eval('1000L')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/charris/.local/lib/python3.3/site-packages/numpy/lib/utils.py", line 1136, in safe_eval
    ast = compiler.parse(source, mode="eval")
  File "/usr/lib64/python3.3/ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 1
    1000L
        ^
```

Yes, you just call @njsmith's `fixup_header()` as the first step in `safe_eval()`.

safe_eval can't work with 1L because it is invalid syntax, thus the approach to use the tokenizer to fix the syntax first and then AST parse it
we should probably also just use ast.literal_eval

@juliantaylor The `safe_eval` can be fixed as Robert suggests, making it truly 'safe' and working across python versions, which seems like a useful update.

I doubt it matters, but since these are field shapes if I understand it right... Field shapes cannot actually be large enough to not fit normal ints.

on win64 they probably can

actually how is loading 64 bit requiring data on 32 bit OS handled? will stuff simply overflow or is there a failsafe?

Fields are limited to normal ints, a bit weird, but it is stuck in any case.

Edit: Sorry these are not just fields of course...

@hardkrash 
It did something for me, when I printed it it had replaced the L with a \x01 or similar (don't remember the exact number anymore), but I'm not good enough with re to do what I want (especially when encoding comes into play) - I can just do planless trial and error with re & different encodings - so I try to stay away from it and fell back on the simpler to grasp replace.
Regardless - a huge thanks for letting us know the problem area and workaround!!

b'\1' turns into the character 0x01,  it needs to be the back reference string '\1'. To get that it needs to be b'\1'. By escaping the backslash.

The solution that they are proposing with the cleaned up tokenization seems the best to me.

> On Oct 12, 2014, at 6:47 AM, ahed87 notifications@github.com wrote:
> 
> @hardkrash 
> It did something for me, when I printed it it had replaced the L with a \x01 or similar (don't remember the exact number anymore), but I'm not good enough with re to do what I want (especially when encoding comes into play) - I can just do planless trial and error with re & different encodings - so I try to stay away from it and fell back on the simpler to grasp replace.
> Regardless - a huge thanks for letting us know the problem area and workaround!!
> 
> —
> Reply to this email directly or view it on GitHub.


Resolving patch:
diff --git a/numpy/lib/format.py b/numpy/lib/format.py
--- a/numpy/lib/format.py
+++ b/numpy/lib/format.py
@@ -141,7 +141,7 @@
 import io
 import warnings
 from numpy.lib.utils import safe_eval
-from numpy.compat import asbytes, isfileobj, long, basestring
+from numpy.compat import asbytes, asstr, isfileobj, long, basestring
 
 if sys.version_info[0] >= 3:
     import pickle
@@ -410,6 +410,45 @@ def read_array_header_2_0(fp):
     """
     _read_array_header(fp, version=(2, 0))
 
+
+def _filter_header(s):
+    """Clean up 'L' in npz header ints.
+
+    Cleans up the 'L' in strings representing integers. Needed to allow npz
+    headers produced in Python2 to be read in Python3.
+
+    Parameters
+    ----------
+    s : byte string
+        Npy file header.
+
+    Returns
+    -------
+    header : str
+        Cleaned up header.
+
+    """
+    import tokenize
+    if sys.version_info[0] >= 3:
+        from io import StringIO
+    else:
+        from StringIO import StringIO
+ 
+    tokens = []
+    last_token_was_number = False
+    for token in tokenize.generate_tokens(StringIO(asstr(s)).read):
+        token_type = token[0]
+        token_string = token[1]
+        if (last_token_was_number and
+                token_type == tokenize.NAME and
+                token_string == "L"):
+            continue
+        else:
+            tokens.append(token)
+        last_token_was_number = (token_type == tokenize.NUMBER)
+    return tokenize.untokenize(tokens)
+
+ 
 def _read_array_header(fp, version):
     """
     see read_array_header_1_0
@@ -434,6 +473,7 @@ def _read_array_header(fp, version):
     #   "shape" : tuple of int
     #   "fortran_order" : bool
     #   "descr" : dtype.descr
+    header = _filter_header(header)
     try:
         d = safe_eval(header)
     except SyntaxError as e:


Test patch:
diff --git a/numpy/lib/tests/data/python3.npy b/numpy/lib/tests/data/python3.npy
new file mode 100644
Binary files /dev/null and b/numpy/lib/tests/data/python3.npy differ
diff --git a/numpy/lib/tests/data/win64python2.npy b/numpy/lib/tests/data/win64python2.npy
new file mode 100644
Binary files /dev/null and b/numpy/lib/tests/data/win64python2.npy differ
diff --git a/numpy/lib/tests/test_format.py b/numpy/lib/tests/test_format.py
--- a/numpy/lib/tests/test_format.py
+++ b/numpy/lib/tests/test_format.py
@@ -524,6 +524,16 @@ def test_compressed_roundtrip():
     assert_array_equal(arr, arr1)
 
 
+def test_python2_python3_interoperability():
+    if sys.version_info[0] >= 3:
+        fname = 'win64python2.npy'
+    else:
+        fname = 'python3.npy'
+    path = os.path.join(os.path.dirname(__file__), 'data', fname)
+    data = np.load(path)
+    assert_array_equal(data, np.ones(2))
+
+
 def test_version_2_0():
     f = BytesIO()
     # requires more than 2 byte for header