Issue 1:
Describe the issue:

I have a .npy file generated from an h5py dataset. The data is a pure Numpy array (no subclass) and the file has been written by Numpy, but the dtype contains metadata set by h5py.

In this particular case, the metadata contains a <class 'bytes'> expression, which cannot be handled by safe_eval . I don't care about the metadata, but it renders the entire file unreadable.

File "/home/sjoerd/miniconda3/envs/active-papers/lib/python3.9/site-packages/numpy/lib/format.py", line 776, in read_array
shape, fortran_order, dtype = _read_array_header(
File "/home/sjoerd/miniconda3/envs/active-papers/lib/python3.9/site-packages/numpy/lib/format.py", line 632, in _read_array_header
raise ValueError(msg.format(header)) from e
ValueError: Cannot parse header: "{'descr': [('paper_ref', ('|O', {'vlen': <class 'bytes'>})), ('path', ('|O', {'vlen': <class 'bytes'>}))], 'fortran_order': False, 'shape': (), } \n"

I would consider this a bug in np.save . Serialization of a dtype descriptor that results in an unreadable header should fail loudly. It would be nice if the np.save could then be re-tried with the metadata stripped.

Reproduce the code example:

import numpy as np
np.load("file.npy")

Error message:

Traceback (most recent call last):
File "/home/sjoerd/bayesian_inference_fm/aptool-dump.py", line 65, in dump
arr = np.load(child_target + ".npy")
File "/home/sjoerd/miniconda3/envs/active-papers/lib/python3.9/site-packages/numpy/lib/npyio.py", line 432, in load
return format.read_array(fid, allow_pickle=allow_pickle,
File "/home/sjoerd/miniconda3/envs/active-papers/lib/python3.9/site-packages/numpy/lib/format.py", line 776, in read_array
shape, fortran_order, dtype = _read_array_header(
File "/home/sjoerd/miniconda3/envs/active-papers/lib/python3.9/site-packages/numpy/lib/format.py", line 632, in _read_array_header
raise ValueError(msg.format(header)) from e
ValueError: Cannot parse header: "{'descr': [('paper_ref', ('|O', {'vlen': <class 'bytes'>})), ('path', ('|O', {'vlen': <class 'bytes'>}))], 'fortran_order': False, 'shape': (), } \n"

Runtime information:

Numpy version:

1.24.0

3.9.16 | packaged by conda-forge | (main, Feb 1 2023, 21:39:03)

[GCC 11.3.0]

show_runtime:

WARNING: threadpoolctl not found in system! Install it by pip install threadpoolctl. Once installed, try np.show_runtime again for more detailed build information

[{'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3'],

'found': ['SSSE3',

'SSE41',

'POPCNT',

'SSE42',

'AVX',

'F16C',

'FMA3',

'AVX2'],

'not_found': ['AVX512F',

'AVX512CD',

'AVX512_KNL',

'AVX512_KNM',

'AVX512_SKX',

'AVX512_CLX',

'AVX512_CNL',

'AVX512_ICL']}}]

Context for the issue:

A pure Numpy array (no subclasses) saved with np.save should be readable by np.load. This issue gives a counter-example.

I could try to write a patch to fix this issue, if you are interested.

Issue 2:
The dtype description in an NPY file can be written in a way that prevents the file from being read. The attached array in bad_array.pkl.zip exhibits this problem. In this case the written dtype is [('id', '|V16'), ('has_fit', '|b1'), ('timeax', ('|O', {'vlen': dtype('float64')})), ('', '|V8'), ('profile', ('|O', {'vlen': dtype('float64')})), ('', '|V8'), ('area', '<f8')].

(Note that it was obtained from via h5py - if this is an issue with the format of the arrays produced by that library, I'm happy to file a bug there instead.)

Reproducing code example:

import numpy as np
import pickle
import io

with open('bad_array.pkl', 'rb') as f:
arr = pickle.load(f)

buf = io.BytesIO()
np.save(buf, arr)
buf.seek(0)
np.load(buf)

Error message:

Traceback (most recent call last):
File "scripts/badnpy.py", line 11, in <module>
np.load(buf)
File "/Users/colingavin/Code/908/clarent2/algo/py/.venv/lib/python3.7/site-packages/numpy/lib/npyio.py", line 453, in load
pickle_kwargs=pickle_kwargs)
File "/Users/colingavin/Code/908/clarent2/algo/py/.venv/lib/python3.7/site-packages/numpy/lib/format.py", line 712, in read_array
shape, fortran_order, dtype = _read_array_header(fp, version)
File "/Users/colingavin/Code/908/clarent2/algo/py/.venv/lib/python3.7/site-packages/numpy/lib/format.py", line 578, in _read_array_header
d = safe_eval(header)
File "/Users/colingavin/Code/908/clarent2/algo/py/.venv/lib/python3.7/site-packages/numpy/lib/utils.py", line 1139, in safe_eval
return ast.literal_eval(source)
File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 91, in literal_eval
return _convert(node_or_string)
File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 79, in _convert
map(_convert, node.values)))
File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 74, in _convert
return list(map(_convert, node.elts))
File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 72, in _convert
return tuple(map(_convert, node.elts))
File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 72, in _convert
return tuple(map(_convert, node.elts))
File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 79, in _convert
map(_convert, node.values)))
File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 90, in _convert
return _convert_signed_num(node)
File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 63, in _convert_signed_num
return _convert_num(node)
File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 55, in _convert_num
raise ValueError('malformed node or string: ' + repr(node))
ValueError: malformed node or string: <_ast.Call object at 0x10b965a58>

Numpy/Python version information:

1.17.0 3.7.2 (default, Jul 14 2019, 10:22:38)

[Clang 9.0.0 (clang-900.0.39.2)]