Repository: numpy/numpy

Issue:
Loading aligned dtype error (Trac #1619)
_Original ticket http://projects.scipy.org/numpy/ticket/1619 on 2010-09-24 by trac user Ihor.Melnyk, assigned to unknown._

```
import numpy as np

t = np.dtype('i1, i4, i1', align=True)
d = np.zeros(1, t)

np.save("test.npy", d)
data = np.load("test.npy")


Traceback (most recent call last):
  File "D:\Projects\Cuda\Cuda_Git\pathwise\liinc\model\feeds\numpy_bug.py", line 8, in <module>
    data = np.load("test.npy")
  File "D:\Projects\Cuda\Cuda_Git\pathwise\pathwise\vendors\lib64\x64\python\numpy\lib\npyio.py", line 314, in load
    return format.read_array(fid)
  File "D:\Projects\Cuda\Cuda_Git\pathwise\pathwise\vendors\lib64\x64\python\numpy\lib\format.py", line 440, in read_array
    shape, fortran_order, dtype = read_array_header_1_0(fp)
  File "D:\Projects\Cuda\Cuda_Git\pathwise\pathwise\vendors\lib64\x64\python\numpy\lib\format.py", line 358, in read_array_header_1_0
    dtype = numpy.dtype(d['descr'])
ValueError: two fields with the same name
```



Discussion:
Attachment added by trac user jpeel on 2010-12-29: [0001-BF-added-fix-for-loading-aligned-dtype.patch](http://projects.scipy.org/numpy/attachment/ticket/1619/0001-BF-added-fix-for-loading-aligned-dtype.patch)

_trac user jpeel wrote on 2010-12-29_

I've come up with a possible solution, but first let me explain the real problem here. 

The real problem here is that dtype() can't handle a list that describes an aligned dtype. For instance, in the example given above, the list that is put into dtype() when the file is loaded is

`[('f0', '|i1'), ('', '|V3'), ('f1', '<i4'), ('f2', '|i1')]`

on my machine. `(_, '|V3')` indicates that this is an aligned dtype. The first element in each tuple is the name of the field. When dtype finds a name equal to `_`, it currently sets the name of that field to `'f#ind'` where #ind is the index of the tuple in the list. That means that `('', '|V3')` in the above list has its name set to `'f1'` which then causes an error when the next item has a name of `'f1'`.

Okay, so the problem then is that dtype() currently doesn't know how to read in an aligned type correctly. Here is how I approached solving this.
1. If the align parameter doesn't equal 1 and the name of the first tuple is not equal to `''`, then set checkalign to 1 so that this input list could indicate an aligned array.
2. If checkalign, then trigger that a tuple deals with alignment when:
   a. a tuple's name is `''`
   b. a tuple's title is NULL
   c. a tuple has only two elements
   d. the datatype of the tuple is VOID
3. When we find a tuple that meets all of the above conditions,
   a. add the size of the VOID tuple to the totalsize
   b. set maxalign to MAX(maxalign, current element size + previous element size)
   c. adjust the name index accordingly
4. Make sure that the alignment is set correctly when checkalign==1.
5. Readjust the size of the nameslist tuple if necessary. (if there were any aligning tuples, then nameslist initialized to be too large).

3.c. refers to an additional index, ii, to keep track of the indices in the nameslist. This is necessary because i refers to the index in the input list.

There might be a better approach than this, but this is one possible solution. Other approaches might involve changing the npy format or how aligned dtypes are shown. However, I think that those will be much harder to implement.

In the patch that I'm attaching, I've kept everything in the _convert_from_array_desc function, but it might be a good idea to break it up a bit or make some other changes to make the code more manageable.

_@rgommers wrote on 2011-03-11_

Charles Harris sent a message to the Numpy mailing list about this ticket on March 3rd, probably good to reply there. For completeness, his comments:

```
In [2]: np.dtype('i1, i4, i1', align=True)
Out[2]: dtype([('f0', '|i1'), ('', '|V3'), ('f1', '<i4'), ('f2', '|i1')])

In [3]: dtype([('f0', '|i1'), ('', '|V3'), ('f1', '<i4'), ('f2', '|i1')])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)

/home/charris/<ipython console> in <module>()

ValueError: two fields with the same name

Note that the second field in the dtype is inserted for alignment purposes and isn't 
named. However, the list in the dtype can not be used to define a dtype because the 
empty name is replaced by 'f1', which conflicts with the following field.  The patch 
attached to the ticket has a rather complicated work around for this that I would 
rather avoid. I would prefer some sort of reserved name for alignment fields, maybe 
something like '*0', '*1', etc. Another question I have is if alignment is supposed to 
be preserved across architectures, that is, should it be possible to pickle/savez on 
one architecture and read things in on another and still have the elements aligned. I 
suspect this isn't worth doing, but one way or the other should be decided.
```

_@mwiebe wrote on 2011-03-16_

I think the bug here is in **repr** not producing a good string to reconstruct the data type. Rather than coming up with an awkward workaround, I would suggest **repr** print the following when the exact type would not be reconstructed with the normal repr:

```
dtype({'names': ['f0', 'f1', 'f2'], 'formats': [dtype('int8'), dtype('int32'), dtype('int8')], 'offsets': [0, 4, 8]})
```

Unfortunately, however, this is still not good enough! It produces a type with itemsize 9 instead of 12. I think it's also necessary to extend the dtype construction syntax to something like:

```
dtype({'names': ['f0', 'f1', 'f2'], 'formats': [dtype('int8'), dtype('int32'), dtype('int8')], 'offsets': [0, 4, 8]}, itemsize=12)
```

Also, to see why the proposed patch doesn't work, consider the following example.

```
>>> t = np.dtype('i4, i1', align=True)
>>> s = eval(repr(t))
>>> t
dtype([('f0', '<i4'), ('f1', '|i1')])
>>> s
dtype([('f0', '<i4'), ('f1', '|i1')])
>>> t.itemsize
8
>>> s.itemsize
5
```

Here there is no clue provided to get the correct itemsize, extending the dtype constructor to have an itemsize parameter is the only reasonable way I can think of to make this work in general.

_@charris wrote on 2011-03-16_

What if the dtype included an alignment number? Then align=1, would be no alignment, align=4 would be on 4 byte boundaries, etc. I think that would provide enough information to recreate the structure.

_@mwiebe wrote on 2011-03-16_

Yes, that would work for the specific case of aligned data types, but it's possible (from C) to create data types with sizes that match neither the tightly packed nor the aligned pattern, and I think it's desirable to support that.

I think at some point a function like 'dtype.restrict_to_fields(fields)' would be useful, which would make a data type exactly matching the dtype, but only having the fields mentioned. This would generate a more general data type, so to support the possibility of such a function it needs to be more general.

_@mwiebe wrote on 2011-03-16_

On the other hand, also including an alignment number would be required for doing reasonable type promotions between structured types. If any of inputs are aligned, it would make sense for the promoted type to be aligned as well, and it may have a different set of fields than any of the inputs.

_@pv wrote on 2011-04-02_

+1 for adding the itemsize parameter. This could be a workaround for #2383 if we decide not to include the trailing padding.

-1 for `align` keyword, as it seems that it is not a very explicit specification of the data structure. I'm also not sure if it's portable if stored in `.npy` files. Native-alignment can in principle checked in UpdateFlags...

_@mwiebe wrote on 2011-06-21_

I believe the align keyword can be well-defined, and C compilers in general have a consistent approach. The reason I really want this in is for in-memory calculations, where operations combining or manipulating aligned structured arrays should produce new aligned structured arrays, not throw away the alignment and force lots of extra copying and buffering.

I'm not sure what the .npy format is doing, I wish it would have defined a NumPy + Python independent specification for the structure. I've fixed the repr issue, but the .npy format still breaks.

_@mwiebe wrote on 2011-06-23_

Repr now produces reliable strings:

```
>>> np.dtype('i1, i4, i1', align=True)
dtype({'names':['f0','f1','f2'], 'formats':['i1','<i4','i1'], 'offsets':[0,4,8], 'itemsize':12}, align=True)
```

but the original issue with the .npy file format is still there.

Still needs a fix for .npy format.


Resolving patch:
diff --git a/numpy/lib/format.py b/numpy/lib/format.py
--- a/numpy/lib/format.py
+++ b/numpy/lib/format.py
@@ -259,6 +259,43 @@ def dtype_to_descr(dtype):
     else:
         return dtype.str
 
+def descr_to_dtype(descr):
+    '''
+    descr may be stored as dtype.descr, which is a list of
+    (name, format, [shape]) tuples. Offsets are not explicitly saved, rather
+    empty fields with name,format == '', '|Vn' are added as padding.
+
+    This function reverses the process, eliminating the empty padding fields.
+    '''
+    if isinstance(descr, (str, dict)):
+        # No padding removal needed
+        return numpy.dtype(descr)
+
+    fields = []
+    offset = 0
+    for field in descr:
+        if len(field) == 2:
+            name, descr_str = field
+            dt = descr_to_dtype(descr_str)
+        else:
+            name, descr_str, shape = field
+            dt = numpy.dtype((descr_to_dtype(descr_str), shape))
+
+        # Ignore padding bytes, which will be void bytes with '' as name
+        # Once support for blank names is removed, only "if name == ''" needed)
+        is_pad = (name == '' and dt.type is numpy.void and dt.names is None)
+        if not is_pad:
+            fields.append((name, dt, offset))
+
+        offset += dt.itemsize
+
+    names, formats, offsets = zip(*fields)
+    # names may be (title, names) tuples
+    nametups = (n  if isinstance(n, tuple) else (None, n) for n in names)
+    titles, names = zip(*nametups)
+    return numpy.dtype({'names': names, 'formats': formats, 'titles': titles,
+                        'offsets': offsets, 'itemsize': offset})
+
 def header_data_from_array_1_0(array):
     """ Get the dictionary of header metadata from a numpy.ndarray.
 
@@ -523,7 +560,7 @@ def _read_array_header(fp, version):
         msg = "fortran_order is not a valid bool: %r"
         raise ValueError(msg % (d['fortran_order'],))
     try:
-        dtype = numpy.dtype(d['descr'])
+        dtype = descr_to_dtype(d['descr'])
     except TypeError as e:
         msg = "descr is not a valid dtype descriptor: %r"
         raise ValueError(msg % (d['descr'],))


Test patch:
diff --git a/numpy/lib/tests/test_format.py b/numpy/lib/tests/test_format.py
--- a/numpy/lib/tests/test_format.py
+++ b/numpy/lib/tests/test_format.py
@@ -524,6 +524,30 @@ def test_compressed_roundtrip():
     assert_array_equal(arr, arr1)
 
 
+# aligned
+dt1 = np.dtype('i1, i4, i1', align=True)
+# non-aligned, explicit offsets
+dt2 = np.dtype({'names': ['a', 'b'], 'formats': ['i4', 'i4'],
+                'offsets': [1, 6]})
+# nested struct-in-struct
+dt3 = np.dtype({'names': ['c', 'd'], 'formats': ['i4', dt2]})
+# field with '' name
+dt4 = np.dtype({'names': ['a', '', 'b'], 'formats': ['i4']*3})
+# titles
+dt5 = np.dtype({'names': ['a', 'b'], 'formats': ['i4', 'i4'],
+                'offsets': [1, 6], 'titles': ['aa', 'bb']})
+
+@pytest.mark.parametrize("dt", [dt1, dt2, dt3, dt4, dt5])
+def test_load_padded_dtype(dt):
+    arr = np.zeros(3, dt)
+    for i in range(3):
+        arr[i] = i + 5
+    npz_file = os.path.join(tempdir, 'aligned.npz')
+    np.savez(npz_file, arr=arr)
+    arr1 = np.load(npz_file)['arr']
+    assert_array_equal(arr, arr1)
+
+
 def test_python2_python3_interoperability():
     if sys.version_info[0] >= 3:
         fname = 'win64python2.npy'
@@ -533,7 +557,6 @@ def test_python2_python3_interoperability():
     data = np.load(path)
     assert_array_equal(data, np.ones(2))
 
-
 def test_pickle_python2_python3():
     # Test that loading object arrays saved on Python 2 works both on
     # Python 2 and Python 3 and vice versa