Repository: numpy/numpy

Issue:
New in NumPy 1.14: failures in PyTables 
In the following snippet:

```python
import numpy as np
data_proper = [(456, b'dbe', 1.2), (2, b'ded', 1.3)]
data_improper = [[456, b'dbe', 1.2], [2, b'ded', 1.3]]
descr = [('f0', '<i4'), ('f1', 'S3'), ('f2', '<f8')]
dtype = np.dtype((np.record, descr))

np.array(data_improper, dtype=dtype)
```

Execution of the last line in 1.13 used to raise:
```
TypeError: a bytes-like object is required, not 'int'
```
which allowed the logic in [numpy/core/records.py#L676](https://github.com/numpy/numpy/blob/master/numpy/core/records.py#L676) to recover and reinterpret the line as `np.array(data_proper, dtype=dtype)`.

The trouble is that in NumPy 1.14
```
ValueError: invalid literal for int() with base 10: b'dbe'
```
is being raised instead, which results in the failure. 

This is a gist of the issue behind numerous test failures in pyTables when used with NumPy 1.14.
  
BUG: np.save() and np.load() are not idempotent when align=True or fields are discontiguous
Given an array like this:

```
arr = np.array([(1, 2), (3, 4)], dtype=np.dtype([('a','u2'), ('b','u4')], align=True))
```

The dtype is:

```
{'names':['a','b'], 'formats':['<u2','<u4'], 'offsets':[0,4], 'itemsize':8, 'aligned':True}
```

Note the difference with the default `align=False` dtype:

```
[('a', '<u2'), ('b', '<u4')]
```

Now, the problem is that `np.save('foo.npy', arr)` saves the aligned dtype this way:

```
[('a', '<u2'), ('', '|V2'), ('b', '<u4')]
```

Then, `np.load('foo.npy')` produces:

```
array([(1, [62, -18], 2), (3, [62, -18], 4)], 
      dtype=[('a', '<u2'), ('f1', 'V2'), ('b', '<u4')])
```

This is despite the fact that `np.load()` can parse the original dtype with `names, formats, offsets` - only `np.save()` doesn't save it that way.  Editing `foo.npy` by hand lets `np.load()` produce an identical array to `arr` (if you try it, the 9th byte must change from 0x66 to 0x96 to allow for the longer `descr`).

As far as I can tell, fixing this would be as simple as storing `str(arr.dtype)` instead of `arr.dtype.descr` in the NPY file.  For aligned arrays the two seem to produce the same string.  But I'm not sure if there's some reason this wasn't done in the first place.

I'm using NumPy 1.11.1.



Discussion:
The error comes from cPython, but where it actually originates is hard to tell, the traceback has been truncated too far. I think we will need a bisect on this.
Perhaps the easy fix is to just to catch `ValueError` in `rec.array` as well?
@eric-wieser I'd worry that that might not be the only place affected. I think we need to know just what changed.
I think the simplest reproduction is:
```python
>>> np.array(456, [('i', int)])
TypeError: a bytes-like object is required, not 'int'  # 1.13
array((456,), dtype=[('i', '<i4')])  # 1.14
```
However, the new behavior seems kinda consistent, when compared to:
```python
>>> np.array(456).astype([('i', int)])
array((456,), dtype=[('i', '<i4')])  # 1.13 and 1.14
```
and
```python
>>> np.array(np.array(456), [('i', int)])
array((456,), dtype=[('i', '<i4')])  # 1.13 and 1.14
```
  
For reference, the 1.13.3 error is `TypeError: expected a readable buffer object`.
I do think the catch in `records.py` is too narrow, too dependent on underlying implementation details, so that needs a fix.
> so that needs a fix.

Frankly, I think we should deprecate that code path entirely. If you pass malformed data, expecting numpy to guess what you meant seems undesirable
The change came in with #6053. I don't think this is so much a bug, as a change in behavior.
I think we need a bigger discussion on the way forward, so pinging @ahaldane . IIRC, h5py was also affected by this change.
I'm reading along. Following on what @eric-wieser said, the change in behavior was to allow scalars to broadcast to all fields during array creation, not just during assignment. The difference in behavior of these two lines seemed inconsistent in 1.13:
```python
>>> a = np.zeros(3, dtype='i4,i4') # no scalar assignment problem
>>> a[:] = 456  # no scalar assignment problem
>>> np.array([456], dtype='i4,i4')
TypeError: expected a readable buffer object  # can't initialize to scalar in 1.13
```

Furthermore, in 1.13:
```python
>>> np.array([456, 'dbe'], dtype='i4,S3')
TypeError: expected a readable buffer object
>>> np.rec.array([456, 'dbe'], dtype='i4,S3')
rec.array((456, 'dbe'), 
          dtype=[('f0', '<i4'), ('f1', 'S3')])
```
Why should the first line fail, but the second succeed?

So my feeling is that the special casing in `records.py` isn't ideal, but maybe there is away to keep things back-compatible, I'm not sure yet.

By the way, I will be at a conference later this week, so may be away from my computer.
  
The problem seems isolated to the `record.py` module, so for 1.14 adding `ValueError` to the list of caught errors is probably an adequate fix. @eric-wieser's point about trying to fix up malformed input is the long term problem, and we probably need more time than 1.14 allows to deal with that.
@charris Thank you. The improper record array object input to `numpy.record.array` was all coming from hard-coded list of lists in pyTable's test files, so it is easy, albeit tedious, to modify. I am prepping a PR for pyTable now.
  
@charris I have opened two PRs for PyTables repo, one addressing errors caused by the current issues, and [the other](https://github.com/PyTables/PyTables/pull/663) dealing with deprecation warnings.
>  so for 1.14 adding ValueError to the list of caught errors is probably an adequate fix

Agreed, although we might want to see if we can squeeze any other new errors out of the conversion too. Maybe even a catchall `Exception` would be worthwhile.

> and we probably need more time than 1.14 allows to deal with that.

Although if all our CI-testing users are already fixing this, this might be an easy time to slip in a deprecation warning too.


OK, made the suggested fix in #10543. Note that the @ahaldane reverted the causal changes for 1.14.1, but getting a FutureWarning out there early is probably a good thing.
Yeah this is an old problem. arrays with aligned dtypes cannot be loaded in general. See my comments in #3176 #6359 #2215.

It is on my list of things to fix after my PR #6053 gets in (hopefully in 1.13).


Resolving patch:
diff --git a/numpy/doc/structured_arrays.py b/numpy/doc/structured_arrays.py
--- a/numpy/doc/structured_arrays.py
+++ b/numpy/doc/structured_arrays.py
@@ -133,10 +133,9 @@
 
      Offsets may be chosen such that the fields overlap, though this will mean
      that assigning to one field may clobber any overlapping field's data. As
-     an exception, fields of :class:`numpy.object` type .. (see 
-     :ref:`object arrays <arrays.object>`) cannot overlap with other fields,
-     because of the risk of clobbering the internal object pointer and then
-     dereferencing it.
+     an exception, fields of :class:`numpy.object` type cannot overlap with
+     other fields, because of the risk of clobbering the internal object
+     pointer and then dereferencing it.
 
      The optional 'aligned' value can be set to ``True`` to make the automatic
      offset computation use aligned offsets (see :ref:`offsets-and-alignment`),
@@ -235,6 +234,11 @@
 alignment conditions, the array will have the ``ALIGNED`` :ref:`flag
 <numpy.ndarray.flags>` set.
 
+A convenience function :func:`numpy.lib.recfunctions.repack_fields` converts an
+aligned dtype or array to a packed one and vice versa. It takes either a dtype
+or structured ndarray as an argument, and returns a copy with fields re-packed,
+with or without padding bytes.
+
 .. _titles:
 
 Field Titles
@@ -396,27 +400,61 @@
 Accessing Multiple Fields
 ```````````````````````````
 
-One can index a structured array with a multi-field index, where the index is a
-list of field names::
+One can index and assign to a structured array with a multi-field index, where
+the index is a list of field names.
+
+.. warning::
+    The behavior of multi-field indexes will change from Numpy 1.15 to Numpy
+    1.16.
 
- >>> a = np.zeros(3, dtype=[('a', 'i8'), ('b', 'i4'), ('c', 'f8')])
+In Numpy 1.16, the result of indexing with a multi-field index will be a view
+into the original array, as follows::
+
+ >>> a = np.zeros(3, dtype=[('a', 'i4'), ('b', 'i4'), ('c', 'f4')])
  >>> a[['a', 'c']]
- array([(0, 0.0), (0, 0.0), (0, 0.0)],
-       dtype={'names':['a','c'], 'formats':['<i8','<f8'], 'offsets':[0,11], 'itemsize':19})
+ array([(0, 0.), (0, 0.), (0, 0.)],
+      dtype={'names':['a','c'], 'formats':['<i4','<f4'], 'offsets':[0,8], 'itemsize':12})
+
+Assignment to the view modifies the original array. The view's fields will be
+in the order they were indexed. Note that unlike for single-field indexing, the
+view's dtype has the same itemsize as the original array, and has fields at the
+same offsets as in the original array, and unindexed fields are merely missing.
+
+In Numpy 1.15, indexing an array with a multi-field index returns a copy of
+the result above for 1.16, but with fields packed together in memory as if
+passed through :func:`numpy.lib.recfunctions.repack_fields`. This is the
+behavior since Numpy 1.7.
+
+.. warning::
+   The new behavior in Numpy 1.16 leads to extra "padding" bytes at the
+   location of unindexed fields. You will need to update any code which depends
+   on the data having a "packed" layout. For instance code such as::
+
+    >>> a[['a','c']].view('i8')  # will fail in Numpy 1.16
+    ValueError: When changing to a smaller dtype, its size must be a divisor of the size of original dtype
+
+   will need to be changed. This code has raised a ``FutureWarning`` since
+   Numpy 1.12.
+
+   The following is a recommended fix, which will behave identically in Numpy
+   1.15 and Numpy 1.16::
+
+    >>> from numpy.lib.recfunctions import repack_fields
+    >>> repack_fields(a[['a','c']]).view('i8')  # supported 1.15 and 1.16
+    array([0, 0, 0])
+
+Assigning to an array with a multi-field index will behave the same in Numpy
+1.15 and Numpy 1.16. In both versions the assignment will modify the original
+array::
+
  >>> a[['a', 'c']] = (2, 3)
  >>> a
  array([(2, 0, 3.0), (2, 0, 3.0), (2, 0, 3.0)],
        dtype=[('a', '<i8'), ('b', '<i4'), ('c', '<f8')])
 
-The resulting array is a view into the original array, such that assignment to
-the view modifies the original array. The view's fields will be in the order
-they were indexed. Note that unlike for single-field indexing, the view's dtype
-has the same itemsize as the original array, and has fields at the same offsets
-as in the original array, and unindexed fields are merely missing.
-
-Since the view is a structured array itself, it obeys the assignment rules
-described above. For example, this means that one can swap the values of two
-fields using appropriate multi-field indexes::
+This obeys the structured array assignment rules described above. For example,
+this means that one can swap the values of two fields using appropriate
+multi-field indexes::
 
  >>> a[['a', 'c']] = a[['c', 'a']]
 
diff --git a/numpy/lib/recfunctions.py b/numpy/lib/recfunctions.py
--- a/numpy/lib/recfunctions.py
+++ b/numpy/lib/recfunctions.py
@@ -732,6 +732,84 @@ def rec_append_fields(base, names, data, dtypes=None):
     return append_fields(base, names, data=data, dtypes=dtypes,
                          asrecarray=True, usemask=False)
 
+def repack_fields(a, align=False, recurse=False):
+    """
+    Re-pack the fields of a structured array or dtype in memory.
+
+    The memory layout of structured datatypes allows fields at arbitrary
+    byte offsets. This means the fields can be separated by padding bytes,
+    their offsets can be non-monotonically increasing, and they can overlap.
+
+    This method removes any overlaps and reorders the fields in memory so they
+    have increasing byte offsets, and adds or removes padding bytes depending
+    on the `align` option, which behaves like the `align` option to `np.dtype`.
+
+    If `align=False`, this method produces a "packed" memory layout in which
+    each field starts at the byte the previous field ended, and any padding
+    bytes are removed.
+
+    If `align=True`, this methods produces an "aligned" memory layout in which
+    each field's offset is a multiple of its alignment, and the total itemsize
+    is a multiple of the largest alignment, by adding padding bytes as needed.
+
+    Parameters
+    ----------
+    a : ndarray or dtype
+       array or dtype for which to repack the fields.
+    align : boolean
+       If true, use an "aligned" memory layout, otherwise use a "packed" layout.
+    recurse : boolean
+       If True, also repack nested structures.
+
+    Returns
+    -------
+    repacked : ndarray or dtype
+       Copy of `a` with fields repacked, or `a` itself if no repacking was
+       needed.
+
+    Examples
+    --------
+
+    >>> def print_offsets(d):
+    ...     print("offsets:", [d.fields[name][1] for name in d.names])
+    ...     print("itemsize:", d.itemsize)
+    ...
+    >>> dt = np.dtype('u1,i4,f4', align=True)
+    >>> dt
+    dtype({'names':['f0','f1','f2'], 'formats':['u1','<i4','<f8'], 'offsets':[0,4,8], 'itemsize':16}, align=True)
+    >>> print_offsets(dt)
+    offsets: [0, 4, 8]
+    itemsize: 16
+    >>> packed_dt = repack_fields(dt)
+    >>> packed_dt
+    dtype([('f0', 'u1'), ('f1', '<i4'), ('f2', '<f8')])
+    >>> print_offsets(packed_dt)
+    offsets: [0, 1, 5]
+    itemsize: 13
+
+    """
+    if not isinstance(a, np.dtype):
+        dt = repack_fields(a.dtype, align=align, recurse=recurse)
+        return a.astype(dt, copy=False)
+
+    if a.names is None:
+        return a
+
+    fieldinfo = []
+    for name in a.names:
+        tup = a.fields[name]
+        if recurse:
+            fmt = repack_fields(tup[0], align=align, recurse=True)
+        else:
+            fmt = tup[0]
+
+        if len(tup) == 3:
+            name = (tup[2], name)
+
+        fieldinfo.append((name, fmt))
+
+    dt = np.dtype(fieldinfo, align=align)
+    return np.dtype((a.type, dt))
 
 def stack_arrays(arrays, defaults=None, usemask=True, asrecarray=False,
                  autoconvert=False):


Test patch:
diff --git a/numpy/core/tests/test_multiarray.py b/numpy/core/tests/test_multiarray.py
--- a/numpy/core/tests/test_multiarray.py
+++ b/numpy/core/tests/test_multiarray.py
@@ -4796,9 +4796,25 @@ def test_field_names(self):
             fn2 = func('f2')
             b[fn2] = 3
 
-            assert_equal(b[['f1', 'f2']][0].tolist(), (2, 3))
-            assert_equal(b[['f2', 'f1']][0].tolist(), (3, 2))
-            assert_equal(b[['f1', 'f3']][0].tolist(), (2, (1,)))
+            # In 1.16 code below can be replaced by:
+            # assert_equal(b[['f1', 'f2']][0].tolist(), (2, 3))
+            # assert_equal(b[['f2', 'f1']][0].tolist(), (3, 2))
+            # assert_equal(b[['f1', 'f3']][0].tolist(), (2, (1,)))
+            with suppress_warnings() as sup:
+                sup.filter(FutureWarning,
+                           ".* selecting multiple fields .*")
+
+                assert_equal(b[['f1', 'f2']][0].tolist(), (2, 3))
+                assert_equal(b[['f2', 'f1']][0].tolist(), (3, 2))
+                assert_equal(b[['f1', 'f3']][0].tolist(), (2, (1,)))
+                # view of subfield view/copy
+                assert_equal(b[['f1', 'f2']][0].view(('i4', 2)).tolist(),
+                             (2, 3))
+                assert_equal(b[['f2', 'f1']][0].view(('i4', 2)).tolist(),
+                             (3, 2))
+                view_dtype = [('f1', 'i4'), ('f3', [('', 'i4')])]
+                assert_equal(b[['f1', 'f3']][0].view(view_dtype).tolist(),
+                             (2, (1,)))
 
         # non-ascii unicode field indexing is well behaved
         if not is_py3:
@@ -4808,6 +4824,51 @@ def test_field_names(self):
             assert_raises(ValueError, a.__setitem__, u'\u03e0', 1)
             assert_raises(ValueError, a.__getitem__, u'\u03e0')
 
+    # can be removed in 1.16
+    def test_field_names_deprecation(self):
+
+        def collect_warnings(f, *args, **kwargs):
+            with warnings.catch_warnings(record=True) as log:
+                warnings.simplefilter("always")
+                f(*args, **kwargs)
+            return [w.category for w in log]
+
+        a = np.zeros((1,), dtype=[('f1', 'i4'),
+                                  ('f2', 'i4'),
+                                  ('f3', [('sf1', 'i4')])])
+        a['f1'][0] = 1
+        a['f2'][0] = 2
+        a['f3'][0] = (3,)
+        b = np.zeros((1,), dtype=[('f1', 'i4'),
+                                  ('f2', 'i4'),
+                                  ('f3', [('sf1', 'i4')])])
+        b['f1'][0] = 1
+        b['f2'][0] = 2
+        b['f3'][0] = (3,)
+
+        # All the different functions raise a warning, but not an error
+        assert_equal(collect_warnings(a[['f1', 'f2']].__setitem__, 0, (10, 20)),
+                     [FutureWarning])
+        # For <=1.12 a is not modified, but it will be in 1.13
+        assert_equal(a, b)
+
+        # Views also warn
+        subset = a[['f1', 'f2']]
+        subset_view = subset.view()
+        assert_equal(collect_warnings(subset_view['f1'].__setitem__, 0, 10),
+                     [FutureWarning])
+        # But the write goes through:
+        assert_equal(subset['f1'][0], 10)
+        # Only one warning per multiple field indexing, though (even if there
+        # are multiple views involved):
+        assert_equal(collect_warnings(subset['f1'].__setitem__, 0, 10), [])
+
+        # make sure views of a multi-field index warn too
+        c = np.zeros(3, dtype='i8,i8,i8')
+        assert_equal(collect_warnings(c[['f0', 'f2']].view, 'i8,i8'),
+                     [FutureWarning])
+
+
     def test_record_hash(self):
         a = np.array([(1, 2), (1, 2)], dtype='i1,i2')
         a.flags.writeable = False
diff --git a/numpy/core/tests/test_records.py b/numpy/core/tests/test_records.py
--- a/numpy/core/tests/test_records.py
+++ b/numpy/core/tests/test_records.py
@@ -11,6 +11,7 @@
 import warnings
 import textwrap
 from os import path
+import pytest
 
 import numpy as np
 from numpy.testing import (
@@ -360,6 +361,7 @@ def test_nonwriteable_setfield(self):
         with assert_raises(ValueError):
             r.setfield([2,3], *r.dtype.fields['f'])
 
+    @pytest.mark.xfail(reason="See gh-10411, becomes real error in 1.16")
     def test_out_of_order_fields(self):
         # names in the same order, padding added to descr
         x = self.data[['col1', 'col2']]
diff --git a/numpy/lib/tests/test_recfunctions.py b/numpy/lib/tests/test_recfunctions.py
--- a/numpy/lib/tests/test_recfunctions.py
+++ b/numpy/lib/tests/test_recfunctions.py
@@ -9,8 +9,8 @@
 from numpy.testing import assert_, assert_raises
 from numpy.lib.recfunctions import (
     drop_fields, rename_fields, get_fieldstructure, recursive_fill_fields,
-    find_duplicates, merge_arrays, append_fields, stack_arrays, join_by
-    )
+    find_duplicates, merge_arrays, append_fields, stack_arrays, join_by,
+    repack_fields)
 get_names = np.lib.recfunctions.get_names
 get_names_flat = np.lib.recfunctions.get_names_flat
 zip_descr = np.lib.recfunctions.zip_descr
@@ -192,6 +192,18 @@ def test_find_duplicates_ignoremask(self):
         assert_equal(sorted(test[-1]), control)
         assert_equal(test[0], a[test[-1]])
 
+    def test_repack_fields(self):
+        dt = np.dtype('u1,f4,i8', align=True)
+        a = np.zeros(2, dtype=dt)
+
+        assert_equal(repack_fields(dt), np.dtype('u1,f4,i8'))
+        assert_equal(repack_fields(a).itemsize, 13)
+        assert_equal(repack_fields(repack_fields(dt), align=True), dt)
+
+        # make sure type is preserved
+        dt = np.dtype((np.record, dt))
+        assert_(repack_fields(dt).type is np.record)
+
 
 class TestRecursiveFillFields(object):
     # Test recursive_fill_fields.