Repository: numpy/numpy

Issue:
`.view` no longer works on field selections of structured arrays
See [this SO question](https://stackoverflow.com/questions/48267058/using-np-view-with-changes-to-structured-arrays-in-numpy-1-14):
```python
>>> some_data = np.array([('foo', 3.5, 2.15), ('bar', 2.8, 5.3), ('baz', 1.2, 3.7)], 
                     dtype=[('col1', '<U20'), ('A', '<f8'), ('B', '<f8')])
>>> ab = some_data[['A', 'B']]
```
```python
>>> np.__version__
'1.13.3'
>>> ab.dtype
dtype([('A', '<f8'), ('B', '<f8')])
>>> ab.view(('<f8', 2))
FutureWarning: Numpy has detected that you may be viewing or writing to an array returned by selecting multiple fields in a structured array. 

This code may break in numpy 1.13 because this will return a view instead of a copy -- see release notes for details.
array([[ 3.5 ,  2.15],
       [ 2.8 ,  5.3 ],
       [ 1.2 ,  3.7 ]])
```

vs

```python
>>> np.__version__
'1.15.0.dev...'
>>> ab.dtype
dtype({'names':['A','B'], 'formats':['<f8','<f8'], 'offsets':[80,88], 'itemsize':96})
>>> ab.view(('<f8', 2))
ValueError: Changing the dtype to a subarray type is only supported if the total itemsize is unchanged
```
Since we now return a view rather than a copy from `arr[fields]`, we end up with a dtype with padding. This prevents `.view` being called, even though the memory layout is sufficient to allow it
MAINT: cancel multifield copy->view and copy-by-position plans (alternate proposal)
This is a branch where I essentially reverted the major changes of #6053, since it causes some back-compat problems. Putting it here so we can compare, before deciding which way to go.  See discussion in #11526. This revers the "multifield indexing returns a view" change, and the "multifield assignment occurs by-position instead of by-name" change.

#6053 wouldn't all be a loss though. This PR still keeps all the little fixes from that PR, and I also make multifield assignment consistent: Before #6053 you would get different behavior depending on whether you did array vs scalar assignment. In #6053 I chose to make all of numpy use "copy-by-position" for multifield assignment, in this PR everything is "copy-by-name". That means there is still a back-compat break in this PR for structured scalar assignment, but probably it has much less effect than #6053.

So if we go this route, at least numpy will be more self-consistent, while still being fairly back-compatible.

Also, at least in #6053 I have worked out how I think multifield indexing behavior should work, without all the quirks of current numpy. The day we implement the new standalone structured dtypes I suggest we use the behavior I worked out in #6053, if we decide to remove it here.

A lot of the code change here is just pasting back in things I removed in #6053.
BUG: np.save() and np.load() are not idempotent when align=True or fields are discontiguous
Given an array like this:

```
arr = np.array([(1, 2), (3, 4)], dtype=np.dtype([('a','u2'), ('b','u4')], align=True))
```

The dtype is:

```
{'names':['a','b'], 'formats':['<u2','<u4'], 'offsets':[0,4], 'itemsize':8, 'aligned':True}
```

Note the difference with the default `align=False` dtype:

```
[('a', '<u2'), ('b', '<u4')]
```

Now, the problem is that `np.save('foo.npy', arr)` saves the aligned dtype this way:

```
[('a', '<u2'), ('', '|V2'), ('b', '<u4')]
```

Then, `np.load('foo.npy')` produces:

```
array([(1, [62, -18], 2), (3, [62, -18], 4)], 
      dtype=[('a', '<u2'), ('f1', 'V2'), ('b', '<u4')])
```

This is despite the fact that `np.load()` can parse the original dtype with `names, formats, offsets` - only `np.save()` doesn't save it that way.  Editing `foo.npy` by hand lets `np.load()` produce an identical array to `arr` (if you try it, the 9th byte must change from 0x66 to 0x96 to allow for the longer `descr`).

As far as I can tell, fixing this would be as simple as storing `str(arr.dtype)` instead of `arr.dtype.descr` in the NPY file.  For aligned arrays the two seem to produce the same string.  But I'm not sure if there's some reason this wasn't done in the first place.

I'm using NumPy 1.11.1.



Discussion:
Yeah, I explicitly mentioned this in the 1.13 and 1.14 release notes. Actually, since we have not provided a `pack_fields` method it is difficult for downstream users to update their code to get the old behavior: I originally thought the user could just replace `arr[fields']` by `arr[fields].copy()`, but that does not remove padding bytes either.
Should slicing a dtype shrink the itemsize?

ie, if a dtype has fields `a, b, c, d`, and you slice with `['b', 'd']`, should the resulting view dtype still span the entire original, or just the fields selected? Doing the latter would solve the problem for contiguous views within dtypes (which are probably most common)

A consequence of this would be that the resulting array would be non-"contiguous"
You mean like `arr[['b', 'c']].view('i8')` if all the fields are originally `i4`. True, that could potentially be allowed.

On the other hand sometimes it's better to fail more rather than less often. If people get errors about viewing multifield slices more easily, maybe that's better than hiding it in special cases. Then people are forced to write code that is safe in all situations, for example using `repack_fields`.
Even the less objectionable case where the dtypes match

```
dt = np.dtype([
    ('a', np.int16),
    ('b', np.int16),
    ('c', np.int16),
])
arr = np.empty(10, dt)

# both of these no longer work, but could have well-defined behavior
arr[['a']].view(np.int16)
arr[['a', 'b']].view((np.int16, 2))
```

Essentially, I'm proposing a `trim_fields` behavior, where one `offset` is always zero, and the `itemsize` is never beyond the element occupying the last byte
Right, but it still bothers me that `arr[['a', 'c']].view((np.int16, 2))` would still fail, which would be unexpected so someone who found that `[['a', 'b']]` worked. It might be better to force the user to write `repack_fields(arr[['a', 'c']]).view((np.int16, 2))` in all 3 cases.

Also, I'm pretty sure that even with this `trim_fields` behavior the view often isn't allowed because the resulting array is not C-contiguous. Compare to:
```python
>>> a = np.ones((10,3), dtype='i4')
>>> a[:,0:2].view('i8')
ValueError: To change to a dtype of a different size, the array must be C-contiguous
>>> a[:,0:2].view(('i4', 2))
ValueError: Changing the dtype to a subarray type is only supported if the total itemsize is unchanged
```
@ahaldane Is this now fixed for 1.15?
Yes, I just checked master. Master now gives the 1.13 behavior, just with an updated futurewarning.

I'm not sure we should close, since this issue will pop up again in 1.16.
Good, I'll just update the milestone :).
I ran across code in `mapping.c` that is marked for removal in 1.16, but could not find a pull request that is waiting for a merge. It seems there is work to be done here?
Yes, that's on my TODO list. See discussion in #10411, essentially we need to fix #8100 first.

I should have some time to work on it starting in two weeks, but feel free to take a shot at it before. There is already some work on it in #10931 but as I comment there a more extensive fix is needed.
We have returned to the 1.13 behavior now. PR #10411 was merged, issue #8100 is closed. Can we close this or do we want to explore returning views again?
Hmm in my opinion #8100 was mistakenly closed, since I accidentally used the word "fix" in a sentence in #10411's description.

I am also not sure we've made a decision about structured views. As far as I am aware (I was out for a bit so may have missed it) I think we are still considering turning it into a view. #10411 just pushed off the decision/implementation.
Unused variable warning is causing problems.
Yeah this is an old problem. arrays with aligned dtypes cannot be loaded in general. See my comments in #3176 #6359 #2215.

It is on my list of things to fix after my PR #6053 gets in (hopefully in 1.13).

This was mistakenly closed by #10411, reopening.

I have more extensive comments on what needs to be fixed in #10931 and #7797.
@mattip @ahaldane  What is the current status of the fixes for this issue?
It's fixed!

(by #12358). I'll go ahead and close.

Resolving patch:
diff --git a/numpy/doc/structured_arrays.py b/numpy/doc/structured_arrays.py
--- a/numpy/doc/structured_arrays.py
+++ b/numpy/doc/structured_arrays.py
@@ -35,26 +35,24 @@
  array([('Rex', 5, 81.0), ('Fido', 5, 27.0)],
        dtype=[('name', 'S10'), ('age', '<i4'), ('weight', '<f4')])
 
-Structured arrays are designed for low-level manipulation of structured data,
-for example, for interpreting binary blobs. Structured datatypes are
-designed to mimic 'structs' in the C language, making them also useful for
-interfacing with C code. For these purposes, numpy supports specialized
-features such as subarrays and nested datatypes, and allows manual control over
-the memory layout of the structure.
-
-For simple manipulation of tabular data other pydata projects, such as pandas,
-xarray, or DataArray, provide higher-level interfaces that may be more
-suitable. These projects may also give better performance for tabular data
-analysis because the C-struct-like memory layout of structured arrays can lead
-to poor cache behavior.
+Structured datatypes are designed to be able to mimic 'structs' in the C
+language, and share a similar memory layout. They are meant for interfacing with
+C code and for low-level manipulation of structured buffers, for example for
+interpreting binary blobs. For these purposes they support specialized features
+such as subarrays, nested datatypes, and unions, and allow control over the
+memory layout of the structure.
+
+Users looking to manipulate tabular data, such as stored in csv files, may find
+other pydata projects more suitable, such as xarray, pandas, or DataArray.
+These provide a high-level interface for tabular data analysis and are better
+optimized for that use. For instance, the C-struct-like memory layout of
+structured arrays in numpy can lead to poor cache behavior in comparison.
 
 .. _defining-structured-types:
 
 Structured Datatypes
 ====================
 
-To use structured arrays one first needs to define a structured datatype.
-
 A structured datatype can be thought of as a sequence of bytes of a certain
 length (the structure's :term:`itemsize`) which is interpreted as a collection
 of fields. Each field has a name, a datatype, and a byte offset within the
@@ -180,7 +178,9 @@
  mappingproxy({'x': (dtype('int64'), 0), 'y': (dtype('float32'), 8)})
 
 Both the ``names`` and ``fields`` attributes will equal ``None`` for
-unstructured arrays.
+unstructured arrays. The recommended way to test if a dtype is structured is
+with `if dt.names is not None` rather than `if dt.names`, to account for dtypes
+with 0 fields.
 
 The string representation of a structured datatype is shown in the "list of
 tuples" form if possible, otherwise numpy falls back to using the more general
@@ -404,11 +404,10 @@
 the index is a list of field names.
 
 .. warning::
-    The behavior of multi-field indexes will change from Numpy 1.15 to Numpy
-    1.16.
+    The behavior of multi-field indexes changed from Numpy 1.15 to Numpy 1.16.
 
-In Numpy 1.16, the result of indexing with a multi-field index will be a view
-into the original array, as follows::
+The result of indexing with a multi-field index is a view into the original
+array, as follows::
 
  >>> a = np.zeros(3, dtype=[('a', 'i4'), ('b', 'i4'), ('c', 'f4')])
  >>> a[['a', 'c']]
@@ -420,41 +419,58 @@
 view's dtype has the same itemsize as the original array, and has fields at the
 same offsets as in the original array, and unindexed fields are merely missing.
 
-In Numpy 1.15, indexing an array with a multi-field index returns a copy of
-the result above for 1.16, but with fields packed together in memory as if
-passed through :func:`numpy.lib.recfunctions.repack_fields`. This is the
-behavior since Numpy 1.7.
-
 .. warning::
-   The new behavior in Numpy 1.16 leads to extra "padding" bytes at the
-   location of unindexed fields. You will need to update any code which depends
-   on the data having a "packed" layout. For instance code such as::
+    In Numpy 1.15, indexing an array with a multi-field index returned a copy of
+    the result above, but with fields packed together in memory as if
+    passed through :func:`numpy.lib.recfunctions.repack_fields`.
+
+    The new behavior as of Numpy 1.16 leads to extra "padding" bytes at the
+    location of unindexed fields compared to 1.15. You will need to update any
+    code which depends on the data having a "packed" layout. For instance code
+    such as::
+
+     >>> a = np.zeros(3, dtype=[('a', 'i4'), ('b', 'i4'), ('c', 'f4')])
+     >>> a[['a','c']].view('i8')  # Fails in Numpy 1.16
+     ValueError: When changing to a smaller dtype, its size must be a divisor of the size of original dtype
+
+    will need to be changed. This code has raised a ``FutureWarning`` since
+    Numpy 1.12, and similar code has raised ``FutureWarning`` since 1.7.
+
+    In 1.16 a number of functions have been introduced in the
+    :module:`numpy.lib.recfunctions` module to help users account for this
+    change. These are
+    :func:`numpy.lib.recfunctions.repack_fields`.
+    :func:`numpy.lib.recfunctions.structured_to_unstructured`,
+    :func:`numpy.lib.recfunctions.unstructured_to_structured`,
+    :func:`numpy.lib.recfunctions.apply_along_fields`,
+    :func:`numpy.lib.recfunctions.assign_fields_by_name`,  and
+    :func:`numpy.lib.recfunctions.require_fields`.
 
-    >>> a[['a','c']].view('i8')  # will fail in Numpy 1.16
-    ValueError: When changing to a smaller dtype, its size must be a divisor of the size of original dtype
+    The function :func:`numpy.lib.recfunctions.repack_fields` can always be
+    used to reproduce the old behavior, as it will return a packed copy of the
+    structured array. The code above, for example, can be replaced with:
 
-   will need to be changed. This code has raised a ``FutureWarning`` since
-   Numpy 1.12.
+     >>> repack_fields(a[['a','c']]).view('i8')  # supported in 1.16
+     array([0, 0, 0])
 
-   The following is a recommended fix, which will behave identically in Numpy
-   1.15 and Numpy 1.16::
+    Furthermore, numpy now provides a new function
+    :func:`numpy.lib.recfunctions.structured_to_unstructured` which is a safer
+    and more efficient alternative for users who wish to convert structured
+    arrays to unstructured arrays, as the view above is often indeded to do.
+    This function allows safe conversion to an unstructured type taking into
+    account padding, often avoids a copy, and also casts the datatypes
+    as needed, unlike the view. Code such as:
 
-    >>> from numpy.lib.recfunctions import repack_fields
-    >>> repack_fields(a[['a','c']]).view('i8')  # supported 1.15 and 1.16
-    array([0, 0, 0])
+     >>> a = np.zeros(3, dtype=[('x', 'f4'), ('y', 'f4'), ('z', 'f4')])
+     >>> a[['x', 'z']].view('f4')
 
-    The :module:`numpy.lib.recfunctions` module has other new methods
-    introduced in numpy 1.16 to help users account for this change. These are
-    :func:`numpy.lib.recfunctions.structured_to_unstructured`,
-    :func:`numpy.lib.recfunctions.unstructured_to_structured`, 
-    :func:`numpy.lib.recfunctions.apply_along_fields`, 
-    :func:`numpy.lib.recfunctions.assign_fields_by_name`,  and
-    :func:`numpy.lib.recfunctions.require_fields`.
+    can be made safer by replacing with:
+
+     >>> structured_to_unstructured(a[['x', 'z']])
+     array([0, 0, 0])
 
 
-Assigning to an array with a multi-field index will behave the same in Numpy
-1.15 and Numpy 1.16. In both versions the assignment will modify the original
-array::
+Assignment to an array with a multi-field index modifies the original array::
 
  >>> a[['a', 'c']] = (2, 3)
  >>> a


Test patch:
diff --git a/numpy/core/tests/test_multiarray.py b/numpy/core/tests/test_multiarray.py
--- a/numpy/core/tests/test_multiarray.py
+++ b/numpy/core/tests/test_multiarray.py
@@ -4979,25 +4979,9 @@ def test_field_names(self):
             fn2 = func('f2')
             b[fn2] = 3
 
-            # In 1.16 code below can be replaced by:
-            # assert_equal(b[['f1', 'f2']][0].tolist(), (2, 3))
-            # assert_equal(b[['f2', 'f1']][0].tolist(), (3, 2))
-            # assert_equal(b[['f1', 'f3']][0].tolist(), (2, (1,)))
-            with suppress_warnings() as sup:
-                sup.filter(FutureWarning,
-                           ".* selecting multiple fields .*")
-
-                assert_equal(b[['f1', 'f2']][0].tolist(), (2, 3))
-                assert_equal(b[['f2', 'f1']][0].tolist(), (3, 2))
-                assert_equal(b[['f1', 'f3']][0].tolist(), (2, (1,)))
-                # view of subfield view/copy
-                assert_equal(b[['f1', 'f2']][0].view(('i4', 2)).tolist(),
-                             (2, 3))
-                assert_equal(b[['f2', 'f1']][0].view(('i4', 2)).tolist(),
-                             (3, 2))
-                view_dtype = [('f1', 'i4'), ('f3', [('', 'i4')])]
-                assert_equal(b[['f1', 'f3']][0].view(view_dtype).tolist(),
-                             (2, (1,)))
+            assert_equal(b[['f1', 'f2']][0].tolist(), (2, 3))
+            assert_equal(b[['f2', 'f1']][0].tolist(), (3, 2))
+            assert_equal(b[['f1', 'f3']][0].tolist(), (2, (1,)))
 
         # non-ascii unicode field indexing is well behaved
         if not is_py3:
@@ -5007,50 +4991,6 @@ def test_field_names(self):
             assert_raises(ValueError, a.__setitem__, u'\u03e0', 1)
             assert_raises(ValueError, a.__getitem__, u'\u03e0')
 
-    # can be removed in 1.16
-    def test_field_names_deprecation(self):
-
-        def collect_warnings(f, *args, **kwargs):
-            with warnings.catch_warnings(record=True) as log:
-                warnings.simplefilter("always")
-                f(*args, **kwargs)
-            return [w.category for w in log]
-
-        a = np.zeros((1,), dtype=[('f1', 'i4'),
-                                  ('f2', 'i4'),
-                                  ('f3', [('sf1', 'i4')])])
-        a['f1'][0] = 1
-        a['f2'][0] = 2
-        a['f3'][0] = (3,)
-        b = np.zeros((1,), dtype=[('f1', 'i4'),
-                                  ('f2', 'i4'),
-                                  ('f3', [('sf1', 'i4')])])
-        b['f1'][0] = 1
-        b['f2'][0] = 2
-        b['f3'][0] = (3,)
-
-        # All the different functions raise a warning, but not an error
-        assert_equal(collect_warnings(a[['f1', 'f2']].__setitem__, 0, (10, 20)),
-                     [FutureWarning])
-        # For <=1.12 a is not modified, but it will be in 1.13
-        assert_equal(a, b)
-
-        # Views also warn
-        subset = a[['f1', 'f2']]
-        subset_view = subset.view()
-        assert_equal(collect_warnings(subset_view['f1'].__setitem__, 0, 10),
-                     [FutureWarning])
-        # But the write goes through:
-        assert_equal(subset['f1'][0], 10)
-        # Only one warning per multiple field indexing, though (even if there
-        # are multiple views involved):
-        assert_equal(collect_warnings(subset['f1'].__setitem__, 0, 10), [])
-
-        # make sure views of a multi-field index warn too
-        c = np.zeros(3, dtype='i8,i8,i8')
-        assert_equal(collect_warnings(c[['f0', 'f2']].view, 'i8,i8'),
-                     [FutureWarning])
-
     def test_record_hash(self):
         a = np.array([(1, 2), (1, 2)], dtype='i1,i2')
         a.flags.writeable = False
@@ -5074,6 +5014,16 @@ def test_empty_structure_creation(self):
         np.array([(), (), (), (), ()], dtype={'names': [], 'formats': [],
                                            'offsets': [], 'itemsize': 12})
 
+    def test_multifield_indexing_view(self):
+        a = np.ones(3, dtype=[('a', 'i4'), ('b', 'f4'), ('c', 'u4')])
+        v = a[['a', 'c']]
+        assert_(v.base is a)
+        assert_(v.dtype == np.dtype({'names': ['a', 'c'],
+                                     'formats': ['i4', 'u4'],
+                                     'offsets': [0, 8]}))
+        v[:] = (4,5)
+        assert_equal(a[0].item(), (4, 1, 5))
+
 class TestView(object):
     def test_basic(self):
         x = np.array([(1, 2, 3, 4), (5, 6, 7, 8)],
diff --git a/numpy/core/tests/test_records.py b/numpy/core/tests/test_records.py
--- a/numpy/core/tests/test_records.py
+++ b/numpy/core/tests/test_records.py
@@ -379,7 +379,6 @@ def test_nonwriteable_setfield(self):
         with assert_raises(ValueError):
             r.setfield([2,3], *r.dtype.fields['f'])
 
-    @pytest.mark.xfail(reason="See gh-10411, becomes real error in 1.16")
     def test_out_of_order_fields(self):
         # names in the same order, padding added to descr
         x = self.data[['col1', 'col2']]
diff --git a/numpy/lib/tests/test_io.py b/numpy/lib/tests/test_io.py
--- a/numpy/lib/tests/test_io.py
+++ b/numpy/lib/tests/test_io.py
@@ -355,6 +355,16 @@ def test_record(self):
         c.seek(0)
         assert_equal(c.readlines(), [b'1 2\n', b'3 4\n'])
 
+    @pytest.mark.skipif(Path is None, reason="No pathlib.Path")
+    def test_multifield_view(self):
+        a = np.ones(1, dtype=[('x', 'i4'), ('y', 'i4'), ('z', 'f4')])
+        v = a[['x', 'z']]
+        with temppath(suffix='.npy') as path:
+            path = Path(path)
+            np.save(path, v)
+            data = np.load(path)
+            assert_array_equal(data, v)
+
     def test_delimiter(self):
         a = np.array([[1., 2.], [3., 4.]])
         c = BytesIO()