Repository: numpy/numpy

Issue:
record array is made up of numpy.void, not numpy.record
I raised this previously on the PyTables issue list, but it they have asked me to transfer to numpy. (https://github.com/PyTables/PyTables/issues/271)

```
>>> import numpy as np
>>> a = (1, 2, 3)
>>> ra = np.rec.fromrecords([a])
>>> na = np.array([a], dtype='i8,i8,i8').view(type=np.recarray)
>>> na
rec.array([(1, 2, 3)], 
      dtype=[('f0', '<i8'), ('f1', '<i8'), ('f2', '<i8')])
>>> ra
rec.array([(1, 2, 3)], 
      dtype=[('f0', '<i8'), ('f1', '<i8'), ('f2', '<i8')])
>>> type(ra)
<class 'numpy.core.records.recarray'>
>>> type(na)
<class 'numpy.core.records.recarray'>
>>> ra[0]
(1, 2, 3)
>>> na[0]
(1, 2, 3)
>>> type(ra[0])
<class 'numpy.core.records.record'>
>>> type(na[0])
<class 'numpy.void'>
```

Any reason why .view can't return record arrays instead of sort of fake record arrays, or voidarrays? Is there a cast missing in the code somewhere?



Discussion:
Just wondering if anyone is in a position to confirm/deny this as an issue?

Bueller?
Bueller?

bump for this little annoyer - I can confirm this bug and it regularly affects me... accessing structure arrays with ['fieldname'] as the longtime workaround just looks so wrong to boot...

Anything that could be done to fix this issue would be appreciated... recarrays are powerful but this issue makes them tricky to use.

Does #5921 fix this issue, or is it not quite the same?

I just checked, and this doesn't appear to be fixed.  I'll take a look though.

Although it's not clear to me why `.view(type=numpy.recarray)` returns an `ndarray` object and not a `numpy.recarray` object.  Is that intentional? I'm not even sure.

@embray `.view(type=numpy.recarray)` does return a recarray, but it is hard to represent it in string form. If you look carefully in the repr there is an extra 'view' tacked on:

```
>>> np.array([a], dtype='i8,i8,i8').view(type=np.recarray)
array([(1, 2, 3)], 
  dtype=[('f0', '<i8'), ('f1', '<i8'), ('f2', '<i8')]).view(numpy.recarray)
```

I discussed it in https://github.com/numpy/numpy/pull/5523.

Also, I added new docs for record arrays which explain what I've done in https://github.com/numpy/numpy/pull/5482, which aren't visible in the online docs yet. Improvements welcome.

Also, I consider this issue fixed, as explained in #5523. 

In the example code, `ra` is a recarray with dtype of `np.record`, while `na` is a recarray with dtype of `np.void`.

You might consider this issue fixed, but highly related https://github.com/numpy/numpy/issues/4443 is still broken.  Just tested against the issue in that example and the others I considered dupes.

@nevion ha, I didn't mean to sound so final.

But #4443 is working as I expect. If you read the new docs I wrote in #5482 ([here](https://github.com/numpy/numpy/pull/5482/files#diff-82f17610f388b0841ccaf83e4485c7e8R216)) `np.rec.array(a)` and `a.view(np.recarray)` (for some ndarray a) give you two differently behaving objects.

The current issue #3581 arose because their reprs were identical, which was confusing. So in  #5523 I made sure their reprs are different, to make it clear to the user that they are different. I couldn't think of a better fix, but if there is one that would be great. The problem is that there are two separate aspects of record arrays: 1. They have type `np.recarray` and 2.  their dtype is `np.record`. I don't see a way of guaranteeing that both properties always come together, since the user can always do `.view(np.recarray)` which misses the `np.record` dtype.

 I think that `a.view(np.recarray)` should be discouraged, since as I explain in the docs it is a kind of "halfway" record array. Users should always use np.rec.array, or one of the other np.rec creation functions.

As I note in the docs you can get a "true" record array using a view by doing

```
a.view(dtype=(np.record, a.dtype), type=np.recarray)
```

Okay, so I guess the current behavior could be considered consistent, if confusing.  In any case it's maybe not worth worrying about since, last I heard, recarrays are all but deprecated?

Recarrays seem pretty widely used, but have never been 'official', which may be an oversight. I'm not sure what the long term plan for them should be. Feedback welcome, and maybe a post on the mailing list for discussion.

Sorry to be late in this, but I'm slightly confused why `recarray` wouldn't just turn the `dtype` into a proper `np.record` if a view is taken? This could be handled by adding an appropriate `__array_finalize__` to `np.recarray`.

@mhvk when I wrote those PRs I didn't know about `__array_finalize__`, but now I do and I was just thinking about that this morning. It looks like adding

```
def __array_finalize__(self, obj):
    self.dtype = dtype(record, self.dtype)
```

to np.recarray works. Is that the right way to change the dtype?

Edit: I spoke too soon that doesn't work

@ahaldane - yes, in principle, though be careful that you can get into `__array_finalize__` via different routes [1].  You may want to do something like

```
def __array_finalize__(self, obj):
    if obj is not None and type(obj) is not type(self):
        self.dtype = dtype(record, obj.dtype)
```

This way nothing gets changed when initializing or slicing a `recarray`. 

Though in practice, since for initialization or slicing the `dtype` is already correct, you could also just check whether `self.dtype` is correct and only change it if it is not.

[1] http://docs.scipy.org/doc/numpy/user/basics.subclassing.html#the-role-of-array-finalize

@mhvk I see. I think I got it to work (I was missing `()`) with this:

```
def __array_finalize__(self, obj):
    if obj is not None and type(obj) is not type(self): 
        self.dtype = sb.dtype((record, self.dtype))
```

then I see it works: 

```
>>> a = np.zeros(2 dtype='i4,f4')
>>> a.view(np.recarray)
rec.array([(0, 0.0), (0, 0.0)], 
      dtype=[('f0', '<i4'), ('f1', '<f4')])
>>> a.view(np.recarray).dtype
dtype((numpy.record, [('f0', '<i4'), ('f1', '<f4')]))
```

I'll submit a PR. Good thing this came up for discussion again!

@ahaldane - great! hope this solves @embray's issues too...

@mhvk I was thinking this as well. I guess I figured if you take a view with `type=np.recarray` but not with `dtype=(np.record, arr.dtype)` then maybe it would be consistent.  But I agree it's actually more consistent that `recarray` objects always return elements with `np.record` type, and your suggestion is exactly how that is done.


Resolving patch:
diff --git a/numpy/core/records.py b/numpy/core/records.py
--- a/numpy/core/records.py
+++ b/numpy/core/records.py
@@ -423,6 +423,12 @@ def __new__(subtype, shape, dtype=None, buf=None, offset=0, strides=None,
                                       strides=strides, order=order)
         return self
 
+    def __array_finalize__(self, obj):
+        if self.dtype.type is not record:
+            # if self.dtype is not np.record, invoke __setattr__ which will 
+            # convert it to a record if it is a void dtype.
+            self.dtype = self.dtype
+
     def __getattribute__(self, attr):
         # See if ndarray has this attr, and return it if so. (note that this
         # means a field with the same name as an ndarray attr cannot be
@@ -456,6 +462,11 @@ def __getattribute__(self, attr):
     # Undo any "setting" of the attribute and do a setfield
     # Thus, you can't create attributes on-the-fly that are field names.
     def __setattr__(self, attr, val):
+
+        # Automatically convert (void) dtypes to records.
+        if attr == 'dtype' and issubclass(val.type, nt.void):
+            val = sb.dtype((record, val))
+
         newattr = attr not in self.__dict__
         try:
             ret = object.__setattr__(self, attr, val)
@@ -502,8 +513,10 @@ def __repr__(self):
             # show zero-length shape unless it is (0,)
             lst = "[], shape=%s" % (repr(self.shape),)
 
-        if self.dtype.type is record:
+        if (self.dtype.type is record
+                or (not issubclass(self.dtype.type, nt.void)) ):
             # If this is a full record array (has numpy.record dtype),
+            # or if it has a scalar (non-void) dtype with no records,
             # represent it using the rec.array function. Since rec.array
             # converts dtype to a numpy.record for us, use only dtype.descr,
             # not repr(dtype).
@@ -512,7 +525,8 @@ def __repr__(self):
                           (lst, lf, repr(self.dtype.descr)))
         else:
             # otherwise represent it using np.array plus a view
-            # (There is currently (v1.10) no other easy way to create it)
+            # This should only happen if the user is playing
+            # strange games with dtypes.
             lf = '\n'+' '*len("array(")
             return ('array(%s, %sdtype=%s).view(numpy.recarray)' %
                           (lst, lf, str(self.dtype)))
@@ -534,22 +548,6 @@ def field(self, attr, val=None):
         else:
             return self.setfield(val, *res)
 
-    def view(self, dtype=None, type=None):
-        if dtype is None:
-            return ndarray.view(self, type)
-        elif type is None:
-            try:
-                if issubclass(dtype, ndarray):
-                    return ndarray.view(self, dtype)
-            except TypeError:
-                pass
-            dtype = sb.dtype(dtype)
-            if dtype.fields is None:
-                return self.__array__().view(dtype)
-            return ndarray.view(self, dtype)
-        else:
-            return ndarray.view(self, dtype, type)
-
 
 def fromarrays(arrayList, dtype=None, shape=None, formats=None,
                names=None, titles=None, aligned=False, byteorder=None):
@@ -837,10 +835,7 @@ def array(obj, dtype=None, shape=None, offset=0, strides=None, formats=None,
             new = obj
         if copy:
             new = new.copy()
-        res = new.view(recarray)
-        if issubclass(res.dtype.type, nt.void):
-            res.dtype = sb.dtype((record, res.dtype))
-        return res
+        return new.view(recarray)
 
     else:
         interface = getattr(obj, "__array_interface__", None)
@@ -849,7 +844,4 @@ def array(obj, dtype=None, shape=None, offset=0, strides=None, formats=None,
         obj = sb.array(obj)
         if dtype is not None and (obj.dtype != dtype):
             obj = obj.view(dtype)
-        res = obj.view(recarray)
-        if issubclass(res.dtype.type, nt.void):
-            res.dtype = sb.dtype((record, res.dtype))
-        return res
+        return obj.view(recarray)
diff --git a/numpy/doc/structured_arrays.py b/numpy/doc/structured_arrays.py
--- a/numpy/doc/structured_arrays.py
+++ b/numpy/doc/structured_arrays.py
@@ -258,6 +258,19 @@
  >>> recordarr = arr.view(dtype=dtype((np.record, arr.dtype)), 
  ...                      type=np.recarray)
 
+For convenience, viewing an ndarray as type `np.recarray` will automatically
+convert to `np.record` datatype, so the dtype can be left out of the view: ::
+
+ >>> recordarr = arr.view(np.recarray)
+ >>> recordarr.dtype
+ dtype((numpy.record, [('foo', '<i4'), ('bar', '<f4'), ('baz', 'S10')]))
+
+To get back to a plain ndarray both the dtype and type must be reset. The
+following view does so, taking into account the unusual case that the
+recordarr was not a structured type: ::
+
+ >>> arr2 = recordarr.view(recordarr.dtype.fields or recordarr.dtype, np.ndarray)
+
 Record array fields accessed by index or by attribute are returned as a record
 array if the field has a structured type but as a plain ndarray otherwise. ::
 
@@ -272,33 +285,6 @@
 attribute takes precedence. Such fields will be inaccessible by attribute but
 may still be accessed by index.
 
-Partial Attribute Access
-------------------------
-
-The differences between record arrays and plain structured arrays induce a
-small performance penalty. It is possible to apply one or the other view
-independently if desired. To allow field access by attribute only on the array
-object it is sufficient to view an array as a recarray: ::
-
- >>> recarr = arr.view(np.recarray)
-
-This type of view is commonly used, for example in np.npyio and
-np.recfunctions. Note that unlike full record arrays the individual elements of
-such a view do not have field attributes::
-
- >>> recarr[0].foo
- AttributeError: 'numpy.void' object has no attribute 'foo'
-
-To use the np.record dtype only, convert the dtype using the (base_class,
-dtype) form described in numpy.dtype.  This type of view is rarely used. ::
-
- >>> arr_records = arr.view(dtype((np.record, arr.dtype)))
-
-In documentation, the term 'structured array' will refer to objects of type
-np.ndarray with structured dtype, 'record array' will refer to structured
-arrays subclassed as np.recarray and whose dtype is of type np.record, and
-'recarray' will refer to arrays subclassed as np.recarray but whose dtype is
-not of type np.record.
 
 """
 from __future__ import division, absolute_import, print_function
diff --git a/numpy/ma/core.py b/numpy/ma/core.py
--- a/numpy/ma/core.py
+++ b/numpy/ma/core.py
@@ -2966,17 +2966,13 @@ def view(self, dtype=None, type=None, fill_value=None):
                 output = ndarray.view(self, dtype)
         else:
             output = ndarray.view(self, dtype, type)
-        # Should we update the mask ?
+
+        # also make the mask be a view (so attr changes to the view's
+        # mask do no affect original object's mask)
+        # (especially important to avoid affecting np.masked singleton)
         if (getattr(output, '_mask', nomask) is not nomask):
-            if dtype is None:
-                dtype = output.dtype
-            mdtype = make_mask_descr(dtype)
-            output._mask = self._mask.view(mdtype, ndarray)
-            # Try to reset the shape of the mask (if we don't have a void)
-            try:
-                output._mask.shape = output.shape
-            except (AttributeError, TypeError):
-                pass
+            output._mask = output._mask.view()
+
         # Make sure to reset the _fill_value if needed
         if getattr(output, '_fill_value', None) is not None:
             if fill_value is None:
@@ -3157,6 +3153,17 @@ def __setitem__(self, indx, value):
             _mask[indx] = mindx
         return
 
+    def __setattr__(self, attr, value):
+        super(MaskedArray, self).__setattr__(attr, value)
+        if attr == 'dtype' and self._mask is not nomask:
+            self._mask = self._mask.view(make_mask_descr(value), ndarray)
+
+            # Try to reset the shape of the mask (if we don't have a void)
+            # This raises a ValueError if the dtype change won't work
+            try:
+                self._mask.shape = self.shape
+            except (AttributeError, TypeError):
+                pass
 
     def __getslice__(self, i, j):
         """x.__getslice__(i, j) <==> x[i:j]
@@ -4906,6 +4913,9 @@ def anom(self, axis=None, dtype=None):
 
         """
         m = self.mean(axis, dtype)
+        if m is masked:
+            return m
+
         if not axis:
             return (self - m)
         else:


Test patch:
diff --git a/numpy/core/tests/test_records.py b/numpy/core/tests/test_records.py
--- a/numpy/core/tests/test_records.py
+++ b/numpy/core/tests/test_records.py
@@ -88,13 +88,50 @@ def test_recarray_from_repr(self):
         assert_equal(recordarr, recordarr_r)
 
         assert_equal(type(recarr_r), np.recarray)
-        assert_equal(recarr_r.dtype.type, np.void)
+        assert_equal(recarr_r.dtype.type, np.record)
         assert_equal(recarr, recarr_r)
 
         assert_equal(type(recordview_r), np.ndarray)
         assert_equal(recordview.dtype.type, np.record)
         assert_equal(recordview, recordview_r)
 
+    def test_recarray_views(self):
+        a = np.array([(1,'ABC'), (2, "DEF")],
+                     dtype=[('foo', int), ('bar', 'S4')])
+        b = np.array([1,2,3,4,5], dtype=np.int64)
+
+        #check that np.rec.array gives right dtypes
+        assert_equal(np.rec.array(a).dtype.type, np.record)
+        assert_equal(type(np.rec.array(a)), np.recarray)
+        assert_equal(np.rec.array(b).dtype.type, np.int64)
+        assert_equal(type(np.rec.array(b)), np.recarray)
+
+        #check that viewing as recarray does the same
+        assert_equal(a.view(np.recarray).dtype.type, np.record)
+        assert_equal(type(a.view(np.recarray)), np.recarray)
+        assert_equal(b.view(np.recarray).dtype.type, np.int64)
+        assert_equal(type(b.view(np.recarray)), np.recarray)
+
+        #check that view to non-structured dtype preserves type=np.recarray
+        r = np.rec.array(np.ones(4, dtype="f4,i4"))
+        rv = r.view('f8').view('f4,i4')
+        assert_equal(type(rv), np.recarray)
+        assert_equal(rv.dtype.type, np.record)
+
+        #check that we can undo the view
+        arrs = [np.ones(4, dtype='f4,i4'), np.ones(4, dtype='f8')]
+        for arr in arrs:
+            rec = np.rec.array(arr)
+            # recommended way to view as an ndarray:
+            arr2 = rec.view(rec.dtype.fields or rec.dtype, np.ndarray)
+            assert_equal(arr2.dtype.type, arr.dtype.type)
+            assert_equal(type(arr2), type(arr))
+
+    def test_recarray_repr(self):
+        # make sure non-structured dtypes also show up as rec.array
+        a = np.array(np.ones(4, dtype='f8'))
+        assert_(repr(np.rec.array(a)).startswith('rec.array'))
+
     def test_recarray_from_names(self):
         ra = np.rec.array([
             (1, 'abc', 3.7000002861022949, 0),
diff --git a/numpy/ma/tests/test_core.py b/numpy/ma/tests/test_core.py
--- a/numpy/ma/tests/test_core.py
+++ b/numpy/ma/tests/test_core.py
@@ -1401,6 +1401,32 @@ def test_flat(self):
         assert_equal(b01.data, array([[1., 0.]]))
         assert_equal(b01.mask, array([[False, False]]))
 
+    def test_assign_dtype(self):
+        # check that the mask's dtype is updated when dtype is changed
+        a = np.zeros(4, dtype='f4,i4')
+
+        m = np.ma.array(a)
+        m.dtype = np.dtype('f4')
+        repr(m) # raises?
+        assert_equal(m.dtype, np.dtype('f4'))
+
+        # check that dtype changes that change shape of mask too much
+        # are not allowed
+        def assign():
+            m = np.ma.array(a)
+            m.dtype = np.dtype('f8')
+        assert_raises(ValueError, assign)
+
+        b = a.view(dtype='f4', type=np.ma.MaskedArray) # raises?
+        assert_equal(b.dtype, np.dtype('f4'))
+
+        # check that nomask is preserved
+        a = np.zeros(4, dtype='f4')
+        m = np.ma.array(a)
+        m.dtype = np.dtype('f4,i4')
+        assert_equal(m.dtype, np.dtype('f4,i4'))
+        assert_equal(m._mask, np.ma.nomask)
+
 
 #------------------------------------------------------------------------------
 class TestFillingValues(TestCase):