Repository: numpy/numpy

Issue:
100 times poorer performance indexing scalar record array (Trac #1386)
_Original ticket http://projects.scipy.org/numpy/ticket/1386 on 2010-01-31 by trac user oscar.bristol, assigned to unknown._

Assigning to a field in a scalar record array like this
B['a'] = A['a']
takes around 50 times as long as assigning like this:
B['a'][:] = A['a']
and 100 times as long as assigning to a size 1 record array.
I've attached a script (test.py) that compares scalars and size 1 arrays and the two different methods of indexing above. On my machine, the results are:

$ python test.py 100000
array,  B['a'][:] = A['a']      0.25 seconds
array,  B['a']    = A['a']      0.213 seconds
scalar, B['a'][:] = A['a']      0.395 seconds
scalar, B['a']    = A['a']      22.0 seconds



Discussion:
Attachment added by trac user oscar.bristol on 2010-01-31: [test.py](http://projects.scipy.org/numpy/attachment/ticket/1386/test.py)

_trac user oscar.bristol wrote on 2010-01-31_

(Second attempt)
Assigning to a field in a scalar record array like this:

```
B['a'][:] = A['a'][:]
```

takes around 50 times as long as assigning like this:

```
B['a'][:] = A['a']
```

and 100 times as long as assigning to a size 1 record array. I've attached a script (test.py) that compares scalars and size 1 arrays and the two different methods of indexing above. On my machine, the results are:

```
$ python test.py 100000 
array,  B['a'][:] = A['a'] 0.25 seconds
array,  B['a']    = A['a'] 0.213 seconds
scalar, B['a'][:] = A['a'] 0.395 seconds
scalar, B['a']    = A['a'] 22.0 seconds
```

Attachment added by trac user jpeel on 2011-01-12: [0001-BF-poor-performance-indexing-scalar-record-array.patch](http://projects.scipy.org/numpy/attachment/ticket/1386/0001-BF-poor-performance-indexing-scalar-record-array.patch)

_trac user jpeel wrote on 2011-01-12_

The problem was that the scalar void type's setfield method makes a copy of the input array even if it is already an array. I made a fast track for when the right hand side value is an array. The fast track uses the getfield method (which is used in the B['a'][:] scalar example above) to get a Numpy array for B['a']. I then just use PyArray_CopyObject to do the transfer. Now, this last method of assignment is right with the others in time (actually faster than scalar B['a'][:] because a slice is not done).

Milestone changed to `NumPy 2.0` by trac user jpeel on 2011-01-12

Now it's only 10x as fast ;) @seberg @juliantaylor Either of you interested in taking a look at this?


Resolving patch:
diff --git a/benchmarks/benchmarks/bench_indexing.py b/benchmarks/benchmarks/bench_indexing.py
--- a/benchmarks/benchmarks/bench_indexing.py
+++ b/benchmarks/benchmarks/bench_indexing.py
@@ -47,3 +47,26 @@ def time_mmap_slicing(self):
     def time_mmap_fancy_indexing(self):
         for i in range(1000):
             self.fp[self.indexes]
+
+
+class IndexingStructured0D(Benchmark):
+    def setup(self):
+        self.dt = np.dtype([('a', 'f4', 256)])
+
+        self.A = np.zeros((), self.dt)
+        self.B = self.A.copy()
+
+        self.a = np.zeros(1, self.dt)[0]
+        self.b = self.a.copy()
+
+    def time_array_slice(self):
+        self.B['a'][:] = self.A['a']
+
+    def time_array_all(self):
+        self.B['a'] = self.A['a']
+
+    def time_scalar_slice(self):
+        self.b['a'][:] = self.a['a']
+
+    def time_scalar_all(self):
+        self.b['a'] = self.a['a']
diff --git a/numpy/core/_internal.py b/numpy/core/_internal.py
--- a/numpy/core/_internal.py
+++ b/numpy/core/_internal.py
@@ -288,55 +288,23 @@ def _newnames(datatype, order):
         return tuple(list(order) + nameslist)
     raise ValueError("unsupported order value: %s" % (order,))
 
-def _index_fields(ary, names):
-    """ Given a structured array and a sequence of field names
-    construct new array with just those fields.
+def _copy_fields(ary):
+    """Return copy of structured array with padding between fields removed.
 
     Parameters
     ----------
     ary : ndarray
-        Structured array being subscripted
-    names : string or list of strings
-        Either a single field name, or a list of field names
+       Structured array from which to remove padding bytes
 
     Returns
     -------
-    sub_ary : ndarray
-        If `names` is a single field name, the return value is identical to
-        ary.getfield, a writeable view into `ary`. If `names` is a list of
-        field names the return value is a copy of `ary` containing only those
-        fields. This is planned to return a view in the future.
-    
-    Raises
-    ------
-    ValueError
-        If `ary` does not contain a field given in `names`.
-
+    ary_copy : ndarray
+       Copy of ary with padding bytes removed
     """
     dt = ary.dtype
-
-    #use getfield to index a single field
-    if isinstance(names, basestring):
-        try:
-            return ary.getfield(dt.fields[names][0], dt.fields[names][1])
-        except KeyError:
-            raise ValueError("no field of name %s" % names)
-
-    for name in names:
-        if name not in dt.fields:
-            raise ValueError("no field of name %s" % name)
-
-    formats = [dt.fields[name][0] for name in names]
-    offsets = [dt.fields[name][1] for name in names]
-
-    view_dtype = {'names': names, 'formats': formats,
-                  'offsets': offsets, 'itemsize': dt.itemsize}
-
-    # return copy for now (future plan to return ary.view(dtype=view_dtype))
-    copy_dtype = {'names': view_dtype['names'],
-                  'formats': view_dtype['formats']}
-    return array(ary.view(dtype=view_dtype), dtype=copy_dtype, copy=True)
-
+    copy_dtype = {'names': dt.names,
+                  'formats': [dt.fields[name][0] for name in dt.names]}
+    return array(ary, dtype=copy_dtype, copy=True)
 
 def _get_all_field_offsets(dtype, base_offset=0):
     """ Returns the types and offsets of all fields in a (possibly structured)
@@ -478,6 +446,12 @@ def _view_is_safe(oldtype, newtype):
         If the new type is incompatible with the old type.
 
     """
+
+    # if the types are equivalent, there is no problem.
+    # for example: dtype((np.record, 'i4,i4')) == dtype((np.void, 'i4,i4'))
+    if oldtype == newtype:
+        return
+
     new_fields = _get_all_field_offsets(newtype)
     new_size = newtype.itemsize
 


Test patch:
diff --git a/numpy/core/tests/test_multiarray.py b/numpy/core/tests/test_multiarray.py
--- a/numpy/core/tests/test_multiarray.py
+++ b/numpy/core/tests/test_multiarray.py
@@ -3754,8 +3754,8 @@ def test_field_names(self):
             b[0][fn1] = 2
             assert_equal(b[fn1], 2)
             # Subfield
-            assert_raises(IndexError, b[0].__setitem__, fnn, 1)
-            assert_raises(IndexError, b[0].__getitem__, fnn)
+            assert_raises(ValueError, b[0].__setitem__, fnn, 1)
+            assert_raises(ValueError, b[0].__getitem__, fnn)
             # Subfield
             fn3 = func('f3')
             sfn1 = func('sf1')