Repository: numba/numba

Issue:
weird behavior of Records.make_c_struct.

OK so the idea I want to implement is make a `cfunc` and pass that function into some C++ code which calls the function pointer as shown in this python script. Here `test_callback` is some cython module which compiles a C function which calls the function pointer.
```
from test_callback import callback_C
from numba import cfunc,carray,types

wrapper = types.Record.make_c_struct([
   ('m', types.intc),
   ('n', types.complex128),
])

sig = types.void(types.CPointer(wrapper))

@cfunc(sig)
def C_caller(argptr):
	arg = carray(argptr,1)
	arg[0].m = 1
	arg[0].n = -1j
	return 

print wrapper
callback_C(C_caller)
```

note that the make_c_struct does not align the memory properly,
the result of `print wrapper` gives: `Record(m[type=int32;offset=0],n[type=complex128;offset=16];24;True)`
but the offset for `n` should be 4 bytes not 16. 
I can reverse the order and it works fine but in the current order will not work. 

weird behavior of Records.make_c_struct.

OK so the idea I want to implement is make a `cfunc` and pass that function into some C++ code which calls the function pointer as shown in this python script. Here `test_callback` is some cython module which compiles a C function which calls the function pointer.
```
from test_callback import callback_C
from numba import cfunc,carray,types

wrapper = types.Record.make_c_struct([
   ('m', types.intc),
   ('n', types.complex128),
])

sig = types.void(types.CPointer(wrapper))

@cfunc(sig)
def C_caller(argptr):
	arg = carray(argptr,1)
	arg[0].m = 1
	arg[0].n = -1j
	return 

print wrapper
callback_C(C_caller)
```

note that the make_c_struct does not align the memory properly,
the result of `print wrapper` gives: `Record(m[type=int32;offset=0],n[type=complex128;offset=16];24;True)`
but the offset for `n` should be 4 bytes not 16. 
I can reverse the order and it works fine but in the current order will not work. 



Discussion:
Thanks for the report. I think what you are seeing is due to the following. The `intc` is:
```
size: 4
align: 4
misalign: 0
offset: 0
```
then the `complex128` ends up as:
```
size: 16
align: 8
misalign: 4
offset: 16
```
i.e. it would be a misaligned load if packed against the 4 byte alignment of the `intc` so gets padded to the 16 byte offset. I think it might be possible to adjust the offset computation to be the `offset += align - misalign`, but probably not the suggested 4 byte offset as it would be misaligned for the type. @sklam thoughts?
@stuartarchibald OK I think maybe I don't quite understand how the data is actually packed into the struct.

however the outstanding issue is that the code I wrote does not work with the integer first, if you want I Can provide the source code for the C++/cython code which fails when you change the order in the struct. 
Looks like there's a bug in the alignment of `n`.

<details>

```python
from numba import cfunc,carray,types
from cffi import FFI
from pprint import pprint


wrapper = types.Record.make_c_struct([
    ('m', types.intc),
    ('n', types.complex128),
])

print(wrapper)


ffi = FFI()
ffi.cdef("""
struct complex128 { double real, imag; };
typedef struct _Rec{
    int m;
    struct complex128 n;
} Rec;
""")

c_rec_type = ffi.typeof('Rec')
pprint(dir(c_rec_type))
print(c_rec_type)
for k, fd in c_rec_type.fields:
    print(k, fd.offset, fd.type)
```
</details>

Prints:

```
Record(m[type=int32;offset=0],n[type=complex128;offset=16];24;True)
['cname', 'fields', 'kind']
<ctype 'Rec'>
m 0 <ctype 'int'>
n 8 <ctype 'struct complex128'>
```

`n` should have 8-byte offset but it's getting a 16-byte offset from `Record.make_c_struct`.
the `offset += align - misalign` suggestion seems to have fixed it.  i'll be opening a patch soon.
Oh, there's some trouble: 

```
In [9]: np.dtype([('m', np.intc), ('n', np.complex128)], align=True)
Out[9]: dtype({'names':['m','n'], 'formats':['<i4','<c16'], 'offsets':[0,16], 'itemsize':32}, align=True)
```

Numpy thinks `np.complex128` has 16-byte alignment but LLVM thinks it has 8-byte alignment.
According to https://github.com/numpy/numpy/issues/3768, the 16-byte alignment was due to sparc64.
so, this is fixed since numpy1.16.
Thanks for the report. I think what you are seeing is due to the following. The `intc` is:
```
size: 4
align: 4
misalign: 0
offset: 0
```
then the `complex128` ends up as:
```
size: 16
align: 8
misalign: 4
offset: 16
```
i.e. it would be a misaligned load if packed against the 4 byte alignment of the `intc` so gets padded to the 16 byte offset. I think it might be possible to adjust the offset computation to be the `offset += align - misalign`, but probably not the suggested 4 byte offset as it would be misaligned for the type. @sklam thoughts?
@stuartarchibald OK I think maybe I don't quite understand how the data is actually packed into the struct.

however the outstanding issue is that the code I wrote does not work with the integer first, if you want I Can provide the source code for the C++/cython code which fails when you change the order in the struct. 
Looks like there's a bug in the alignment of `n`.

<details>

```python
from numba import cfunc,carray,types
from cffi import FFI
from pprint import pprint


wrapper = types.Record.make_c_struct([
    ('m', types.intc),
    ('n', types.complex128),
])

print(wrapper)


ffi = FFI()
ffi.cdef("""
struct complex128 { double real, imag; };
typedef struct _Rec{
    int m;
    struct complex128 n;
} Rec;
""")

c_rec_type = ffi.typeof('Rec')
pprint(dir(c_rec_type))
print(c_rec_type)
for k, fd in c_rec_type.fields:
    print(k, fd.offset, fd.type)
```
</details>

Prints:

```
Record(m[type=int32;offset=0],n[type=complex128;offset=16];24;True)
['cname', 'fields', 'kind']
<ctype 'Rec'>
m 0 <ctype 'int'>
n 8 <ctype 'struct complex128'>
```

`n` should have 8-byte offset but it's getting a 16-byte offset from `Record.make_c_struct`.
the `offset += align - misalign` suggestion seems to have fixed it.  i'll be opening a patch soon.
Oh, there's some trouble: 

```
In [9]: np.dtype([('m', np.intc), ('n', np.complex128)], align=True)
Out[9]: dtype({'names':['m','n'], 'formats':['<i4','<c16'], 'offsets':[0,16], 'itemsize':32}, align=True)
```

Numpy thinks `np.complex128` has 16-byte alignment but LLVM thinks it has 8-byte alignment.
According to https://github.com/numpy/numpy/issues/3768, the 16-byte alignment was due to sparc64.
so, this is fixed since numpy1.16.

Resolving patch:
diff --git a/numba/numpy_support.py b/numba/numpy_support.py
--- a/numba/numpy_support.py
+++ b/numba/numpy_support.py
@@ -6,7 +6,7 @@
 
 import numpy as np
 
-from . import errors, types, config, npdatetime, utils
+from . import errors, types, config, utils
 
 
 version = tuple(map(int, np.__version__.split('.')[:2]))
@@ -149,6 +149,7 @@ def as_dtype(nbtype):
     raise NotImplementedError("%r cannot be represented as a Numpy dtype"
                               % (nbtype,))
 
+
 def as_struct_dtype(rec):
     """Convert Numba Record type to NumPy structured dtype
     """
@@ -160,8 +161,25 @@ def as_struct_dtype(rec):
         'offsets': [rec.offset(k) for k in names],
         'itemsize': rec.size,
     }
+    _check_struct_alignment(rec, fields)
     return np.dtype(fields, align=rec.aligned)
 
+
+def _check_struct_alignment(rec, fields):
+    """Check alignment compatibility with Numpy"""
+    if rec.aligned:
+        for k, dt in zip(fields['names'], fields['formats']):
+            llvm_align = rec.alignof(k)
+            npy_align = dt.alignment
+            if llvm_align is not None and npy_align != llvm_align:
+                msg = (
+                    'NumPy is using a different alignment ({}) '
+                    'than Numba/LLVM ({}) for {}. '
+                    'This is likely a NumPy bug.'
+                )
+                raise ValueError(msg.format(npy_align, llvm_align, dt))
+
+
 def is_arrayscalar(val):
     return np.dtype(type(val)) in FROM_DTYPE
 
@@ -202,12 +220,10 @@ def select_array_wrapper(inputs):
     An index into *inputs* is returned.
     """
     max_prio = float('-inf')
-    selected_input = None
     selected_index = None
     for index, ty in enumerate(inputs):
         # Ties are broken by choosing the first winner, as in Numpy
         if isinstance(ty, types.ArrayCompatible) and ty.array_priority > max_prio:
-            selected_input = ty
             selected_index = index
             max_prio = ty.array_priority
 
@@ -267,7 +283,7 @@ def supported_ufunc_loop(ufunc, loop):
         supported_types = '?bBhHiIlLqQfd'
         # check if all the types involved in the ufunc loop are
         # supported in this mode
-        supported_loop =  all(t in supported_types for t in loop_types)
+        supported_loop = all(t in supported_types for t in loop_types)
 
     return supported_loop
 
@@ -445,6 +461,7 @@ def _get_bytes_buffer(ptr, nbytes):
     arrty = ctypes.c_byte * nbytes
     return arrty.from_address(ptr)
 
+
 def _get_array_from_ptr(ptr, nbytes, dtype):
     return np.frombuffer(_get_bytes_buffer(ptr, nbytes), dtype)
 
diff --git a/numba/types/npytypes.py b/numba/types/npytypes.py
--- a/numba/types/npytypes.py
+++ b/numba/types/npytypes.py
@@ -44,7 +44,7 @@ def key(self):
         return self.count
 
 
-_RecordField = collections.namedtuple('_RecordField', 'type,offset')
+_RecordField = collections.namedtuple('_RecordField', 'type,offset,alignment')
 
 
 class Record(Type):
@@ -84,8 +84,10 @@ def make_c_struct(cls, name_types):
             # align
             misaligned = offset % align
             if misaligned:
-                offset += size - misaligned
-            fields.append((k, {'type': ty, 'offset': offset}))
+                offset += align - misaligned
+            fields.append((k, {
+                'type': ty, 'offset': offset, 'alignment': align,
+            }))
             offset += size
         # Adjust sizeof structure
         abi_size = ctx.get_abi_sizeof(ir.LiteralStructType(lltypes))
@@ -99,9 +101,13 @@ def __init__(self, fields, size, aligned):
 
         # Create description
         descbuf = []
-        fmt = "{}[type={};offset={}]"
+        fmt = "{}[type={};offset={}{}]"
         for k, infos in fields:
-            descbuf.append(fmt.format(k, infos.type, infos.offset))
+            if infos.alignment is not None:
+                extra = ';alignment={}'.format(infos.alignment)
+            else:
+                extra = ''
+            descbuf.append(fmt.format(k, infos.type, infos.offset, extra))
 
         desc = ','.join(descbuf)
         name = 'Record({};{};{})'.format(desc, self.size, self.aligned)
@@ -111,11 +117,15 @@ def __init__(self, fields, size, aligned):
     def _normalize_fields(cls, fields):
         """
         fields:
-            [name: str, { type: Type, offset: int }]
+            [name: str, { type: Type, offset: int, [ alignment: int ] }]
         """
         res = []
         for name, infos in sorted(fields, key=lambda x: (x[1]['offset'], x[0])):
-            fd = _RecordField(type=infos['type'], offset=infos['offset'])
+            fd = _RecordField(
+                type=infos['type'],
+                offset=infos['offset'],
+                alignment=infos.get('alignment'),
+            )
             res.append((name, fd))
         return res
 
@@ -144,6 +154,13 @@ def typeof(self, key):
         """
         return self.fields[key].type
 
+    def alignof(self, key):
+        """Get the specified alignment of the field.
+
+        Since field alignment is optional, this may return None.
+        """
+        return self.fields[key].alignment
+
     @property
     def members(self):
         """An ordered list of (name, type) for the fields.


Test patch:
diff --git a/numba/tests/test_numpy_support.py b/numba/tests/test_numpy_support.py
--- a/numba/tests/test_numpy_support.py
+++ b/numba/tests/test_numpy_support.py
@@ -10,7 +10,7 @@
 import numpy as np
 
 import numba.unittest_support as unittest
-from numba import config, numpy_support, types
+from numba import numpy_support, types
 from .support import TestCase, tag
 from .enum_usecases import Shake, RequestError
 
@@ -121,20 +121,20 @@ def check(dtype, fields, size, aligned):
 
         dtype = np.dtype([('a', np.int16), ('b', np.int32)])
         check(dtype,
-              fields={'a': (types.int16, 0),
-                      'b': (types.int32, 2)},
+              fields={'a': (types.int16, 0, None),
+                      'b': (types.int32, 2, None)},
               size=6, aligned=False)
 
         dtype = np.dtype([('a', np.int16), ('b', np.int32)], align=True)
         check(dtype,
-              fields={'a': (types.int16, 0),
-                      'b': (types.int32, 4)},
+              fields={'a': (types.int16, 0, None),
+                      'b': (types.int32, 4, None)},
               size=8, aligned=True)
 
         dtype = np.dtype([('m', np.int32), ('n', 'S5')])
         check(dtype,
-              fields={'m': (types.int32, 0),
-                      'n': (types.CharSeq(5), 4)},
+              fields={'m': (types.int32, 0, None),
+                      'n': (types.CharSeq(5), 4, None)},
               size=9, aligned=False)
 
     @tag('important')
@@ -187,7 +187,8 @@ def _base_check_datetime_values(self, func, np_type, nb_type):
         f = func
         for unit in [
             '', 'Y', 'M', 'D', 'h', 'm', 's',
-            'ms', 'us', 'ns', 'ps', 'fs', 'as']:
+            'ms', 'us', 'ns', 'ps', 'fs', 'as',
+        ]:
             if unit:
                 t = np_type(3, unit)
             else:
@@ -258,6 +259,7 @@ def __init__(self, types):
             assert len(in_) == self.nin
             assert len(out) == self.nout
 
+
 # Typical types for np.add, np.multiply, np.isnan
 _add_types = ['??->?', 'bb->b', 'BB->B', 'hh->h', 'HH->H', 'ii->i', 'II->I',
               'll->l', 'LL->L', 'qq->q', 'QQ->Q', 'ee->e', 'ff->f', 'dd->d',
diff --git a/numba/tests/test_record_dtype.py b/numba/tests/test_record_dtype.py
--- a/numba/tests/test_record_dtype.py
+++ b/numba/tests/test_record_dtype.py
@@ -9,60 +9,76 @@
 from numba.compiler import compile_isolated
 from numba.itanium_mangler import mangle_type
 from numba.utils import IS_PY3
+from numba.config import IS_WIN32
+from numba.numpy_support import version as numpy_version
 from .support import tag
 
 
 def get_a(ary, i):
     return ary[i].a
 
+
 def get_b(ary, i):
     return ary[i].b
 
+
 def get_c(ary, i):
     return ary[i].c
 
+
 def make_getitem(item):
     # This also exercises constant lookup from a closure variable
     def get_xx(ary, i):
         return ary[i][item]
     return get_xx
 
+
 # Issue #1664: constant index lookup should fall back to regular getitem
 def get_zero_a(ary, _unused):
     return ary[0].a
 
+
 getitem_a = make_getitem('a')
 getitem_b = make_getitem('b')
 getitem_c = make_getitem('c')
 
+
 def get_a_subarray(ary, i):
     return ary.a[i]
 
+
 def get_b_subarray(ary, i):
     return ary.b[i]
 
+
 def get_c_subarray(ary, i):
     return ary.c[i]
 
+
 def get_a_zero(ary, _unused):
     return ary.a[0]
 
+
 def make_getitem_subarray(item):
     # This also exercises constant lookup from a closure variable
     def get_xx_subarray(ary, i):
         return ary[item][i]
     return get_xx_subarray
 
+
 getitem_a_subarray = make_getitem_subarray('a')
 getitem_b_subarray = make_getitem_subarray('b')
 getitem_c_subarray = make_getitem_subarray('c')
 
+
 def get_two_arrays_a(ary1, ary2, i):
     return ary1[i].a + ary2[i].a
 
+
 def get_two_arrays_b(ary1, ary2, i):
     return ary1[i].b + ary2[i].b
 
+
 def get_two_arrays_c(ary1, ary2, i):
     return ary1[i].c + ary2[i].c
 
@@ -74,39 +90,49 @@ def get_two_arrays_distinct(ary1, ary2, i):
 def set_a(ary, i, v):
     ary[i].a = v
 
+
 def set_b(ary, i, v):
     ary[i].b = v
 
+
 def set_c(ary, i, v):
     ary[i].c = v
 
+
 def make_setitem(item):
     def set_xx(ary, i, v):
         ary[i][item] = v
     return set_xx
 
+
 setitem_a = make_setitem('a')
 setitem_b = make_setitem('b')
 setitem_c = make_setitem('c')
 
+
 def set_a_subarray(ary, i, v):
     ary.a[i] = v
 
+
 def set_b_subarray(ary, i, v):
     ary.b[i] = v
 
+
 def set_c_subarray(ary, i, v):
     ary.c[i] = v
 
+
 def make_setitem_subarray(item):
     def set_xx_subarray(ary, i, v):
         ary[item][i] = v
     return set_xx_subarray
 
+
 setitem_a_subarray = make_setitem('a')
 setitem_b_subarray = make_setitem('b')
 setitem_c_subarray = make_setitem('c')
 
+
 def set_record(ary, i, j):
     ary[i] = ary[j]
 
@@ -176,6 +202,7 @@ def record_write_array(ary):
     ary.h[0] = 3.0
     ary.h[1] = 4.0
 
+
 def record_write_2d_array(ary):
     ary.i = 3
     ary.j[0, 0] = 5.0
@@ -189,15 +216,19 @@ def record_write_2d_array(ary):
 def record_read_array0(ary):
     return ary.h[0]
 
+
 def record_read_array1(ary):
     return ary.h[1]
 
+
 def record_read_2d_array00(ary):
     return ary.j[0,0]
 
+
 def record_read_2d_array10(ary):
     return ary.j[1,0]
 
+
 def record_read_2d_array01(ary):
     return ary.j[0,1]
 
@@ -205,6 +236,7 @@ def record_read_2d_array01(ary):
 def record_read_first_arr(ary):
     return ary.k[2, 2]
 
+
 def record_read_second_arr(ary):
     return ary.l[2, 2]
 
@@ -293,6 +325,49 @@ class Ref(ctypes.Structure):
         dtype = ty.dtype
         self.assertTrue(dtype.isalignedstruct)
 
+    def test_complex_struct(self):
+        class Complex(ctypes.Structure):
+            _fields_ = [
+                ('real', ctypes.c_double),
+                ('imag', ctypes.c_double),
+            ]
+
+        class Ref(ctypes.Structure):
+            _fields_ = [
+                ('apple', ctypes.c_int32),
+                ('mango', Complex),
+            ]
+
+        ty = types.Record.make_c_struct([
+            ('apple', types.intc),
+            ('mango', types.complex128),
+        ])
+        # Correct offsets
+        self.assertEqual(len(ty), 2)
+        self.assertEqual(ty.offset('apple'), Ref.apple.offset)
+        self.assertEqual(ty.offset('mango'), Ref.mango.offset)
+        # Correct size
+        self.assertEqual(ty.size, ctypes.sizeof(Ref))
+        # Is aligned?
+        # NumPy version < 1.16 misalign complex-128 types to 16bytes.
+        # (it seems to align on windows?!)
+        if numpy_version >= (1, 16) or IS_WIN32:
+            dtype = ty.dtype
+            self.assertTrue(dtype.isalignedstruct)
+        else:
+            with self.assertRaises(ValueError) as raises:
+                dtype = ty.dtype
+            # get numpy alignment
+            npalign = np.dtype(np.complex128).alignment
+            # llvm should align to wordsize
+            llalign = np.dtype(np.intp).alignment
+            self.assertIn(
+                ("NumPy is using a different alignment ({}) "
+                 "than Numba/LLVM ({}) for complex128. "
+                 "This is likely a NumPy bug.").format(npalign, llalign),
+                str(raises.exception),
+            )
+
 
 class TestRecordDtype(unittest.TestCase):
 
@@ -391,7 +466,7 @@ def _test_get_two_equal(self, pyfunc):
         cfunc = self.get_cfunc(pyfunc, (rec[:], rec[:], types.intp))
         for i in range(self.refsample1d.size):
             self.assertEqual(pyfunc(self.refsample1d, self.refsample1d3, i),
-                              cfunc(self.nbsample1d, self.nbsample1d3, i))
+                             cfunc(self.nbsample1d, self.nbsample1d3, i))
 
     def test_two_distinct_arrays(self):
         '''
@@ -488,7 +563,7 @@ def _test_record_args(self, revargs):
         nbval = self.nbsample1d.copy()[0]
         attrs = 'abc'
         valtypes = types.float64, types.int16, types.complex64
-        values = 1.23, 12345, 123+456j
+        values = 1.23, 12345, 123 + 456j
         old_refcnt = sys.getrefcount(nbval)
 
         for attr, valtyp, val in zip(attrs, valtypes, values):
@@ -512,7 +587,7 @@ def _test_record_args(self, revargs):
             got = cfunc(*args)
             try:
                 self.assertEqual(expected, got)
-            except AssertionError as e:
+            except AssertionError:
                 # On ARM, a LLVM misoptimization can produce buggy code,
                 # see https://llvm.org/bugs/show_bug.cgi?id=24669
                 import llvmlite.binding as ll
@@ -534,7 +609,6 @@ def test_record_args(self):
     def test_record_args_reverse(self):
         self._test_record_args(True)
 
-
     def test_two_records(self):
         '''
         Testing the use of two scalar records of the same type
@@ -556,7 +630,6 @@ def test_two_records(self):
             got = cfunc(nbval1, nbval2)
             self.assertEqual(expected, got)
 
-
     def test_two_distinct_records(self):
         '''
         Testing the use of two scalar records of differing type
@@ -572,7 +645,6 @@ def test_two_distinct_records(self):
         got = cfunc(nbval1, nbval2)
         self.assertEqual(expected, got)
 
-
     def test_record_write_array(self):
         '''
         Testing writing to a 1D array within a structured type
@@ -588,7 +660,6 @@ def test_record_write_array(self):
         expected[0].h[1] = 4.0
         np.testing.assert_equal(expected, nbval)
 
-
     @tag('important')
     def test_record_write_2d_array(self):
         '''
@@ -605,7 +676,6 @@ def test_record_write_2d_array(self):
                                       np.float32).reshape(3, 2)
         np.testing.assert_equal(expected, nbval)
 
-
     def test_record_read_array(self):
         '''
         Test reading from a 1D array within a structured type
@@ -622,7 +692,6 @@ def test_record_read_array(self):
         res = cfunc(nbval[0])
         np.testing.assert_equal(res, nbval[0].h[1])
 
-
     @tag('important')
     def test_record_read_2d_array(self):
         '''
@@ -644,7 +713,6 @@ def test_record_read_2d_array(self):
         res = cfunc(nbval[0])
         np.testing.assert_equal(res, nbval[0].j[1, 0])
 
-
     @tag('important')
     def test_record_return(self):
         """
@@ -783,6 +851,7 @@ def _createSampleArrays(self):
         self.nbsample1d2 = np.zeros(3, dtype=recordtype2)
         self.nbsample1d3 = np.zeros(3, dtype=recordtype)
 
+
 class TestRecordDtypeWithStructArraysAndDispatcher(TestRecordDtypeWithStructArrays):
     '''
     Same as TestRecordDtypeWithStructArrays, stressing the Dispatcher's type
@@ -858,9 +927,9 @@ def test_py_argument_charseq(self):
         cfunc = cres.entry_point
 
         for i in range(self.refsample1d.size):
-            chars = "{0}".format(hex(i+10))
-            expected = pyfunc(self.refsample1d, i, chars)
-            got = cfunc(self.nbsample1d, i, chars)
+            chars = "{0}".format(hex(i + 10))
+            pyfunc(self.refsample1d, i, chars)
+            cfunc(self.nbsample1d, i, chars)
             np.testing.assert_equal(self.refsample1d, self.nbsample1d)
 
     def test_py_argument_char_seq_near_overflow(self):