Repository: numpy/numpy

Issue:
NPY header alignment
According to the NPY format [spec](http://docs.scipy.org/doc/numpy-dev/neps/npy-format.html), the header should be padded to be aligned on the 16-byte boundary

> The next `HEADER_LEN` bytes form the header data describing the array’s format. It is an ASCII string which contains a Python literal expression of a dictionary. It is terminated by a newline (‘n’) and padded with spaces (‘x20’) to make the total length of the magic string + 4 + `HEADER_LEN` be evenly divisible by 16 for alignment purposes

The 4 in the above formula accounts for 
- 2 version bytes and 
- `HEADER_LEN` itself, which is 2 bytes in NPY1.0. 

Interestingly, the same formula [is used](https://github.com/numpy/numpy/blob/master/numpy/lib/format.py#L311) for NPY2.0 files, despite the fact that `HEADER_LEN` is now 4 bytes. Therefore, NPY2.0 file header is not guaranteed to be aligned. I think this might be worth pointing out in the docs.

By the way, did anyone ever profile this optimization?

numpy.save() data alignment increase from 16 to 64 bytes
This is regarding the NumPy file format: https://docs.scipy.org/doc/numpy-dev/neps/npy-format.html

The format guarantees 16 byte alignment of the stored array.  But sometimes this is not enough.  For example, current Intel processors support AVX2 which operates on 32 byte chunks.

Currently, if an array is stored with `numpy.save()` and the data happens not to be aligned to 32 bytes (which will happen about 50% of the time), feeding the data to AVX2 instructions (perhaps outside of NumPy) requires extra work to realign the data.  I presume that avoiding such extra work is the reason why there is 16 byte alignment in the first place.

I propose increasing the minimum alignment from 16 bytes to 64 bytes.  As far as I can tell this will be backward compatible because `np.load()` doesn't care how much padding there is before the newline which ends the header.  That is, the NumPy file header version would not need to change.

Why 64 bytes?  It's good for the upcoming AVX-512, it's not too wasteful, and I don't see a current need for more.


Discussion:
@juliantaylor IIRC, you did the npy 2 format. I don't know if this can be fixed in a way compatible with files already out there, or even if it still matters. SPARC might be a problem.

should be a backward compatible change as it just would just change the number of padding spaces

the alignment is very unlikely to be relevant for performance but might be worthwhile to fix it or update the documentation.

> should be a backward compatible change as it just would just change the number of padding spaces

Completely dropping the padding should also be backward compatible, since NumPy does `safe_eval` and any alternative implementations (if any) probably just strips the spaces before further processing.

cpus since haswell don't have any measurable penalty for 16 byte aligned data. This only has a very small effect on sandybridge type cpus.
Also for storing this doesn't really have any effect at all.
A weak argument can be made for the general memory allocation inside numpy, but there is only very little that uses avx inside numpy and there is no aligned calloc available that is compatible with free, so its not worth it to have a very minimal effect on by now rather old cpus.

As for avx512, I don't have the hardware to test alignment performance, but given how their newer cpus work, the goal of that instruction set and how numpy itself works I doubt 64 bit alignment is necessary.
@juliantaylor Please see for example VMOVAPS which is described here: http://www.felixcloutier.com/x86/MOVAPS.html

The description says:

> When the source or destination operand is a memory operand, the operand must be aligned on a 16-byte (128-bit version) or 32-byte (VEX.256 encoded version) boundary or a general-protection exception (#GP) will be generated.

So I'm not really talking about the performance penalty of unaligned access, but rather the fact that a fault will occur if code expecting full alignment (to 32 bytes in the AVX2 case) receives data not so aligned.
AVX, like SSE before it, has unaligned instructions, e.g. VMOVUPS. So you can use AVX on unaligned memory just fine, and compilers knowing that it has no significant penalty will also use these type of instructions.
An issue is that you can't directly use operations with memory operands without unaligned loading first, but in practice this makes no difference anymore, at least I have not seen any code yet where a difference can be measured.
actually no, its even better, except for the explicit aligned ones like VMOVAPS and unlike SSE, most avx instructions (like VADDPS) can operate on unaligned memory operands. So there is really no issue.
@juliantaylor I think you're coming at this from the perspective of someone who would write the SIMD code themselves, as opposed to someone who merely hopes to use SIMD on data stored in the NumPy format.  If I have a library that uses aligned instructions and a file format which is not sufficiently aligned, there is indeed still an issue that must be resolved--either with extra code to realign the data, or by increasing the alignment in the file format.  I believe the latter is both less work overall and also in keeping with the idea that aligned data is good (else why did we have padding to 16 bytes in the first place?).
It is still a file format, if you really have such a special library that can't deal with it, you should first file a bug and get them to support 16 byte alignment on haswell and newer and until that is fixed you can always read the header manually and mmap the right offset. Then you have page aligned memory.
gh-8085 still needs fixing, so we could increase the padding in that fix.
@juliantaylor there is a problem with what you said here:

> you can always read the header manually and mmap the right offset. Then you have page aligned memory.

On Linux, the offset argument to `mmap()` *must* be page-aligned (e.g. to 4096 bytes).  So this idea can't be used on Linux (which happens to be the platform I'm using).

Resolving patch:
diff --git a/numpy/lib/format.py b/numpy/lib/format.py
--- a/numpy/lib/format.py
+++ b/numpy/lib/format.py
@@ -100,9 +100,9 @@
 The next HEADER_LEN bytes form the header data describing the array's
 format. It is an ASCII string which contains a Python literal expression
 of a dictionary. It is terminated by a newline (``\\n``) and padded with
-spaces (``\\x20``) to make the total length of
-``magic string + 4 + HEADER_LEN`` be evenly divisible by 16 for alignment
-purposes.
+spaces (``\\x20``) to make the total of
+``len(magic string) + 2 + len(length) + HEADER_LEN`` be evenly divisible
+by 64 for alignment purposes.
 
 The dictionary contains three keys:
 
@@ -163,6 +163,7 @@
 
 MAGIC_PREFIX = b'\x93NUMPY'
 MAGIC_LEN = len(MAGIC_PREFIX) + 2
+ARRAY_ALIGN = 64 # plausible values are powers of 2 between 16 and 4096
 BUFFER_SIZE = 2**18  # size of buffer for reading npz files in bytes
 
 # difference between version 1.0 and 2.0 is a 4 byte (I) header length
@@ -304,27 +305,33 @@ def _write_array_header(fp, d, version=None):
         header.append("'%s': %s, " % (key, repr(value)))
     header.append("}")
     header = "".join(header)
-    # Pad the header with spaces and a final newline such that the magic
-    # string, the header-length short and the header are aligned on a
-    # 16-byte boundary.  Hopefully, some system, possibly memory-mapping,
-    # can take advantage of our premature optimization.
-    current_header_len = MAGIC_LEN + 2 + len(header) + 1  # 1 for the newline
-    topad = 16 - (current_header_len % 16)
-    header = header + ' '*topad + '\n'
     header = asbytes(_filter_header(header))
 
-    hlen = len(header)
-    if hlen < 256*256 and version in (None, (1, 0)):
+    hlen = len(header) + 1 # 1 for newline
+    padlen_v1 = ARRAY_ALIGN - ((MAGIC_LEN + struct.calcsize('<H') + hlen) % ARRAY_ALIGN)
+    padlen_v2 = ARRAY_ALIGN - ((MAGIC_LEN + struct.calcsize('<I') + hlen) % ARRAY_ALIGN)
+
+    # Which version(s) we write depends on the total header size; v1 has a max of 65535
+    if hlen + padlen_v1 < 2**16 and version in (None, (1, 0)):
         version = (1, 0)
-        header_prefix = magic(1, 0) + struct.pack('<H', hlen)
-    elif hlen < 2**32 and version in (None, (2, 0)):
+        header_prefix = magic(1, 0) + struct.pack('<H', hlen + padlen_v1)
+        topad = padlen_v1
+    elif hlen + padlen_v2 < 2**32 and version in (None, (2, 0)):
         version = (2, 0)
-        header_prefix = magic(2, 0) + struct.pack('<I', hlen)
+        header_prefix = magic(2, 0) + struct.pack('<I', hlen + padlen_v2)
+        topad = padlen_v2
     else:
         msg = "Header length %s too big for version=%s"
         msg %= (hlen, version)
         raise ValueError(msg)
 
+    # Pad the header with spaces and a final newline such that the magic
+    # string, the header-length short and the header are aligned on a
+    # ARRAY_ALIGN byte boundary.  This supports memory mapping of dtypes
+    # aligned up to ARRAY_ALIGN on systems like Linux where mmap()
+    # offset must be page-aligned (i.e. the beginning of the file).
+    header = header + b' '*topad + b'\n'
+
     fp.write(header_prefix)
     fp.write(header)
     return version
@@ -468,18 +475,18 @@ def _read_array_header(fp, version):
     # header.
     import struct
     if version == (1, 0):
-        hlength_str = _read_bytes(fp, 2, "array header length")
-        header_length = struct.unpack('<H', hlength_str)[0]
-        header = _read_bytes(fp, header_length, "array header")
+        hlength_type = '<H'
     elif version == (2, 0):
-        hlength_str = _read_bytes(fp, 4, "array header length")
-        header_length = struct.unpack('<I', hlength_str)[0]
-        header = _read_bytes(fp, header_length, "array header")
+        hlength_type = '<I'
     else:
         raise ValueError("Invalid version %r" % version)
 
+    hlength_str = _read_bytes(fp, struct.calcsize(hlength_type), "array header length")
+    header_length = struct.unpack(hlength_type, hlength_str)[0]
+    header = _read_bytes(fp, header_length, "array header")
+
     # The header is a pretty-printed string representation of a literal
-    # Python dictionary with trailing newlines padded to a 16-byte
+    # Python dictionary with trailing newlines padded to a ARRAY_ALIGN byte
     # boundary. The keys are strings.
     #   "shape" : tuple of int
     #   "fortran_order" : bool


Test patch:
diff --git a/numpy/lib/tests/test_format.py b/numpy/lib/tests/test_format.py
--- a/numpy/lib/tests/test_format.py
+++ b/numpy/lib/tests/test_format.py
@@ -615,6 +615,11 @@ def test_version_2_0():
         format.write_array(f, d)
         assert_(w[0].category is UserWarning)
 
+    # check alignment of data portion
+    f.seek(0)
+    header = f.readline()
+    assert_(len(header) % format.ARRAY_ALIGN == 0)
+
     f.seek(0)
     n = format.read_array(f)
     assert_array_equal(d, n)
@@ -758,6 +763,7 @@ def test_read_array_header_1_0():
     s.seek(format.MAGIC_LEN)
     shape, fortran, dtype = format.read_array_header_1_0(s)
 
+    assert_(s.tell() % format.ARRAY_ALIGN == 0)
     assert_((shape, fortran, dtype) == ((3, 6), False, float))
 
 
@@ -770,6 +776,7 @@ def test_read_array_header_2_0():
     s.seek(format.MAGIC_LEN)
     shape, fortran, dtype = format.read_array_header_2_0(s)
 
+    assert_(s.tell() % format.ARRAY_ALIGN == 0)
     assert_((shape, fortran, dtype) == ((3, 6), False, float))