Repository: yt-project/yt

Issue:
Extraheader with Gadget
<!--To help us understand and resolve your issue, please fill out the form to
the best of your ability.-->
<!--You can feel free to delete the sections that do not apply.-->

### Bug report

**Bug summary**

<!--A short 1-2 sentences that succinctly describes the bug-->
When reading a gadget file with an extra header (e.g. https://obswww.unige.ch/~lhausamm/swift/IC/AgoraDisk/Gear/snapshot_0000), you do not take it into account here https://github.com/yt-project/yt/blob/master/yt/frontends/gadget/data_structures.py#L396

**Code for reproduction**

<!--A minimum code snippet required to reproduce the bug, also minimizing the
number of dependencies required.-->
I am using the AGORA analysis script (https://bitbucket.org/mornkr/agora-analysis-script/src/8f12b112e19614d66274ece5a5bf21af23a76778/script-Paper4-disk-new.py?at=default&fileviewer=file-view-default) where they define the extra header here https://bitbucket.org/mornkr/agora-analysis-script/src/8f12b112e19614d66274ece5a5bf21af23a76778/script-Paper4-disk-new.py?at=default&fileviewer=file-view-default#script-Paper4-disk-new.py-150 and the code fails when adding a field here https://bitbucket.org/mornkr/agora-analysis-script/src/8f12b112e19614d66274ece5a5bf21af23a76778/script-Paper4-disk-new.py?at=default&fileviewer=file-view-default#script-Paper4-disk-new.py-501

<!-- If you need to use a data file to trigger the issue you're having, consider
using one of the datasets from the yt data hub (http://yt-project.org/data). If
your issue cannot be triggered using a public dataset, you can use the yt
curldrop (https://docs.hub.yt/services.html#curldrop) to share data
files. Please include a link to the dataset in the issue if you use the
curldrop.-->

**Actual outcome**

<!--The output produced by the above code, which may be a screenshot, console
output, etc.-->

```
Traceback (most recent call last):
  File "./plot_solution.py", line 519, in <module>
    pf.add_field(("gas", "density_squared"), function=_density_squared, units="g**2/cm**6")
  File "/usr/local/lib/python2.7/dist-packages/yt/data_objects/static_output.py", line 1191, in add_field
    self.index
  File "/usr/local/lib/python2.7/dist-packages/yt/data_objects/static_output.py", line 504, in index
    self, dataset_type=self.dataset_type)
  File "/usr/local/lib/python2.7/dist-packages/yt/geometry/particle_geometry_handler.py", line 39, in __init__
    super(ParticleIndex, self).__init__(ds, dataset_type)
  File "/usr/local/lib/python2.7/dist-packages/yt/geometry/geometry_handler.py", line 50, in __init__
    self._setup_geometry()
  File "/usr/local/lib/python2.7/dist-packages/yt/geometry/particle_geometry_handler.py", line 50, in _setup_geometry
    self._initialize_particle_handler()
  File "/usr/local/lib/python2.7/dist-packages/yt/geometry/particle_geometry_handler.py", line 104, in _initialize_particle_handler
    self._initialize_indices()
  File "/usr/local/lib/python2.7/dist-packages/yt/geometry/particle_geometry_handler.py", line 134, in _initialize_indices
    self.io._initialize_index(data_file, self.regions)
  File "/usr/local/lib/python2.7/dist-packages/yt/frontends/gadget/io.py", line 351, in _initialize_index
    data_file, count, 0, regions, DLE, DRE)
  File "/usr/local/lib/python2.7/dist-packages/yt/frontends/gadget/io.py", line 334, in _get_morton_from_position
    pp = np.fromfile(f, dtype=self._endian + self._float_type,
TypeError: cannot concatenate 'str' and 'int' objects

```

**Expected outcome**

<!--A description of the expected outcome from the code snippet-->
<!--If this used to work in an earlier version of yt, please note the
version it used to work on-->

**Version Information**
<!--Please specify your platform and versions of the relevant libraries you are
using:-->
  * Operating System: Linux Ubuntu 16.04
  * Python Version: 2.7.12
  * yt version: 3.4.1 (installed through pip)
  * Other Libraries (if applicable): 

<!--Please tell us how you installed yt and python e.g., from source,
pip, conda. If you installed from conda, please specify which channel you used
if not the default-->




Discussion:
Thanks for the report and steps to reproduce. I will try to take a look at this soon.
Hi @loikki, I ran into the same issue with another dataset. Basically, I can trigger the issue by

```python
from yt.frontends.gadget.api import GadgetDataset
ds = GadgetDataset('path/to/snapshot')
ds.field_list
```

But my dataset is too large (> 30G) to share. I'm wondering if you could share your dataset again (the first link is broken)? I'd like to take a deeper look and hopefully come up with a fix.
We do have a number of twiddles for specifying additional fields and whatnot (`gadget_header_spec` https://github.com/yt-project/yt/blob/master/yt/frontends/gadget/definitions.py#L17 ), but it looks like this falls into a corner case.
@MatthewTurk I guess the fix would be to check the code and use a calculated header size instead of 256, for example in this [line](https://github.com/yt-project/yt/blob/15c0691d9de2abdf320eaaccab5d2b8ac3af4aef/yt/frontends/gadget/data_structures.py#L396) as @loikki pointed out.
Sure, if real-world datasets don't have that exact amount of padding, we should inspect the data and calculate it.

FWIW I'm pretty sure that constant comes from the documentation for the the Gadget-2 output format in the user guide, I think it was fair of yt to expect real-world datasets to conform to the format documented there although if that's not the case we should definitely fix it :)

Presumably there's a way to do this either by looking very low-level at the bytes in the header or through creative use of the `struct` module.
@ngoldbaum Yeah, 256 is the Gadget standard. But since [`header_spec` is already configurable](https://github.com/yt-project/yt/blob/15c0691d9de2abdf320eaaccab5d2b8ac3af4aef/yt/frontends/gadget/data_structures.py#L107), to be self-consistent, I think we should use a caculated header size.

I'm willing to make the change if you think this is the right way to solve the issue.

Resolving patch:
diff --git a/yt/frontends/gadget/data_structures.py b/yt/frontends/gadget/data_structures.py
--- a/yt/frontends/gadget/data_structures.py
+++ b/yt/frontends/gadget/data_structures.py
@@ -50,18 +50,26 @@ def _fix_unit_ordering(unit):
     return unit
 
 
-def _get_gadget_format(filename):
+def _byte_swap_32(x):
+    return struct.unpack('>I', struct.pack('<I', x))[0]
+
+
+def _compute_header_size(header_spec):
+    return sum(field[1] * np.dtype(field[2]).itemsize for field in header_spec)
+
+
+def _get_gadget_format(filename, header_size):
     # check and return gadget binary format with file endianness
     ff = open(filename, 'rb')
     (rhead,) = struct.unpack('<I', ff.read(4))
     ff.close()
-    if (rhead == 134217728):
+    if (rhead == _byte_swap_32(8)):
         return 2, '>'
     elif (rhead == 8):
         return 2, '<'
-    elif (rhead == 65536):
+    elif (rhead == _byte_swap_32(header_size)):
         return 1, '>'
-    elif (rhead == 256):
+    elif (rhead == header_size):
         return 1, '<'
     else:
         raise RuntimeError("Incorrect Gadget format %s!" % str(rhead))
@@ -69,7 +77,7 @@ def _get_gadget_format(filename):
 
 class GadgetBinaryFile(ParticleFile):
     def __init__(self, ds, io, filename, file_id):
-        gformat = _get_gadget_format(filename)
+        gformat = _get_gadget_format(filename, ds._header_size)
         with open(filename, "rb") as f:
             if gformat[0] == 2:
                 f.seek(f.tell() + SNAP_FORMAT_2_OFFSET)
@@ -116,6 +124,19 @@ def __init__(self, filename, dataset_type="gadget_binary",
             return
         self._header_spec = self._setup_binary_spec(
             header_spec, gadget_header_specs)
+        self._header_size = _compute_header_size(self._header_spec)
+        if self._header_size != 256:
+            only_on_root(
+                mylog.warn,
+                "Non-standard header size is detected! "
+                "Gadget-2 standard header is 256 bytes, but yours is %s. "
+                "Make sure a non-standard header is actually expected. "
+                "Otherwise something is wrong, "
+                "and you might want to check how the dataset is loaded. "
+                "Futher information about header specification can be found in "
+                "https://yt-project.org/docs/dev/examining/loading_data.html#header-specification.",
+                self._header_size
+            )
         self._field_spec = self._setup_binary_spec(
             field_spec, gadget_field_specs)
         self._ptype_spec = self._setup_binary_spec(
@@ -157,7 +178,8 @@ def __init__(self, filename, dataset_type="gadget_binary",
             self.length_unit.convert_to_units('kpc')
             self.mass_unit.convert_to_units('Msun')
 
-    def _setup_binary_spec(self, spec, spec_dict):
+    @classmethod
+    def _setup_binary_spec(cls, spec, spec_dict):
         if isinstance(spec, str):
             _hs = ()
             for hs in spec.split("+"):
@@ -171,7 +193,7 @@ def __repr__(self):
     def _get_hvals(self):
         # The entries in this header are capitalized and named to match Table 4
         # in the GADGET-2 user guide.
-        gformat = _get_gadget_format(self.parameter_filename)
+        gformat = _get_gadget_format(self.parameter_filename, self._header_size)
         f = open(self.parameter_filename, 'rb')
         if gformat[0] == 2:
             f.seek(f.tell() + SNAP_FORMAT_2_OFFSET)
@@ -350,7 +372,7 @@ def _set_code_unit_attributes(self):
         self.specific_energy_unit = self.quan(*specific_energy_unit)
 
     @staticmethod
-    def _validate_header(filename):
+    def _validate_header(filename, header_size):
         '''
         This method automatically detects whether the Gadget file is big/little endian
         and is not corrupt/invalid using the first 4 bytes in the file.  It returns a
@@ -367,20 +389,22 @@ def _validate_header(filename):
 
         # First int32 is 256 for a Gadget2 binary file with SnapFormat=1,
         # 8 for a Gadget2 binary file with SnapFormat=2 file,
-        # or the byte swapped equivalents (65536 and 134217728).
+        # or the byte swapped equivalents.
         # The int32 following the header (first 4+256 bytes) must equal this
         # number.
+        # Note that 256 is the Gadget2 standard value, but other value could be
+        # set using the header_size argument.
         try:
             (rhead,) = struct.unpack('<I', f.read(4))
         except struct.error:
             f.close()
             return False, 1
         # Use value to check endianness
-        if rhead == 256:
+        if rhead == header_size:
             endianswap = '<'
-        elif rhead == 65536:
+        elif rhead == _byte_swap_32(header_size):
             endianswap = '>'
-        elif rhead in (8, 134217728):
+        elif rhead in (8, _byte_swap_32(8)):
             # This is only true for snapshot format 2
             # we do not currently support double precision
             # snap format 2 data
@@ -393,7 +417,7 @@ def _validate_header(filename):
         np0 = sum(struct.unpack(endianswap + 'IIIIII', f.read(6 * 4)))
         # Read in size of position block. It should be 4 bytes per float,
         # with 3 coordinates (x,y,z) per particle. (12 bytes per particle)
-        f.seek(4 + 256 + 4, 0)
+        f.seek(4 + header_size + 4, 0)
         np1 = struct.unpack(endianswap + 'I', f.read(4))[0] / (4 * 3)
         f.close()
         # Compare
@@ -405,9 +429,16 @@ def _validate_header(filename):
             return False, 1
 
     @classmethod
-    def _is_valid(self, *args, **kwargs):
+    def _is_valid(cls, *args, **kwargs):
+        if 'header_spec' in kwargs:
+            # Compute header size if header is customized
+            header_spec = cls._setup_binary_spec(
+                kwargs['header_spec'], gadget_header_specs)
+            header_size = _compute_header_size(header_spec)
+        else:
+            header_size = 256
         # First 4 bytes used to check load
-        return GadgetDataset._validate_header(args[0])[0]
+        return GadgetDataset._validate_header(args[0], header_size)[0]
 
 
 class GadgetHDF5Dataset(GadgetDataset):
diff --git a/yt/frontends/gadget/io.py b/yt/frontends/gadget/io.py
--- a/yt/frontends/gadget/io.py
+++ b/yt/frontends/gadget/io.py
@@ -242,7 +242,7 @@ def __init__(self, ds, *args, **kwargs):
         self._fields = ds._field_spec
         self._ptypes = ds._ptype_spec
         self.data_files = set([])
-        gformat = _get_gadget_format(ds.parameter_filename)
+        gformat = _get_gadget_format(ds.parameter_filename, ds._header_size)
         # gadget format 1 original, 2 with block name
         self._format = gformat[0]
         self._endian = gformat[1]
@@ -344,7 +344,8 @@ def _get_morton_from_position(self, data_file, count, offset_count,
     def _initialize_index(self, data_file, regions):
         DLE = data_file.ds.domain_left_edge
         DRE = data_file.ds.domain_right_edge
-        self._float_type = data_file.ds._validate_header(data_file.filename)[1]
+        self._float_type = data_file.ds._validate_header(
+            data_file.filename, data_file.ds._header_size)[1]
         if self.index_ptype == "all":
             count = sum(data_file.total_particles.values())
             return self._get_morton_from_position(


Test patch:
diff --git a/yt/frontends/gadget/testing.py b/yt/frontends/gadget/testing.py
new file mode 100644
--- /dev/null
+++ b/yt/frontends/gadget/testing.py
@@ -0,0 +1,109 @@
+import numpy as np
+
+from .definitions import (
+    gadget_field_specs,
+    gadget_header_specs,
+    gadget_ptype_specs,
+)
+from .data_structures import GadgetDataset
+from .io import IOHandlerGadgetBinary
+
+
+vector_fields = dict(IOHandlerGadgetBinary._vector_fields)
+
+block_ids = {
+    'Coordinates': 'POS',
+    'Velocities': 'VEL',
+    'ParticleIDs': 'ID',
+    'Mass': 'MASS',
+    'InternalEnergy': 'U',
+    'Density': 'RHO',
+    'SmoothingLength': 'HSML',
+}
+
+
+def write_record(fp, data, endian):
+    dtype = endian + 'i4'
+    size = np.array(data.nbytes, dtype=dtype)
+    fp.write(size.tobytes())
+    fp.write(data.tobytes())
+    fp.write(size.tobytes())
+
+
+def write_block(fp, data, endian, fmt, block_id):
+    assert fmt in [1, 2]
+    block_id = '%-4s' % block_id
+    if fmt ==2:
+        block_id_dtype = np.dtype(
+            [('id', 'S', 4), ('offset', endian + 'i4', 1)]
+        )
+        block_id_data = np.zeros(1, dtype=block_id_dtype)
+        block_id_data['id'] = block_id
+        block_id_data['offset'] = data.nbytes + 8
+        write_record(fp, block_id_data, endian)
+    write_record(fp, data, endian)
+
+
+def fake_gadget_binary(
+        filename='fake_gadget_binary',
+        npart=(100, 100, 100, 0, 100, 0),
+        header_spec='default',
+        field_spec='default',
+        ptype_spec='default',
+        endian='', fmt=2
+    ):
+    """Generate a fake Gadget binary snapshot."""
+    header_spec = GadgetDataset._setup_binary_spec(
+        header_spec, gadget_header_specs
+    )
+    field_spec = GadgetDataset._setup_binary_spec(
+        field_spec, gadget_field_specs
+    )
+    ptype_spec = GadgetDataset._setup_binary_spec(
+        ptype_spec, gadget_ptype_specs
+    )
+    with open(filename, 'wb') as fp:
+        # Generate and write header block
+        header_dtype = np.dtype(
+            [(name, endian + dtype, dim) for name, dim, dtype in header_spec]
+        )
+        header = np.zeros(1, dtype=header_dtype)
+        header['Npart'] = npart
+        header['Nall'] = npart
+        header['NumFiles'] = 1
+        header['BoxSize'] = 1
+        header['HubbleParam'] = 1
+        write_block(fp, header, endian, fmt, 'HEAD')
+
+        npart = dict(zip(ptype_spec, npart))
+        for fs in field_spec:
+            # Parse field name and particle type
+            if isinstance(fs, str):
+                field = fs
+                ptype = ptype_spec
+            else:
+                field, ptype = fs
+                if isinstance(ptype, str):
+                    ptype = (ptype,)
+            # Determine field dimension
+            if field in vector_fields:
+                dim = vector_fields[field]
+            else:
+                dim = 1
+            # Determine dtype (in numpy convention)
+            if field == 'ParticleIDs':
+                dtype = 'u4'
+            else:
+                dtype = 'f4'
+            dtype = endian + dtype
+            # Generate and write field block
+            data = []
+            for pt in ptype:
+                data += [np.random.rand(npart[pt], dim)]
+            data = np.concatenate(data).astype(dtype)
+            if field in block_ids:
+                block_id = block_ids[field]
+            else:
+                block_id = ''
+            write_block(fp, data, endian, fmt, block_id)
+    return filename
diff --git a/yt/frontends/gadget/tests/test_outputs.py b/yt/frontends/gadget/tests/test_outputs.py
--- a/yt/frontends/gadget/tests/test_outputs.py
+++ b/yt/frontends/gadget/tests/test_outputs.py
@@ -15,13 +15,17 @@
 #-----------------------------------------------------------------------------
 
 from collections import OrderedDict
+from itertools import product
+import os
 
+import yt
 from yt.testing import requires_file
 from yt.utilities.answer_testing.framework import \
     data_dir_load, \
     requires_ds, \
     sph_answer
 from yt.frontends.gadget.api import GadgetHDF5Dataset, GadgetDataset
+from yt.frontends.gadget.testing import fake_gadget_binary
 
 isothermal_h5 = "IsothermalCollapse/snap_505.hdf5"
 isothermal_bin = "IsothermalCollapse/snap_505"
@@ -45,17 +49,24 @@
 iso_kwargs = dict(bounding_box=[[-3, 3], [-3, 3], [-3, 3]])
 
 
+def test_gadget_binary():
+    header_specs = ['default', 'default+pad32']
+    for header_spec, endian, fmt in product(header_specs, '<>', [1, 2]):
+        fake_snap = fake_gadget_binary(
+            header_spec=header_spec,
+            endian=endian,
+            fmt=fmt
+        )
+        ds = yt.load(fake_snap, header_spec=header_spec)
+        assert isinstance(ds, GadgetDataset)
+        ds.field_list
+        os.remove(fake_snap)
+
+
 @requires_file(isothermal_h5)
-@requires_file(isothermal_bin)
-@requires_file(BE_Gadget)
-@requires_file(LE_SnapFormat2)
-def test_gadget_dataset():
+def test_gadget_hdf5():
     assert isinstance(data_dir_load(isothermal_h5, kwargs=iso_kwargs),
                       GadgetHDF5Dataset)
-    assert isinstance(data_dir_load(isothermal_bin, kwargs=iso_kwargs),
-                      GadgetDataset)
-    assert isinstance(data_dir_load(BE_Gadget), GadgetDataset)
-    assert isinstance(data_dir_load(LE_SnapFormat2), GadgetDataset)
 
 
 @requires_file(keplerian_ring)