Package: pyread-eagle
Read routines for Peano-Hilbert key sorted Eagle snapshotsPure-python port of J. Helly'sread_eagle.QuickstartIf you're used to using the originalread_eagle, all you should need to do is change import statements fromimport read_eagletoimport pyread_eagle as read_eagle, or similar (from pyread_eagle import EagleSnapshot, etc.). Existing scripts should work seamlessly with this change.InstallationInstall from pip with:'pip install pyread_eagle'Install from github:Download via web UI, or 'git clonehttps://github.com/kyleaoman/pyread_eagle.git'Install dependencies if necessary (see 'setup.py'), some may be found in other repositories by kyleaoman.Global install (Linux):cd to directory with 'setup.py'run 'sudo pip install -e .' (-e installs via symlink, so pulling repository will do a 'live' update of the installation)User install (Linux):cd to directory with 'setup.py'ensure '~/lib/python3.7/site-packages' or similar is on your PYTHONPATH (e.g. 'echo $PYTHONPATH'), if not, add it (perhaps in .bash_profile or similar)run 'pip install --prefix ~ -e .' (-e installs via symlink, so pulling repository will do a 'live' update of the installation)cd to a directory outside the module and launch python; you should be able to do 'from pyread_eagle import *'Alternately, pip can retrieve and install the version on github directly:'pip3 install git+https://github.com/kyleaoman/pyread_eagle.git'DocumentationThe documentation below is adapted from the originalread_eagle readme file.These routines can be used if Gadget was run with the-DEAGLE_SORT_OUTPUTflag enabled. They provide a quick way to read in spatial regions
without having to read all the data or even open all of the files.It works by splitting the simulation volume into a grid and ensuring
that particles in the same grid cell are stored consecutively in the
snapshot files. Some extra datasets are added to the snapshot which
specify which grid cells are stored in which files and the location
of the start of each cell in the file.The procedure to read in data is as follows:Open the snapshot by specifying the name of one fileFlag the grid cells to be read in by callingselect_regionone or
more times.Callread_datasetonce for each quantity to be read, specifying
which particle type to read and the hdf5 dataset name.You can then either close/delete the snapshot object or callclear_selectionto
read in a different region.Any of the datasets in thePartTypeXgroups can be read in. The code
makes no assumptions about what datasets are present in the snapshot.Unlike the originalread_eagle, this version is written directly in python
and therefore does not depend on having a properly configured C compiler
or hdf5 libraries, all that is needed is python, numpy and h5py.Python2 is not supported. Ithinkpyread_eaglewill run in python2.7,
but since python2 is deprecated as of Jan 2020, I have not and have no intention
to test this.Simple example usagefrom pyread_eagle import EagleSnapshot

snap = EagleSnapshot("./snap_020.0.hdf5")
snap.select_region(4.0, 5.0, 2.0, 3.0, 3.0, 4.0)
xyz = snap.read_dataset(0, "Coordinates")
ids = snap.read_dataset(0, "ParticleIDs")
del snapSpeedpyread_eagleis comparable in speed toread_eagle. The actual read_dataset calls
vary depending on the region (how contiguous it is in the dataset) between very roughly 2x faster
and 5x slower than the C version. Given the limited flexibility of theh5pyAPI, I believe it
will be difficult to speed this up further.Error handlingIf a routine fails it will raise an exception. Because of differences in
the structure of the code relative toread_eagle, not all the same sanity
checks are made, not all the same errors can arise, and therefore not all
the same error messages may appear. I have also chosen to be more lax and
not try to catch every edge case by hand, rather letting failures happen
and be reported organically by the underlying python libraries. Please
report any errors whose messages are obscure, and of course any bugs.Random samplingNote that the output of pyread_eagle will differ from the original version
when the sampling rate is set < 1.0. In the original version, a subsample
was drawn by comparing a pseudo-random number to the sampling rate for each
particle. This resulted in a subsample ofapproximatelythe requested
fraction of particles. Since the seed was set explicitly, the results were
reproducible. Since pyread_eagle uses the numpy PRNG, the output cannot
match what would be obtained using the C PRNG. Instead, a random sample of
consistent length is drawn (the seed is still set explicitly, so results
remain reproducible run-to-run), equal to floor(number of particle * sampling rate),
evaluatedfile-by-file. So, the overall total number of particles times the
sampling rate will not always be equal to the length of the output array,
due to rounding errors (the largest possible difference is the number
of files in the snapshot).Description of the routines and parametersOpen snapshotsnap = pyread_eagle.EagleSnapshot(fname, verbose=False)This opens the snapshot which contains the specified file. Closing
explicitly is not necessary: in python the snapshot will be automatically
deallocated once there are no more references to it.Parametersfname: name of any one file in the snapshotverbose: ifTrue, print extra messages during function calls (default: False)Return value
An instance of theEagleSnapshotclassClose snapshotdel snapsnap.close()Deallocates memory associated with the snap object.
Not strictly necessary - will happen automatically when the snap variable goes
out of scope. Attempting to call methods of a closed snapshot raises anEagleSnapshotClosedException.Region selectionsnap.select_region(xmin, xmax, ymin, ymax, zmin, zmax)All grid cells overlapping the specified region are flagged to be read
in by subsequentread_dataset calls. You can callselect_regionmultiple
times to make oddly shaped or disjoint selections.If selected regions overlap or the same region is selected multiple times
particles in these regions will still only be read in once.Parametersxmin: the minimum x coordinate of the region to readxmax: the maximum x coordinate of the region to readymin: the minimum y coordinate of the region to readymax: the maximum y coordinate of the region to readzmin: the minimum z coordinate of the region to readzmax: the maximum z coordinate of the region to readsnap.select_grid_cells(ixmin, ixmax, iymin, iymax, izmin, izmax)All grid cells in the specified range of grid coordinates are flagged to
be read in by subsequentread_datasetcalls. You can callselect_grid_cellsmultiple times to make oddly shaped or disjoint selections.The coordinates ixmin, ixmax etc are integer coordinates in the hash grid,
starting from zero. The maximum coordinate is(2**hashbits)-1. The value of
hashbits is stored insnap.hashbits.If selected regions overlap or the same region is selected multiple times
particles in these regions will still only be read in once.Parametersixmin: the minimum x coordinate of the region to readixmax: the maximum x coordinate of the region to readiymin: the minimum y coordinate of the region to readiymax: the maximum y coordinate of the region to readizmin: the minimum z coordinate of the region to readizmax: the maximum z coordinate of the region to readsnap.select_rotated_region(centre, xvec, yvec, zvec, length)Selects a box-shaped region rotated relative to the simulation box axes.
The[xyz]vecparameters should be a set of mutually orthogonal unit vectors
defining the axes of the box to be selected.Parameterscentre: coordinate centre of the region to select, shape (3, )xvec: selection box 'x' unit vector in simulation box coordinates, shape (3, )yvec: selection box 'y' unit vector in simulation box coordinates, shape (3, )zvec: selection box 'z' unit vector in simulation box coordinates, shape (3, )length: selection box lengths (not half-length) along each axis, shape (3, )Count particlesn = snap.count_particles(itype)This returns the number of particles of the specified type which will be
read by the nextread_datasetcall. Note that only whole grid cells can
be read so some particles outside the selected region may be read in.
These are included in the count.In python this routine is not usually needed, but it is included for
compatibility with codes formerly using the read_eagle python wrapper.Parametersitype: which particle type to count (integer,0-5)Return value
The number of particles to be read inGet particle locationsfile_index, file_offset = snap.get_particle_locations(itype)This returns two arrays which each have one element for each selected
particle.file_indexcontains the index of the file each particle is in.file_offsetcontains the position in the file, numbering from zero.Parametersitype: which particle type to count (integer,0-5)Return valuefile_index- integer array with index of the file containing each particlefile_offset- integer array with position of each particle in its fileRead datasetsdata = snap.read_dataset(itype, name)This reads in the specified dataset for all particles of type itype in the
selected region(s). Use repeated calls to read in multiple datasets.The type of array you get back reflects the type of the dataset in the file.Parametersitype: which particle type to read (integer,0-5)name: the HDF5 name of the dataset, relative to thePartTypeXgroupReturn value
The contents of the dataset for the selected particlesdata = snap.read_extra_dataset(itype, name, basename)This is the same asread_dataset, except that it reads from a set of
"auxiliary files" comprised of the same number of parts as the snapshot and
which contain additional datasets with the same sorting, distribution across
files, etc. as the datasets in the main snapshot files.Parametersitype: which particle type to read (integer,0-5)name: the HDF5 name of the dataset, relative to thePartTypeXgroupbasename: name (including path, if necessary) of the auxiliary files, omitting the.X.hdf5portionClear selectionsnap.clear_selection()This clears the flags which specify which grid cells should be
read in on the next read_dataset() call. If you've already read
in a region and you want to read a different region you should call
clear_selection() before calling select_region() again.Split selectionsnap.split_selection(ThisTask, NTask)This is for use in MPI programs to allow parallel I/O.
When running under MPI all read_eagle routines are collective -
they must be called on all processes.The procedure to read a region in parallel is as follows:All processes open the snapshotAll processes select the SAME regionCallsplit_selection. This causes each processor to select
a subset of the originally selected particles.Callread_dataset()on all processors to read the required
quantitiesThis results in all of the particles in the specified region
being read in exactly once with the data spread across the MPI
processes.I have not made any attempt to use pyread_eagle with e.g. mpi4py
or other MPI-ish implementations in python. If you succeed in this,
please get in touch!ParametersThisTask: rank of this process in an MPI programNTask: number of processes in an MPI programSet sampling ratesnap.set_sampling_rate(rate)If rate is set < 1 before reading datasets, a random subsample
of particles will be read, with the fraction of particles read
set by the rate. See the note on random subsamples (above).Parametersrate: fraction of particles to be read (rate >= 1 for all particles)List datasetsdsets = snap.datasets(itype)Returns a list of the available datasets for a given particle type.
This can be iterated to read all datasets, for instance.Parametersitype: which particle type to read (integer,0-5)