User guide for GADGET-2
Volker Springel
volker@mpa-garching.mpg.de
Max-Plank-Institute for Astrophysics, Garching, Germany

May 1, 2005

Contents
1 Introduction

3

2 Basic usage
2.1 Compilation requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2 Starting the code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3 Interrupting a run . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.4 Restarting a run . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.4.1 Restarting from restart-files . . . . . . . . . . . . . . . . . . . . . . .
2.4.2 Restarting from snapshot-files . . . . . . . . . . . . . . . . . . . . . .

3
3
4
5
5
5
6

3 Types of simulations

7

4 Makefile options
4.1 Basic operation mode of code . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2 Things that are always recommended . . . . . . . . . . . . . . . . . . . . . . .
4.3 TreePM options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.4 Single/double precision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.5 Time integration options . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.6 Output options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.7 Things for special behaviour . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.8 Testing and debugging options . . . . . . . . . . . . . . . . . . . . . . . . . .
4.9 Glass making . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

7
9
9
9
11
11
12
12
14
14

5 Parameterfile
14
5.1 Filenames and file formats . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
5.2 CPU-time limit and restart options . . . . . . . . . . . . . . . . . . . . . . . . 17
5.3 Simulation specific parameters . . . . . . . . . . . . . . . . . . . . . . . . . . 18

1

V. Springel

5.4 Cosmological parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.5 Memory allocation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.6 Gravitational force accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.7 Time integration accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.8 Output of snapshot files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.9 System of units . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.10 SPH parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.11 Gravitational softening . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

19
20
21
22
24
25
26
27

6 File formats of GADGET-2: Snapshots & Initial conditions29
6.1 Structure of the default snapshot file format . . . . . . . . . . . . . . . . . . . 29
6.2 A variant of the default file format . . . . . . . . . . . . . . . . . . . . . . . . 34
6.3 The HDF file format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
6.4 Format of initial conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
7 Diagnostic output files
36
7.1 Information file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
7.2 Performance statistics of the gravitational tree . . . . . . . . . . . . . . . . . . 36
7.3 CPU-time statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
7.4 Energy statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
8 Differences between GADGET-1 and GADGET-2
40
8.1 Architectural changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
8.2 Notes on migrating from GADGET-1 to GADGET-2 . . . . . . . . . . . . . . . 41
8.3 Dropped makefile options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
8.4 List of source files of GADGET-2 . . . . . . . . . . . . . . . . . . . . . . . . . 42
8.5 Notes on memory consumption of GADGET-2 . . . . . . . . . . . . . . . . . . 42
9 Examples
44
9.1 Galaxy collision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
9.2 Adiabatic collapse of a gas sphere . . . . . . . . . . . . . . . . . . . . . . . . 45
9.3 Cosmological formation of a cluster of galaxies . . . . . . . . . . . . . . . . . 45
9.4 Large-scale structure formation including gas . . . . . . . . . . . . . . . . . . 46
10 Disclaimer

2

46

User’s guide for GADGET-2

1 Introduction
In its current implementation, the simulation code GADGET-2 (GAlaxies with Dark matter and
Gas intEracT1 ) supports collisionless simulations and smoothed particle hydrodynamics on
massively parallel computers. All communication is done explicitly by means of the message
passing interface (MPI). The code is written in standard ANSI C, and should run on all parallel
platforms that support MPI. So far, the portability of the code has been confirmed on a large
number of systems, including Cray-T3E, IBM-SP/2 and p690 systems, SGI Origins, various
systems from Sun, as well as a large number of Linux PC clusters (‘beowulfs’), based either onIntel or AMD processors.
The code can be used for plain Newtonian dynamics, or for cosmological integrations inarbitrary cosmologies, both with or without periodic boundary conditions. The modeling ofhydrodynamics is optional. The code is adaptive both in space and in time, and its Lagrangiancharacter makes it particularly suitable for simulations of cosmic structure formation.
The main reference for numerical and algorithmic aspects of the code is the paper ‘The cosmological simulation code GADGET-2’ (Springel, 2005, MNRAS, submitted), and references
therein. Further information on the previous public version GADGET-1 is also contained in
the paper ‘GADGET: A code for collisionless and gas-dynamical cosmological simulations’(Springel, Yoshida & White, 2001, New Astronomy, 6, 51). In the following, these papers
will be frequently referenced, and it is recommended to read them before attempting to use thecode. This document provides additional technical information about the code, hopefully in a
sufficiently self-contained fashion to allow anyone interested an independent use of the codefor cosmological N-body/SPH simulations on parallel machines.
GADGET-2 was written by Volker Springel, and is made publicly available under the GNU
general public license. This implies that you may freely copy, distribute, or modify the sources,
but the copyright for the original code remains with the author and the Max-Planck-Institute for
Astrophysics (MPA). If you find the code useful for your scientific work, we kindly ask you toinclude a reference to the code paper on GADGET-2 in all studies that use simulations carriedout with GADGET-2.

2 Basic usage
2.1 Compilation requirements
GADGET-2 needs the following non-standard libraries for compilation:• mpi - the ‘Message Passing Interface’ (version 1.0 or higher). Many vendor suppliedversions exist, in addition to excellent open source implementations, e.g.
MPICH (http://www-unix.mcs.anl.gov/mpi/mpich), or
LAM (http://www.lam-mpi.org).
• gsl - the GNU scientific library. This open-source package can be obtained at
http://www.gnu.org/software/gsl, for example. GADGET-2 only needs this1

This peculiar acronym hints at the code’s origin as a tool for studying galaxy collisions.

3V. Springel

library for a few very simple cosmological integrations at start-up.
• fftw - the ‘Fastest Fourier Transform in the West’. This open-source package can be obtained at http://www.fftw.org. Note that the MPI-capable version 2.x of FFTW
is required, the new version 3 lacks MPI capability at this point. FFTW is only needed
for simulations that use the TreePM algorithm.
• hdf5 - the ‘Hierarchical Data Format’ (version 5.0 or higher, available at
http://hdf.ncsa.uiuc.edu/HDF5). This optional library is only needed when
one wants to read or write snapshot files in HDF format. It is possible to compile and use
the code without this library.
Note that FFTW needs to be compiled with parallel support enabled. This can be achievedby passing the option --enable-mpi to its configure script. When at it, you might aswell add --enable-type-prefix to obtain the libraries in both a single and double precision version. The single-precision version can then be generated after the double precision one by a make clean command, followed by a reconfiguration that also includes the--enable-single option and a renewed make install. Note that without the--enable-type-prefix option in the compilation of FFTW, you should set the option
NOTYPEPREFIX FFTW in GADGET-2’s makefile to generate correct header and library names.
After unpacking the tar-file of GADGET-2, you will find a bunch of .c-files, .h-files, and a
Makefile. You should edit the Makefile and adjust it to your system, if needed. A numberof systems have been predefined in the Makefile (look at the SYSTYPE definitions), and byfollowing these examples, further customised system-types can be easily added. Note thatslight adjustments of the makefile will be needed if any of the above libraries is not installed ina standard location on your system. Also, compiler optimisation options may need adjustment,
depending on the C-compiler that is used. The provided makefile is compatible with GNUmake, i.e. typing make or gmake should then build the executable Gadget2. If your sitedoes not have GNU-make, get it, or modify the makefile accordingly.
The makefile also contains a number of compile-time options which determine the types ofsimulations that can be run with a given GADGET-2 executable. These options are explainedin Section 4. Further code options of GADGET-2 are controlled by a parameter file, which is
described in Section 5.
2.2 Starting the code
To start a simulation, invoke the executable with a command like
mpirun -np 32 ./Gadget2 myparameterfile.paramThis will start the simulation using 32 processors, and with simulation parameters as specified
in the parameter file (see below) of name myparameterfile.param. Note that on somesystems, the mpirun command may have a different syntax. Consult the man-pages or the
local support if in doubt.

4

User’s guide for GADGET-2

The code does not need to be recompiled for a different number of processors, or for adifferent problem size. It is also possible to run GADGET-2 with a single processor only. In thiscase, the leading ‘mpirun -np 1’ can be omitted, and GADGET-2 behaves like a serial code.
It is still necessary to have MPI installed, though, because even then the code will make some(superfluous) calls to the MPI library. For GADGET-1, two separate code versions for serial andparallel mode existed, GADGET-2 uses a single code base to take care of both functionalities.
Also, in GADGET-2, there is no restriction anymore for the processor number to be a powerof two, even though these partition sizes are in general prefered, because they allow the mostefficient communication schemes.
While GADGET-2 is run, it will print out a bunch of log-messages that inform about thepresent steps taken by the code. When you start a simulation interactively, these log-messages
will appear on the screen, but you can also redirect them to a file. For production runs at asupercomputing centre, you will usually have to put the above command into a script-file thatis submitted to the computing-queue. In this case, the standard output of GADGET-2 should be
automatically piped into a file.
2.3 Interrupting a run
Usually, a single submission to a queue system will not provide enough CPU-time to process
a big production run. Therefore, a running simulation can be interrupted after any timestep,and resumed at the very same place later on. If the CPU-time limit is specified correctly in theparameter file, the code will interrupt itself automatically before the CPU-time limit expires,and write a set of ‘restart’-files. Actually, each processor writes its own restart file. These
restart-files can be used to continue the run conveniently (see below).
Sometimes you may want to interrupt the code manually with the possibility to resume itlater on. This can be achieved by generating a file named stop in the output-directory of asimulation, e.g.
echo > /u/vrs/myoutput/stop
The code will then write a restart-file and terminate itself after the current timestep is completed, and the file ‘stop’ will be erased automatically. Restart files are also generated when
the last timestep of a simulation has been completed. They can be used if one later wants toextend the simulation beyond the original final time.
2.4 Restarting a run
2.4.1 Restarting from restart-files
To resume a run from restart-files, start the code with an optional flag in the form
mpirun -np 32 ./Gadget2 myparameterfile.param 1
This will continue the simulation with the set of restart files in the output-directory, with the5

V. Springel

latter being specified in the parameterfile. Restarting in this fashion is transparent to the code,i.e. the simulation behaves after the restart exactly as if had not been interrupted to begin with.
In GADGET-1, this was not the case in general, because a restart always forced the occurrenceof a new domain decomposition and tree construction right after the simulation was resumed.
When the code is started with the restart-flag, the parameterfile is parsed, but only some ofthe parameters are allowed to be changed, while any changes in the others will be ignored.
Which parameters these are is explained in Section 5 about the parameterfile.
It is important to not forget the 1 if you want to restart – otherwise the simulation will restartfrom scratch, i.e. by reading in the initial conditions again! Also note that upon restarting fromrestart-files, the number of processors used for a simulation cannot be changed. Also notethat restart files can in general not be transfered to another system, and that their structure iscompletely different than that of snapshot files. If you want to continue a simulation on anothersystem or with a different number of processors, restarting from a snapshot file is the methodof choice. This will be discussed next.

2.4.2 Restarting from snapshot-files
There are two possibilities to restart a simulation from a previous snapshot-file. In the firstpossibility, one simply adopts the snapshot file as new initial conditions. Note that this option requires changes in the parameterfile: You need to specify the snapshot-file as initialconditions-file, you also need to set TimeBegin (see below) to the correct time corresponding to the snapshot-file. In addition, you should change the base filename for the snapshot files,
since the counter for the outputs will start at 0 again, thereby possibly overwriting outputs you
might already have obtained. Once this is done, you can continue the run from the snapshot-file(without the optional 1).
An alternative to this somewhat inconvenient procedure is to restart the code with a flag equal
to 2 after the name of the parameterfile, i.e. just like above for the restart from restart-files, butwith the 1 replaced by 2. In this case, you only have to change the name of the initial conditions
file and make it equal to the name of the snapshot you want to start from, other parameters in
the parameterfile need not to be changed. In particular, the code will continue the numberingof all further snapshots starting with 1 plus the number of the snapshot that is used as input file.
When you specify a snapshot file that was distributed onto several files (e.g. dump 007.0 todump 007.7) you should only give the base-name (i.e. here dump 007) as initial conditions
file when using this option.
Note that the restart from snapshot files allows a change of the number of processors used
for the simulation. This is not possible if you restart from restart-files. However, restartingfrom restart-files is the preferred method to resume a simulation, because it is much faster,and it minimises possible perturbations in the time integration scheme of a running simulation(restarting from a snapshot forces the individual timestep scheme to be resynchronised). Because in general not all the particles are synchronised at the time of writing a snapshot file (theothers are first-order predicted to the output time), a small perturbation in the time integration
is introduced when restarting from a snapshot file.

6

User’s guide for GADGET-2

3 Types of simulations
Compared with GADGET-1, there are a number of new types of simulations that can be runwith GADGET-2, most importantly variants of the TreePM algorithm. A schematic overview ofthese simulation types is given in Table 1.
Cosmological integrations with comoving coordinates are selected via the parameter
ComovingIntegrationOn in the parameterfile. Otherwise the simulations always assume ordinary Newtonian space. Periodic boundary conditions and the various variants of
the TreePM algorithm require compile-time switches to be set appropriately in the makefile.
In particular, the TreePM algorithm is switched on by passing the desired mesh-size at compile time via the makefile to the code. The relevant parameter is PMGRID, see below. Usingan explicit force split, the long-range force is then computed with Fourier techniques, while
the short-range force is done with the tree. Because the tree needs only be walked locally, a
speed-up can arise, particularly for near to homogeneous particle distributions, but not only
restricted to them. Periodic and non-periodic boundary conditions are implemented for the
TreePM approach. In the latter case, the code will internally compute FFTs of size 2*PMGRID
in order to carry out the required zero-padding. For zoom-simulations, it is possible to let thecode automatically place a refined mesh-layer on the high-resolution region.
Pure SPH simulations in periodic boxes can also be run in periodic boxes whose dimensions
in each direction are multiples of the basic boxsize. Such simulations can only be done without
self-gravity at the moment.
Finally, it is possible to run SPH runs (also with self-gravity) in two dimensions only. However, the latter feature is provided for test-runs only, and is not optimised; here three coordinates
are still stored for all particles and the computations are formally carried out as in the 3D case,
except that all particles lie in one coordinate plane, i.e. either have equal x-, y-, or z-coordinates.

4 Makefile options
Many aspects of the new features of GADGET-2 are controlled with compile-time options in
the makefile rather than run-time options in the parameterfile. This was done in order to allow
the generation of highly optimised binaries by the compiler, even when the underlying source
allows for many different ways to run the code. Unfortunately, this technique has the disadvantage that different simulations may require different binary executables of GADGET-2. If
several simulations are run concurrently, there is hence the danger that a simulation is started orresumed with the ‘wrong’ binary. Note that while GADGET-2 checks the plausibility of someof the most important code options, this is not done for all of them. To minimise the risk ofusing the wrong code for a simulation, my recommendation is therefore to produce a separate
executable for each simulation that is run. A good strategy for doing this in practice is to makea copy of the whole simulation source code together with its makefile in the output directoryof each simulation run, and then use this copy to compile the code and to run the simulation.
The code and its settings become then a logical part of the output generated in the simulation
directory.

7

V. Springel

Type of Simulation

Computational
methods

Remarks

Newtonian space

Gravity: Tree, SPH(optional),
vacuumboundary conditions

2

Periodic long box

No gravity, only SPH,
periodic
boundary
conditions

3

Cosmological, physical coordinates

Gravity: Tree, SPH,
vacuum boundaries

set to zero4

Cosmological,
comoving coordinates

Gravity: Tree, SPH,
vacuum boundaries

set to one

5

Cosmological,
comoving
periodic
box

1

6

7

8

Cosmological,
comoving coordinates,
TreePM
Cosmological,
comoving periodic box,
TreePM
Cosmological,
comoving coordinates,
TreePM, Zoom

9

Cosmological, periodic comoving box,
TreePM, Zoom

10

Newtonian
TreePM

space,

Gravity: Tree with
Ewald-correction,
SPH, periodic boundaries
Gravity: Tree withlong range PM, SPH,
vacuum boundaries
Gravity: Tree withlong range PM, SPH,
periodic boundaries
Gravity: Tree withlong-range
andintermediate-range
PM, SPH, vacuum
boundaries
Gravity: Tree withlong-range
andintermediate-range
PM, SPH, periodic
boundaries
Gravity: Tree withlong-range PM, SPH,
vacuum boundaries

OmegaLambda

should be set to zeroNOGRAVITY needs tobe set, LONG X/Y/Z

may be set to scale thedimensions of the box
ComovingIntegrationOn

ComovingIntegrationOn

PERIODIC needs to be

set
PMGRID needs to be

set
PERIODIC and PMGRID need to be set

PMGRID and PLACEHIGHRESREGION

need to be set

PERIODIC, PMGRID
and PLACEHIGHRESREGION need to be set

PMGRID needs to be

set

Table 1: Schematic overview of the different types of simulations possible with GADGET-2.

8

User’s guide for GADGET-2

The makefile contains a dummy list of all available compile-time code options, with mostof them commented out by default. To activate a certain feature, the corresponding parametershould be commented in, and given the desired value, where appropriate. At the beginning of
the makefile, there is also a brief explanation of each of the options.
Important Note: Whenever you change one of the makefile options described below, a fullrecompilation of the code is necessary. For this reason, the makefile itself has been added to
the list of dependences of each source files, such that a complete recompilation should happen
automatically when the makefile is changed and the command make is given. A manual recompilation of the whole code can be enforced with the command make clean, which will
erase all object files, followed by make.
4.1 Basic operation mode of code

PERIODIC
Set this if you want to have periodic boundary conditions.

UNEQUALSOFTENINGS
Set this if you use particles with different gravitational softening lengths.
4.2 Things that are always recommended

PEANOHILBERTThis is a tuning option. When set, the code will bring the particles into
Peano-Hilbert order after each domain decomposition. This improvescache utilisation and performance.

WALLCLOCKIf set, a wallclock timer is used by the code to measure internal time consumption (see cpu-log file). Otherwise, a timer that measures consumed
processor ticks is used.
4.3 TreePM options

PMGRID=128
This enables the TreePM method, i.e. the long-range force is computed witha PM-algorithm, and the short range force with the tree. The parameter has
to be set to the size of the mesh that should be used, e.g. 64, 96, 128, etc.
The mesh dimensions need not necessarily be a power of two, but the FFT
is fastest for such a choice. Note: If the simulation is not in a periodic box,
then a FFT method for vacuum boundaries is employed, using a mesh withdimension twice that specified by PMGRID.

9V. Springel

PLACEHIGHRESREGION=1+8
If this option is set (will only work together with PMGRID), then the longrange force is computed in two stages: One Fourier-grid is used to cover
the whole simulation volume, allowing the computation of the large-scale
force. A second Fourier mesh is placed on the region occupied by ‘highresolution’ particles, allowing the computation of an intermediate-scale
force. Finally, the force on very small scales is computed by the tree. This
procedure can be useful for ‘zoom-simulations’, where the majority of particles (the high-res particles) are occupying only a small fraction of the volume. To activate this option, the parameter needs to be set to an integer that
encodes the particle types that make up the high-res particles in the form ofa bit mask. For example, if types 0, 1, and 4 are the high-res particles, thenthe parameter should be set to PLACEHIGHRESREGION=1+2+16, i.e. tothe sum 20 + 21 + 24 . The spatial region covered by the high-res grid is determined automatically from the initial conditions. Note: If a periodic box
is used, the high-res zone is not allowed to intersect the box boundaries.

ENLARGEREGION=1.1
The spatial region covered by the high-res zone normally has a fixed size
during the simulation, which initially is set to the smallest region that encompasses all high-res particles. Normally, the simulation will be interrupted if high-res particles leave this region in the course of the run. However, by setting this parameter to a value larger than one, the high-res region
can be expanded. For example, setting it to 1.4 will enlarge its side-lengthby 40% (it remains centred on the high-res particles). Hence, with such asetting, the high-res region may expand or move by a limited amount. If
in addition SYNCHRONIZATION is activated, then the code will be able
to continue even if high-res particles leave the initial high-res grid. In thiscase, the code will update the size and position of the grid that is placed
onto the high-resolution region automatically. To prevent that this potentially happens every single PM step, one should nevertheless assign a valueslightly larger than 1 to ENLARGEREGION.

ASMTH=1.25
This can be used to override the value assumed for the scale that defines thelong-range/short-range force-split in the TreePM algorithm. The default
value is 1.25, in mesh-cells.

RCUT=4.5
This can be used to override the maximum radius in which the short-range
tree-force is evaluated (in case the TreePM algorithm is used). The default
value is 4.5, given in mesh-cells.

10User’s guide for GADGET-2
4.4 Single/double precision

DOUBLEPRECISION
This makes the code store and compute internal particle data in double
precision. Note that output files are nevertheless written by converting thevalues that are saved to single precision.

DOUBLEPRECISION FFTW
If this is set, the code will use the double-precision version of FTTW, provided the latter has been explicitly installed with a ”d” prefix, and NOTYPEPREFIX FFTW is not set. Otherwise the single precision version(”s” prefix) is used.
4.5 Time integration options

SYNCHRONIZATION
When this is set, particles may only increase their timestep if the newtimestep will put them into synchronisation with the higher time level. Thistypically means that only on half of the timesteps of a particle an increaseof its step may occur. Especially for TreePM runs, it is usually advisable toset this option.

FLEXSTEPS
This is an alternative to SYNCHRONIZATION. Particle timesteps are here
allowed to be integer multiples of the minimum timestep that occurs among
the particles, which in turn is rounded down to the nearest power-of-two
devision of the total simulated time-span. This option distributes particles
more evenly over individual system timesteps, particularly once a simulation has run for a while, and may then result in a reduction of work-load
imbalance losses.

PSEUDOSYMMETRIC
When this option is set, the code will try to ‘anticipate’ timestep changes byextrapolating the change of the acceleration into the future. This can sometimes improve the long-term integration behaviour of periodic orbits, because then the adaptive integration becomes more akin to a time reversible
integrator. Use this option with care. Note: This option has no effect if
FLEXSTEPS is set.

NOSTOP WHEN BELOW MINTIMESTEP
If this is activated, the code will not terminate when the timestep falls below
the value of MinSizeTimestep specified in the parameterfile. This is usefulfor runs where one wants to enforce a constant timestep for all particles.
This can be done by activating this option, and by setting MinSizeTimestep
and MaxSizeTimestep to an equal value.

11

V. Springel

NOPMSTEPADJUSTMENT
When this is set, the long-range timestep for the PM force computationis always determined by MaxSizeTimeStep. Otherwise, it is set to theminimum of MaxSizeTimeStep and the timestep obtained for the maximum long-range force with an effective softening scale equal to the PMsmoothing-scale.
4.6 Output options

HAVE HDF5
If this is set, the code will be compiled with support for input and output
in the HDF5 format. You need to have the HDF5 libraries and headers
installed on your computer for this option to work. The HDF5 format canthen be selected as format ”3” in Gadget’s parameterfile.

OUTPUTPOTENTIAL
This will force the code to compute gravitational potentials for all particles
each time a snapshot file is generated. These values are then included in
the snapshot files. Note that the computation of the values of the potential
costs additional time.

OUTPUTACCELERATIONThis will include the physical acceleration of each particle in snapshot files.

OUTPUTCHANGEOFENTROPYThis will include the rate of change of entropy of gas particles in snapshot
files.

OUTPUTTIMESTEPThis will include the timesteps actually taken by each particle in the snapshot files.
4.7 Things for special behaviour

NOGRAVITYThis switches off gravity. Makes only sense for pure SPH simulations in
non-expanding space.

NOTREERND
If this is not set, the tree construction will succeed even when there are a few
particles at identical locations. This is done by ‘rerouting’ particles once
the node-size has fallen below 10−3 of the softening length. However, when
this option is activated, this will be suppressed and the tree construction will
always fail if there are particles at identical or extremely close coordinates.

12

User’s guide for GADGET-2

NOTYPEPREFIX FFTW
If this is set, the fftw-header/libraries are accessed without type prefix(adopting whatever was chosen as default at compile-time of fftw). Otherwise, the type prefix ’d’ for double-precision is used.

LONG X/Y/Z
These options can only be used together with PERIODIC and NOGRAVITY. When set, the options define numerical factors that can be used to
distort the periodic simulation cube into a parallelepiped of arbitrary aspectratio. This can be useful for idealised SPH tests.

TWODIMS
This effectively switches of one dimension in SPH, i.e. the code follows
only 2d hydrodynamics in the xy-, yz-, or xz-plane. This only works withNOGRAVITY and if all coordinates of one of the axis are exactly equal.
Can be useful for idealised SPH tests.

SPH BND PARTICLES
If this is set, particles with a particle-ID equal to zero do not receive any
SPH acceleration. This can be useful for idealised SPH tests, where theseparticles represent fixed ‘walls’.

NOVISCOSITYLIMITER
If this is set, there is no explicit upper limit on the viscosity. In the default version, this limiter will try to protect against possible particle ‘reflections’, which may in principle occur for poor timestepping in the presenceof strong shocks.

COMPUTE POTENTIAL ENERGYWhen this option is set, the code will compute the gravitational potential
energy each time a global statistics is computed. This can be useful for
testing global energy conservation.

ISOTHERM EQSThis special option makes the gas behave like an isothermal gas with equation of state P = c2s ρ. The sound-speed squared, c2s , is set instead of thethermal energy per unit mass in the initial conditions, i.e. you need to specify c2s in the block that is usually reserved for u. The generated snapshot
files will also contain the squared sound speed in this field. If the initialvalue for u is zero, then the initial gas temperature in the parameter file isused to define the sound speed according to c2s = kT /mp , where mp is the
proton mass.

13

V. Springel

ADAPTIVE GRAVSOFT FORGAS
When this option is set, the gravitational softening lengths used for gas
particles is tied to their SPH smoothing length. This can be useful for dissipative collapse simulations. The option requires the setting of UNEQUALSOFTENINGS.

SELECTIVE NO GRAVITY
This can be used for special computations where one wants to exclude certain particle types from receiving gravitational forces. The particle types
that are excluded in this fashion are specified by a bit mask, in the same
way as for the PLACEHIGHRESREGION option.

LONGIDS
If this is set, the code assumes that particle-IDs are stored as 64-bit longintegers. This is only really needed if you want to go beyond ∼ 2 billion
particles.
4.8 Testing and debugging options

FORCETEST=0.01
This can be set to check the force accuracy of the code, and is only included
as a debugging option. The option should be set to a number between 0 and
1 (e.g. 0.01), which specifies the fraction of randomly chosen particles
for which at each timestep forces by direct summation are computed. The
normal tree-forces and the exact direct summation forces are then collected
in a file forcetest.txt for later inspection. Note that the simulation
itself is unaffected by this option, but it will of course run much(!) slower,
particularly if FORCETEST*NumPart*NumPart >> NumPart. Note:
Particle IDs must be set to numbers >=1 for this option to work.
4.9 Glass making

MAKEGLASS=262144
This option can be used to generate a glass-like particle configuration. The
value assigned gives the particle load, which is initially generated as a Poisson sample and then evolved towards a glass with the sign of gravity reversed.

5 Parameterfile
Many code features are controlled by a parameterfile that has to be specified whenever thecode is started. Each parameter value is set by specifying a keyword, followed by a numericalvalue or a character string, separated by whitespace (spaces, tabs) from the keyword. For eachkeyword, a separate line needs to be used, with the value and keyword appearing on the same

14

User’s guide for GADGET-2

line. Between keywords, arbitrary amounts of whitespace (including empty lines) may be used.
The sequence of keywords is arbitrary, but each keywords needs to appear exactly once, even
if its value is not relevant for the particular simulation (otherwise you’ll get an error message).
Note that the keywords are type-sensitive.
Lines with a leading ‘%’ are ignored. In lines with keywords, comments may also be added
after the specification of the value for the corresponding keyword.
In the following, each keyword and the meaning of its value are discussed in detail. Keywords marked with ? may be changed during a run, i.e. changes of the corresponding values
will be taken into account upon starting the code with the restart-flag, but any changes of otherparameter values will be ignored. Note that you normally don’t need to make any changes in
the parameterfile when you restart a run from restart-files.

5.1 Filenames and file formats

OutputDir?

/home/volker/galaxy collisionThis is the pathname of the directory that holds all the output generated by
the simulation (snapshot files, restart files, diagnostic files). Note that thisdirectory needs to exist, otherwise an error message will be produced.

SnapshotFileBase?

snapshotFrom this string, the name of the snapshot files is derived by adding
an underscore, and the number of the snapshot in a 3-digits format. If
NumFilesPerSnapshot>1, each snapshot is distributed into several
files, where a group of processors writes simultaneously to a file. In thiscase, the filenames are also supplemented with a tailing .n, where n designates the number of the file in the group of files representing the snapshot.

SnapFormat?

1

A flag that specifies the file-format to be used for writing snapshot files. A
value of 1 selects the standard file-format of GADGET-2 (which is compatible with the format used in GADGET-1), while a value of 2 selects a moreconvenient variant of this simple binary format. A value of 3 selects the useof HDF5 instead. The structure of these files will be discussed in a separate section below. Other values for SnapFormat could be used to selectcustomised file-formats if you implement appropriate modifications and/or
extensions of the code.

15

V. Springel

NumFilesPerSnapshot?

2

The code can distribute each snapshot onto several files.
This
leads to files that are easier to handle in case the simulation isvery large, and also speeds up I/O, because these files can be written in parallel. The number of processors must be equal or larger
than NumFilesPerSnapshot, because each snapshot file will holdthe data of a group of processors.
Optimum I/O throughput isreached if the number of processors is equal to, or a multiple of
NumFilesPerSnapshot, and if NumFilesWrittenInParallelis equal to NumFilesPerSnapshot or a subdivision of it. With
NumFilesPerSnapshot=1 it is possible to write all particle data injust one snapshot file.

InitCondFile/home/volker/ICs/galaxy.dat

This sets the filename of the initial conditions to be read in at start-up. Notethat the initial conditions file (or files) does not have to reside in the output
directory. The initial conditions can be distributed into several files, in thesame way as snapshot files. In this case, only the basename without thetailing .n number should be specified as initial conditions filename. The
code will recognise the number of files that make up the initial conditions
from the header entries, and load all of these files accordingly. There is noconstraint for the number of these files in relation to the processor number.

ICFormat

1

This flag selects the file format of the initial conditions read in by thecode upon start-up. The possible format choices are the same as for
SnapFormat. It is therefore possible to use different formats for the initial conditions and the produced snapshot files. We recommend that you
convert your IC files to one of the formats of GADGET-2. Alternatively,
you could incorporate a customised reading routine directly into GADGET2, but this requires intimate understanding of the internal workings of thecode.

RestartFile?

restartThis is the base filename of the restart files. For the given example parameter value, each processor will write a restart file restart.N, where N is
the rank number of the processor.

InfoFile?

info.txt

This is the name of a file which will hold a list of all the timesteps. The lastentry always holds the timestep that is currently processed. For a runningsimulation, the command ‘tail info.txt’ will then inform you about
the current status of the simulation. The meaning of the different entries in
the file is discussed in detail later on.

16

User’s guide for GADGET-2

TimingsFile?

timings.txt

This is the name of a log-file that measures various aspects of the performance of the gravitational force computation for each timestep. The meaning of the individual entries will be described later on.

CpuFile?

cpu.txt

This is a log-file that keeps track of the cumulative cpu-consumption ofvarious parts of the code. At each timestep, one line is added to this file.
The meaning of the individual entries will be described later on. A detailed
analysis of them allows an assessment of the code performance, and an
identification of potential bottlenecks.

EnergyFile?

energy.txt

This is the name of a file where information about global energy statistics ofthe simulation is stored. Checking energy conservation can sometimes be agood proxy of time integration accuracy, but it is not a particularly stringent
test. Because computing the gravitational energy requires extra CPU time,it has to be explicitly enabled with the COMPUTE POTENTIAL ENERGY
makefile option. Note also that an accurate computation of the peculiar
gravitational binding energy at high redshift is quite subtle, so that checking for energy conservation is not a good test to evaluate the accuracy of
cosmological integrations.
5.2 CPU-time limit and restart options

TimeLimitCPU?

40000.0

This is the CPU-time limit for the current run (one submission to the computing queue) in seconds. This value should be matched to the corresponding limit of the queueing system, if appropriate. The run will automatically interrupt itself and write a restart file, if 85% of this time has been
consumed. The extra 15% is introduced to guarantee that there is always
enough time left to safely finish the current time step and write the restart
file. Note that this CPU time refers to the wall-clock time on one processor
only.

ResubmitCommand?

xyz.sh

If ResubmitOn=1, the code will execute the given command (xyz.shin this example) when it interrupts the run because the CPU-time limit isabout to be exhausted. Usually, xyz.sh should be set to the name ofan executable script in which one places the necessary commands for resubmission of the job (with the 1-flag set for restart) to the computing
queue. Note that this feature should be used with care – it is always advisable to supervise a running simulation closely.

17

V. Springel

ResubmitOn?

0This flag can be used to enable the automatic re-submission feature of thecode (which is switched on for ResubmitOn=1).

CpuTimeBetRestartFile?

7200

This is the maximum amount of CPU-time (wall-clock time, in seconds)
that may be used by the code before it writes a new restart file. With thisparameter the code can hence be asked to write a restart file every once in awhile. This is meant to provide a precautionary measure against hardware
or software failures, in which case one can resume a simulation from thelast set of restart files. In the above example, a restart file would be written
automatically every 2 hours. The old set of restart files is renamed into
a set of .bak files before the new files are written, so that there is some
protection against a crash during the writing of the restart files itself (quite
typically, this may happen due to disk-full errors, for example).

5.3 Simulation specific parameters

TimeBegin

0

This initialises the time variable of the simulation when a run is started frominitial conditions (in internal units). If comoving integration is selected(ComovingIntegrationOn= 1), the time variable is the dimensionless expansion factor a itself, i.e. TimeBegin= a = 1/(1 + z start ).

TimeMax?

3.0This marks the end of the simulation. The simulation will run up to thispoint, then write a restart-file, and a snapshot file corresponding to thistime (even if the time TimeMax is not in the normal sequence of snapshot
files). If TimeMax is increased later on, the simulation can be simply
continued from the last restart-file. Note that this last snapshot-file will thenbe overwritten, since it was a special dump out of the normal sequence.
For comoving integrations, the time variable is the expansion factor, e.g.
TimeMax= 1.0 will stop the simulation at redshift z = 0.

BoxSize

10000.0

The size of the periodic box (in code units) encompassing the simulation
volume. This parameter is only relevant if the PERIODIC makefile option
is used. Particle coordinates need to lie in the interval [0, BoxSize] inthe initial conditions file.

18

User’s guide for GADGET-2

PeriodicBoundariesOn

1

Enables or disables the use of periodic boundaries. The code needs to becompiled with the PERIODIC makefile option for periodic boundaries,
otherwise it will complain and stop. (Actually, this parameter may beviewed as redundant given that the same option is specified in the makefile, but it is kept to make sure that you run your simulation always withthe ‘right’ executable.) When you use periodic boundary conditions for the
first time, GADGET-2 will compute correction tables for the periodic forces
and the periodic peculiar potential by means of Ewald summation (done inparallel). As this computation takes a bit of time, these tables will be stored
on disk to speed-up the start of the code in future runs.

ComovingIntegrationOn

0This flag enables or disables comoving integration in an expanding universe. For ComovingIntegrationOn=0, the code uses plain Newtonian physics with vacuum or periodic boundary conditions. Time,positions, velocities, and masses are measured in the internal system of units, as specified by the selected system of units.
For
ComovingIntegrationOn=1, the integration is carried out in an expanding universe, using a cosmological model as specified by Omega0,
OmegaLambda, etc. In this cosmological mode, coordinates are comoving, and the time variable is the expansion factor itself. If the code has notbeen compiled with the PERIODIC makefile option, the underlying model
makes use of vacuum boundary conditions, i.e. density fluctuations outside
the particle distribution are assumed to be zero. This requires that your particle distribution represents a spherical region of space around the origin.
If PERIODIC is enabled, the code expects the particle coordinates to lie in
the interval [0, BoxSize].
5.4 Cosmological parameters

HubbleParam

0.7

Value of the Hubble constant in units of 100 km s−1 Mpc−1 . This is onlyused in cosmological integrations when conversions to cgs units are required (e.g. for radiative cooling physics). Note however that the value of
HubbleParam is not needed (in fact, it does not enter the computation atall) in purely collisionless simulations. This is because of the definition of
GADGET-2’s system of units, which eliminates an explicit appearance ofthe value of the Hubble constant in the dynamical equations.

Omega00.3

Cosmological matter density parameter in units of the critical density at
z = 0. Relevant only for comoving integration.

19

V. Springel

OmegaLambda

0.7

Cosmological vacuum energy density (cosmological constant) in units ofthe critical density at z = 0. Relevant only for comoving integration. Fora geometrically flat universe, one has Omega0 + OmegaLambda = 1.
For simulations in Newtonian space, this parameter has to be set to zero.

OmegaBaryon

0.04

Baryon density in units of the critical density at z = 0. This is not explicitly
used in the public version of GADGET-2, but since this paramater may be
useful for the implementation of certain physical processes (as it is done in
the full-physics version of GADGET-2), it has been kept in the code.

5.5 Memory allocation

BufferSize?

15

This specifies the size (in MByte) of a multi-purpose communication bufferused by the code in various parts of the parallel algorithms, for example
during the force computation and the domain decomposition. The buffer
should be large enough to accommodate a ‘fair’ fraction of the particle mix
in order to minimise work-load imbalance losses. In practice, sizes between
a few to 100 MB offer enough room. Upon start-up, the code informs about
how many particles can be fitted into the communication structures during
the various parts of the code that make use of them. There is no problem
if a small BufferSize is used, except that it can lead to somewhat lower
overall performance.

PartAllocFactor?

1.1

Each processor allocates space for PartAllocFactor times the average number of particles per processor. This number needs to be larger
than 1 to allow the simulation to achieve a good work-load balance inthe domain decomposition, which requires that some particle-load imbalance is accepted for better work-load balance. The specified value for
PartAllocFactor acts as a ceiling for the maximum allowed memory
imbalance, and the code will observe this limit in the domain decomposition, trying to achieve the best work-load balance within this bound. It isgood to make PartAllocFactor as large as possible for the available
memory per processor, but values in excess of 3 will usually not improve
performance any more. The recommended minimum value is ∼ 1.05, but
practical choices should be more like 1.5-2.0. For a value of 1.0 the code
will not be able to succeed in the domain decomposition and crash, unlessa single CPU is used.

20

User’s guide for GADGET-2

TreeAllocFactor?

0.7

To construct the BH-tree for N particles, somewhat less than Ninternal tree-nodes are necessary for ‘normal’ particle distributions.
TreeAllocFactor sets the number of internal tree-nodes allocatedin units of the particle number. By experience, space for ' 0.65 Ninternal nodes is usually fully sufficient for typical clustered particle
distributions, so a value of 0.7 should put you on the safe side. If
the employed particle number per processor is very small (less than athousand or so), or if there are many particle pairs with identical ornearly identical coordinates, a higher value may be required. Since thenumber of particles on a given processor may be higher by a factor
PartAllocFactor than the average particle number, the total amount ofmemory requested for the BH tree on a single processor scales proportional
to PartAllocFactor*TreeAllocFactor.

5.6 Gravitational force accuracy

TypeOfOpeningCriterion?

0This selects the type of cell-opening criterion used in the tree walks forcomputing gravitational forces. A value of ‘0’ results in standard Barnes& Hut, while ‘1’ selects a relative criterion that tries to limit the absolute
truncation error of the multipole expansion for every particle-cell interaction. This scheme gives usually higher accuracy at a comparable level ofcomputational cost, something that makes it more costly at high redshift
because of the small residual forces there.

ErrTolTheta

0.7

This is the accuracy criterion (the opening angle θ) of the tree algorithm if the standard Barnes & Hut (BH) opening criterion is used. If
TypeOfOpeningCriterion=1, θ and the BH criterion are only used
for the first force computation. These first forces are only used as input fora recomputation of the forces with the relative cell opening criterion, whichis employed thereafter for the rest of the force computations.

21

V. Springel

ErrTolForceAcc?

0.005

This controls the accuracy of the relative cell-opening criterion (if enabled), where a cell is opened if M l 2 > α|aold |r4 . Here, α is set by
ErrTolForceAcc, M is the mass inside a node, l is its cell side-length,
and aold is the magnitude of the total acceleration of the particle. The latter
is usually estimated based on the gravitational force on the particle in theprevious timestep. Note that independent of this relative criterion, the code
will always open nodes if the point of reference lies within the geometricboundaries of the cubical cell (enlarged by an additional 10% on each side).
This protects against the rare occurrence of very large force errors, which
would otherwise be possible.

5.7 Time integration accuracy

MaxSizeTimestep?

0.01

This parameter sets the maximum timestep a particle may take. This needs
to be set to a sensible value in order to protect against too large timesteps
for particles with very small acceleration. Usually, a few percent up totens of percent of the dynamical time of the system gives sufficient accuracy. For cosmological simulations, the parameter MaxSizeTimestep
has undergone an important change in its definition. Formerly, the parameter was specifying the maximum allowed step in terms of max(∆a), wherea is the scale factor. Now, the parameter specifies max(∆ ln a), where ln
is the natural logarithm. The new definition is equivalent to specifying the
maximum allowed timestep for cosmological simulations as a fraction of
the current Hubble time. A value of ' 0.025 is usually accurate enough formost cosmological runs.

MinSizeTimestep?

0

If a particle requests a timestep smaller than the specified value, the simulation is terminated with a warning message. This can be used to prevent simulations to continue when the timestep has dropped to a very small value,
because such behaviour typically indicates a problem of some sort. However, if NOSTOP WHEN BELOW MINTIMESTEP is set in the Makefile, thetimestep is forced instead to be at least MinSizeTimestep, even if thismeans that the other timestep criteria are ignored. This option should hence
be used with care. It can be useful however to force equal and constanttimesteps for all particles.

22

User’s guide for GADGET-2

TypeOfTimestepCriterion?

0This selects the type of timestep criterion employed by the code for collisionless dynamics. In GADGET-1, there have been 5 possible criteria
for this. However, upon systematic testing with high-resolution simulations, none of them showed a clearly superior performance compared withthe ‘standard’ criterion used traditionally in cosmological simulations (seethe paper by Powers et al., 2003, MNRAS, 338, 14). Due to this result, the choice of timestep criteria in GADGET-2 has been reduced again,
and only the standard criterion is offered at present. So the only possible
value for TypeOfTimestepCriterion is ‘0’, which selects pa timestep√∝ 1/ a, or more precisely, the timestep is given by ∆t = 2η²/|a|,where η =ErrTolIntAccuracy and ² is the gravitational softening
length.

ErrTolIntAccuracy?

0.025

This dimensionless parameter controls the accuracy of the timestep criterion selected by TypeOfTimestepCriterion.

TreeDomainUpdateFrequency?

0.05

A domain decomposition and a full tree construction are not necessarily
carried out every single timestep. Instead, properties of tree nodes can be
dynamically updated as needed. This can improve the performance of thecode if there are many timesteps that advance only a very small fractionof particles (below a percent or so). This parameter determines how often
the domain decomposition and the full tree are reconstructed from scratch.
A value of TreeUpdateFrequency= 0.05, for example, means thatthe domain decomposition and the tree are reconstructed whenever there
have been at least 0.05 N force computations since the last reconstruction,
where N is the total particle number. A value of zero for this parameter
will reconstruct the tree every timestep.

MaxRMSDisplacementFac?

0.25

This defines an additional timestep limiter for the PM-step when the
TreePM method is used. In this case, the code computes the 3D velocity dispersion σ for all particles in the simulation, and restricts the maximum simulation timestep (and hence the PM timestep) such that σ∆t isless than MaxRMSDisplacementFac times the mean particle separation d, or less than MaxRMSDisplacementFac times the scale r s of theshort-range/long-range force split, whichever is smaller.

23

V. Springel
5.8 Output of snapshot files

OutputListOn?

0

A value of 1 signals that the output times are given in the file specified by
OutputListFilename. Otherwise output times are generated automatically in the way described below.

OutputListFilename?

output times.txt

This specifies the name of a file that contains a list of desired output times.
If OutputListOn is set to 1, this list will determine the times whensnapshot-files are produced. The file given by OutputListFilename
should just contain the floating point values of the desired output times inplain ASCII format. The times do not have to be ordered in time, but there
may be at most 350 values. Output times that are in the past relative to thecurrent simulation time will be ignored.

TimeOfFirstSnapshot

0.047619048

This variable selects the time for the first snapshot (ifOutputListOn=0). For comoving integration, the above choice
would therefore produce the first dump at redshift z = 20.

TimeBetSnapshot?

1.0627825

After a snapshot has been written, the time for the next snapshot is determined by either adding TimeBetSnapshot
to
TimeOfFirstSnapshot,orbymultiplying
TimeOfFirstSnapshot with TimeBetSnapshot.
The latteris done for comoving integration, and will hence lead to a series of outputs
that are equally spaced in log(a). The above example steps down toredshift z = 0 in 50 logarithmically spaced steps. Note however that if
OutputListOn=1 this parameter is ignored.

TimeBetStatistics?

0.1This determines the interval of time between two subsequent computationsof the total potential energy of the system. This information is then written
to the file given by EnergyFile, together with information about the
kinetic energies of the different particle types. A first energy statistics isalways produced at the start of the simulation at TimeBegin.

24

User’s guide for GADGET-2

NumFilesWrittenInParallel?

16

This parameter determines how many files may be written (or read) simultaneously by the code when generating (reading) snapshot files, orwhen writing (reading) of restart files. Because each processor writes
its own restart file, it is in principle possible to do this fully in parallel,with each processor writing ‘his’ file at the same time. However, for big
runs, say with 512 processors, a very large number of simultaneous I/O requests for large files may cause severe problems for the operating system,
since the number of available disk drives on the file server will likely besmaller than the number of processors. As a result, the I/O system may
get congested because these requests cannot all be served efficiently at
the same time. In order to avoid such a congestion and to reach an optimum overall throughput of the I/O subsystem, it is often better not towrite too many files simultaneously. This can be achieved with the parameter NumFilesWrittenInParallel, which restricts the number offiles GADGET-2 processes concurrently during restart or snapshot file reading/writing. Note that NumFilesWrittenInParallel must be equal
or smaller than the number of processors used.
5.9 System of units

UnitVelocity in cm per s

1e5This sets the internal velocity unit in cm/sec. The above choice
is convenient – it sets the velocity unit to km/sec.
Note thatthe specification of UnitLength in cm, UnitMass in g, and
UnitVelocity in cm per s also determines the internal unit of time.
The definitions made above imply that in internal units the Hubble constant
has a numerical value independent of h and hence HubbleParam. For
the numerical examples above, the Hubble constant has always the value
0.1 in internal units, independent of h, and the Hubble time is always 10 ininternal units, with one internal time unit corresponding to 9.8 × 10 8 yr/h.
However, you are free to choose a different system of units if you like.
Note also that this implies that for purely gravitational dynamics, the code
will not need to know the value of h at all. HubbleParam is kept in
the parameterfile only because additional physics in the hydrodynamical
sector (which is not contained in the public distribution of GADGET-2) may
require it.

UnitLength in cm

3.085678e21

This sets the internal length unit in cm/h, where H0=
100 h km s−1 Mpc−1 . The above choice is convenient – it sets thelength unit to 1.0 kpc/h.

25

V. Springel

UnitMass in g

1.989e43This sets the internal mass unit in g/h, where H0 = 100 h km s−1 Mpc−1 .
The above choice is convenient – it sets the mass unit to 10 10 M¯ /h.

GravityConstantInternal0

The numerical value of the gravitational constant G in internal
units depends on the system of units you choose.
For example,
for the numerical choices made above, G = 43007.1 in internal
units. For GravityConstantInternal=0 the code calculates thevalue corresponding to the physical value of G for you.
Sometimes, you might want to set G yourself. For example, by specifying GravityConstantInternal=1, UnitLength in cm=1,
UnitMass in g=1, and UnitVelocity in cm per s=1, one obtains a ‘natural’ system of units. Note that the code will neverthelesstry to use the ‘correct’ value of the Hubble constant even in this case, so
you should always set GravityConstantInternal=0 in cosmological integrations.
5.10 SPH parameters

DesNumNgb

64

This is the desired number of SPH smoothing neighbours. Normally, the effective number of neighbours (defined as the mass inside the kernel divided
by the particle mass) is kept constant very close to this value. Should it ever
try to get outside a range ±MaxNumNgbDeviation from DesNumNgb,
the code will readjust the smoothing length such that the number of neighbours is again in this range.

MaxNumNgbDeviation?

2

This sets the allowed variation of the number of neighbours around thetarget value DesNumNgb.

ArtBulkViscCons?

1.0

This sets the value of the artificial viscosity parameter αvisc used by
GADGET-2. See code paper for details.

CourantFac?

0.15

This sets the value of the Courant parameter used in the determination of
the hydrodynamical timestep of SPH particles. Note that GADGET-2’s definition of the SPH smoothing length differs by a factor of 2 from that foundin large parts of the SPH literature. As a consequence, comparable settingsof CourantFac are a factor of 2 smaller in GADGET-2 when comparedwith codes using the different convention.

26

User’s guide for GADGET-2

InitGasTemp

10000

This sets the initial gas temperature in Kelvin when initial conditions are
read. If the temperature is below 104 K, a mean molecular weight corresponding to neutral gas of primordial abundance is assumed, otherwise
complete ionisation is assumed. However, the gas temperature is only set
to a certain temperature if InitGasTemp> 0 and if at the same time thetemperature of the gas particles in the initial conditions file is zero, otherwise the initial gas temperature is left at the value stored in the IC file.

MinGasTemp

20

The minimum allowed gas temperature in Kelvin (this is converted by thecode to a minimum thermal energy per unit mass assuming the mean molecular weight of neutral gas). This may be set to zero in principle, but often
it is desirable to prevent the gas from becoming too cold, e.g. for resolution
reasons or because of lower limits in the implemented cooling function.

MinGasHsmlFractional

0.1This parameter sets the minimum allowed SPH smoothing length in unitsof the gravitational softening length of the gas particles. The smoothing length will be prevented from falling below this value. When thisbound is actually reached, the number of smoothing neighbours will instead be increased above DesNumNgb, even if this means that the upper
limit DesNumNgb+MaxNumNgbDeviation is exceeded. The parameter MinGasHsmlFractional can be zero, allowing the smoothing radius to vary without bounds, which is usually adequate.
5.11 Gravitational softening
The code distinguishes between different particle types. Each type may have a different gravitational softening. As far as gravity is concerned, all the types are treated equivalently by thecode, the names ‘Gas’, ‘Halo’, ‘Disk’, ‘Bulge’, ‘Stars’, and ‘Bndry’, are just arbitrary tags,
still reflecting GADGET-2’s origin as a tool to study colliding galaxies. However, the particlesof the first type (‘Gas’) are indeed treated as SPH particles, i.e. they receive an additional hydrodynamic acceleration, and their internal entropy per unit mass is evolved as independent
thermodynamic variable.
Gravity is softened with a spline, as outlined in the code paper. The softenings quoted here
all refer to ², the roughly equivalent Plummer softening length. Note that for the spline that is
actually used, the force will be exactly Newtonian beyond h = 2.8². The softening lengths are
in internal length units. For comoving integration, the softening refers to the one employed incomoving coordinates, which usually stays fixed during the simulation. However, employing
the *MaxPhys parameters described below the code can also be asked to switch to a fixedsoftening in physical scale below a certain redshift.
If particle types with different softenings are used, the UNEQUALSOFTENINGS option in27

V. Springel

the makefile needs to be set. Unlike in GADGET-1, the present code stores all particle types in asingle tree, and this option tells the code to mark tree nodes that contain particles with different
softenings in a special way.
In cosmological simulations, one sometimes wants to start a simulation with a softening ² comthat is fixed in comoving coordinates (where the physical softening, ² phys = a²com , then grows
proportional to the scale factor a), but at a certain redshift one wants to freeze the resulting
growth of the physical softening ²max
phys at a maximum value. These maximum softening lengths
are specified by the Softening*MaxPhys parameters. In the actual implementation, thecode uses ²0com = min(²com , ²max
phys /a) as comoving softening. Note that this feature is only
enabled for ComovingIntegrationOn=1, otherwise the *MaxPhys values are ignored.

SofteningGas

0

SofteningGas specifies the (comoving) softening of the first particle
group.

SofteningHalo

18

This specifies the (comoving) softening of the second particle group.

SofteningDisk

90

This specifies the (comoving) softening of the third particle group.

SofteningBulge

450

This specifies the (comoving) softening of the fourth particle group.

SofteningStars0This specifies the (comoving) softening of the fifth particle group.

SofteningBndry

0This specifies the (comoving) softening of the sixth particle group.

SofteningGasMaxPhys

0This specifies the maximum physical softening of the first particle group.

SofteningHaloMaxPhys

3

This specifies the maximum physical softening of the second particle group.

SofteningDiskMaxPhys

15

This specifies the maximum physical softening of the third particle group.

SofteningBulgeMaxPhys

75

This specifies the maximum physical softening of the fourth particle group.

SofteningStarsMaxPhys

0This specifies the maximum physical softening of the fifth particle group.

28

User’s guide for GADGET-2

SofteningBndryMaxPhys

0This specifies the maximum physical softening of the sixth particle group.

6 File formats of GADGET-2: Snapshots & Initial conditions
The primary result of a simulation with GADGET-2 are snapshots, which are simply dumps ofthe state of the system at certain times. GADGET-2 supports parallel output by distributing asnapshot into several files, each written by a group of processors. This procedure allows aneasier handling of very large simulations; instead of having to deal with one file of size of a few
GB, say, it is much easier to have several files with a smaller size of a few hundred MB instead.
Also, the time spent for I/O in the simulation code can be reduced if several files are written inparallel.
Each particle dump consists of k files, where k is given by NumFilesPerSnapshot. For
k > 1, the filenames are of the form snapshot XXX.Y, where XXX stands for the numberof the dump, Y for the number of the file within the dump. Say we work on dump 7 withk = 16 files, then the filenames are snapshot 007.0 to snapshot 007.15. For k = 1,
the filenames will just have the form snapshot XXX. The base “snapshot” can be changed
by setting SnapshotFileBase to another value.
Each of the individual files of a given set of snapshot files contains a variable number of particles. However, the files all have the same basic format (this is the case for all three fileformats
supported by GADGET-2), and all of them are in binary. A binary representation of the particle
data is our preferred choice, because it allows much faster I/O than ASCII files. In addition, theresulting files are much smaller, and the data is stored loss-less.
6.1 Structure of the default snapshot file format
In the default file format of GADGET-2 (selected as fileformat 1 for SnapFormat), the datais organised in blocks, each containing a certain information about the particles. For example,
there is a block for the coordinates, and one for the temperatures, etc. A list of the blocks
defined in the public version of GADGET-2 is given in Table 2. The first block has a specialrole, it is a header which contains global information about the particle set, for example thenumber of particles of each type, the number of files used for this snapshot set, etc.
Within each block, the particles are ordered according to the particle type, i.e. gas particles
will come first (type 0), then ‘halo’ particles (type 1), followed by ‘disk’ particles (type 2), andso on. The correspondence between type number and symbolic type name is given in Table 3.
However, it is important to realize that the detailed sequence of particles within the blocks will
still change from snapshot to snapshot. Also, a given particle may not always be stored in thesnapshot file with the same number among the files belonging to one set of snapshot files. Thisis because particles may move around from one processor to another during the coarse of aparallel simulation. In order to trace a particle between different outputs, one therefore has toresort to the particle IDs, which can be used to label particles uniquely.

29

V. Springel
To allow an easy access of the data also in Fortran2 , the blocks are stored using the ‘unformatted binary’ convention of most Fortran implementations. In it, the data of each read or
write statement is bracketed by block-size fields, which give the length of the data block inbytes. These block fields (which are two 4-byte integers, one stored before the data block andone after) can be useful also in C-code to check the consistency of the file structure, to makesure that one reads at the right place, and to skip individual blocks quickly by using the length
information of the block size fields to fast forward in the file without actually having to read thedata. The latter makes it possible to efficiently read only certain blocks, say just temperaturesand densities, but no coordinates and velocities.
Assuming that variables have been allocated/declared appropriately, a possible read-statementof some of these blocks in Fortran could then for example take the form:
read (1) npart, massarr, a, redshift, flagsfr, flagfeedb, nall
read (1) pos
read (1)read (1) id
read (1) masses

In this example, the block containing the velocities, and several fields in the header, would
be skipped. Further read-statements may follow to read additional blocks if present. In thefile read snapshot.f, you can find a simple example of a more complete read-in routine.
There you will also find a few lines that automatically generate the filenames, and further hints
how to read in only certain parts of the data.
When you use C, the block-size fields need to be read or skipped explicitly, which is quite
simple to do. This is demonstrated in the file read snapshot.c. Note that you need togenerate the appropriate block field when you write snapshots in C, for example when generating initial conditions for GADGET-2. Unlike GADGET-1, GADGET-2 will check explicitlythat the block-size fields contain the correct values and refuse to start if not. In the Analysis
directory, you can also find an example for a read-in routine in IDL.
The sequence of blocks in a file is given in Table 2. Note however that not all of the blocks
need to be present, depending on the type of the simulation, and the makefile options of thecode. For example, if there are no gas particles in the snapshot file, the blocks for internalenergy or density will be absent. The presence of the mass block depends on whether or notparticle masses are defined to be constant for certain particle types by means of the Massarr
table in the file header. If a non-zero mass is defined there for a certain particle type, particlesof this type will not be listed with individual masses in the mass block. If such fixed particle
masses are defined for all types that are present in the snapshot file, the mass block will beabsent.
The detailed structure of the fields of the file header is given in Table 4. Note that the header
is filled to a total length of 256 bytes with unused entries, leaving space for extensions that canbe easily ignored by legacy code. Also note that a number of the flags defined in the header are
2

This should not be misunderstood as an encouragement to use Fortran.

30

User’s guide for GADGET-2

#
1
2
3
4
5

Block ID

HDF5 identifier

HEAD
POS
VEL
ID
MASS

Coordinates
Velocities
ParticleIDsMasses6

U

InternalEnergy

7
8

RHO
HSML

Density
SmoothingLength

9

POT

Potential

10

ACCE

Acceleration

11

ENDT

RateOfChangeOfEntropy

12

TSTP

TimeStep

Block content
HeaderPositions
Velocities
Particle ID’s
Masses (only for particle types with variable masses)
Internal energy per unit mass (only SPH
particles)
Density (only SPH particles)
SPH smoothing length h (only SPH particles)
Gravitational potential of particles (onlywhen enabled in makefile)
Acceleration of particles (only when enabled in makefile)
Rate of change of entropic function of
SPH particles (only when enabled in
makefile)
Timestep of particles (only when enabledin makefile)

Table 2: Output blocks defined in GADGET-2’s file formats.

Type
0
1
2
3
4
5

Symbolic Type Name
Gas
Halo
Disk
Bulge
Stars
Bndry

Table 3: Symbolic type names and their corresponding type numbers.

31

V. Springel

not used by the public version of GADGET-2. The data type of each field in the header is either
double (corresponding to real*8 in Fortran), or int (corresponding to int*4 in Fortran),with the exception of the particle numbers, which are unsigned int.
The format, units, and variable types of the individual blocks in the snapshot files are as
follows:
1. Header:struct io header;

The individual fields of the file header are defined in Table 4.
2. Particle positions:float pos[N][3];real*4 pos(3,N)

Comoving coordinates in internal length units (corresponds to h −1 kpc if the above
choice for the system of units is adopted). N=sum(io header.Npart) is the total number of particles in the file. Note: The particles are ordered in the file according to theirtype, so it is guaranteed that particles of type 0 come before those of type 1, and so on.
However, within a type, the sequence of particles can change from dump to dump.
3. Particle velocities:
float vel[N][3];real*4 vel(3,N)

Particle velocities u in internal velocity units (corresponds to km/sec if the default choicefor the system of units is adopted). Peculiar velocities v are obtained by multiplying u√√with a, i.e. v = u a. For non-cosmological integrations, u = v are just ordinary
velocities.
4. Particle identifications:unsigned int id[N];int*4 id(N)Particle number for unique identification. The order of particles may change between
different output dumps, but the IDs can be easily used to bring the particles back into theoriginal order for every dump.
5. Variable particle masses:
float masses[Nm];real*4 masses(Nm)Particle masses for those particle types that have variable particle masses (i.e. zero entries
in massarr). Nm is the sum of those npart entries that have vanishing massarr. If
Nm=0, this block is not present at all.
6. Internal energy:
float u[Ngas];real*4 u(Ngas)

Internal energy per unit mass for the Ngas=npart(0) gas particles in the file. The
block is only present for Ngas>0. Units are again in internal code units, i.e. for theabove system of units, u is given in (km/sec)2 .
7. Density:
float rho[Ngas];32

real*4 rho(Ngas)

User’s guide for GADGET-2
Header Field

Type

Npart[6]uint

HDF5 name
NumPart ThisFile

Massarr[6]doubleMassTableTime

doubleTime

RedshiftdoubleRedshift

FlagSfr

int

Flag Sfr

FlagFeedback
Nall[6]int
int

Flag Feedback
NumPart Total

FlagCooling
NumFilesBoxSizeint
int
doubleFlag Cooling
NumFilesPerSnapshot
BoxSize

Omega0

doubleOmega0OmegaLambda

doubleOmegaLambda

HubbleParam

doubleHubbleParam

FlagAge
FlagMetals
NallHW[6]

int
int
int

Flag StellarAge
Flag Metals
NumPart Total HWflag entr ics

int

Flag Entropy ICs

unusedComment
The number of particles of eachtype in the present file.
The mass of each particle type. If
set to 0 for a type which is present,
individual particle masses from themass block are read instead.
Time of output, or expansion factor
for cosmological simulations.
z = 1/a−1 (only set for cosmological integrations).
Flag for star formation (unused in
the public version of GADGET-2).
Flag for feedback (unused).
Total number of particles of eachtype in the simulation.
Flag for cooling (unused).
Number of files in each snapshot.
Gives the box size if periodic
boundary conditions are used.
Matter density at z = 0 in units ofthe critical density (only relevant forcosmological integrations).
Vacuum energy density at z = 0 inunits of the critical density.
Gives ‘h’, the Hubble constantin units of 100 km s−1 Mpc−1 (forcosmological integrations).
Creation times of stars (unused).
Flag for metallicity values (unused).
For simulations that use more than
232 particles, these fields hold themost significant word of 64-bit total
particle numbers. Otherwise zero.
Flags that initial conditions containentropy instead of thermal energy inthe u block.
Currently unused space which fills
the header to a total length of 256
bytes, leaving room for future additions.

Table 4: Fields in the file header of snapshot files. In the native file formats of GADGET-2,
these fields are the elements of a structure of total length of 256 bytes. In the HDF5
file format, each field is stored as an attribute to a Header data group.
33

V. Springel

Comoving density of SPH particles. Units are again in internal code units, i.e. for theabove system of units, rho is given in 1010 h−1 M¯ /(h−1 kpc)3 .
8. Smoothing length:
float hsml[Ngas];real*4 hsml(Ngas)

Smoothing length of the SPH particles. Given in comoving coordinates in internal length
units.
9. Gravitational potential:
float pot[N];real*4 pot(N)Gravitational potential for particles. This block will only be present if it is explicitly
enabled in the makefile.
10. Accelerations:float acc[N][3];real*4 acc(3,N)

Accelerations of particles. This block will only be present if it is explicitly enabled in themakefile.
11. Rate of entropy production:
float dAdt[Ngas];real*4 dAdt(Ngas)

The rate of change of the entropic function of each gas particles. For adiabatic simulations, the entropy can only increase due to the implemented viscosity. This block will
only be present if it is explicitly enabled in the makefile.
12. Timesteps of particles:
float dt[N];real*4 dt(N)Individual particle timesteps of particles. For cosmological simulations, the values stored
here are ∆ ln(a), i.e. intervals of the natural logarithm of the expansion factor. This block
will only be present if it is explicitly enabled in the makefile.
6.2 A variant of the default file format
Whether or not a certain block is present in GADGET-2’s default fileformat depends on the typeof simulation, and the makefile options. This makes it difficult to write reasonably general I/O
routines for analysis software, capable of dealing with output produced for a variety of makefile
settings. For example, if one wants to analyse the (optional) rate of entropy production of SPH
particles, then the location of the block in the file changes if, for example, the output of thegravitational potential is enabled or not. This problem becomes particularly acute in the fullphysics version of GADGET-2, where already around 40 I/O blocks are defined.
To remedy this problem, a variant of the default fileformat was implemented in GADGET-2(based on a suggestion by Klaus Dolag), which can be selected by setting SnapFormat=2.
Its only difference is that each block is preceeded by a small additional block which contains
an identifier in the form of a 4-character string (filled with spaces where appropriate). This
identifier is contained in the second column of Table 2. Using it, one can write more flexible

34

User’s guide for GADGET-2

I/O routines and analysis software that quickly fast forwards to blocks of interest in a simple
way, without having to know where it is expected in the snapshot file.

6.3 The HDF file format
If the hierarchical data format is selected as format for snapshot files or initial conditions files,
GADGET-2 accesses files with low level HDF5 routines. Advantages of HDF5 lie in its portability (e.g. automatic endianness conversion) and generality (which however results in somewhat
more complicated read and write statements), and in the availability of a number of tools tomanipulate or display HDF5 files. A wealth of information about HDF5 can be found on thewebsite of the project.
GADGET-2 stores the snapshot file data in several data groups, which are organised as follows:/Header/Type0/Positions/Type0/Velocities/Type0/IDs...
/Type1/Positions/Type1/Velocities...
The Header group only contains attributes that hold the fields of the file header in thestandard file format of GADGET-2. The names of these attributes are listed in Table 4. For
each particle type present in the snapshot file, a separate data group is defined, named Type0,
Type1, and so on. The blocks of the standard file format are then stored as datasets withinthese groups. The names of these data sets are listed in Table 2.

6.4 Format of initial conditions
The possible file formats for initial conditions are the same as those for snapshot files, and are
selected with the ICFormat parameter. However, only the blocks up to and including the gas
temperature (if gas particles are included) need to be present; gas densities, SPH smoothinglengths and all further blocks need not be provided.
In preparing initial conditions for simulations with gas particles, the temperature block can
be filled with zero values, in which case the initial gas temperature is set in the parameterfile
with the InitGasTemp parameter. However, even when this is done, the temperature blockmust still be present.
Note that the field Time in the header will be ignored when GADGET-2 is reading an initial conditions file. Instead, you have to set the time of the start of the simulation with the
TimeBegin option in the parameterfile.

35

V. Springel

7 Diagnostic output files
7.1 Information fileIn the file with name specified by InfoFile, you just find a list of all the timesteps. Typical
output in this file looks like this:Begin Step 49, Time: 0.143555, Systemstep: 0.00292969
Begin Step 50, Time: 0.146484, Systemstep: 0.00292969
Begin Step 51, Time: 0.149414, Systemstep: 0.00292969
Begin Step 52, Time: 0.152344, Systemstep: 0.00292969

The first number just counts and identifies all the timesteps, the values given after ‘Time’ aretheir current simulation times, and the ‘Systemstep’-values give the time difference to the preceeding step, which is equal to by how much the simulation as a whole has been advanced since
the last step. For cosmological integrations, this output is supplemented with the redshift, andthe systemstep in linear and logarithmic form:
Begin Step 636, Time: 0.487717, Redshift: 1.05037, Systemstep: 0.000724494, Dloga: 0.00148658
Begin Step 637, Time: 0.488443, Redshift: 1.04732, Systemstep: 0.000725571, Dloga: 0.00148658
Begin Step 638, Time: 0.489169, Redshift: 1.04428, Systemstep: 0.000726651, Dloga: 0.00148658
Begin Step 639, Time: 0.489897, Redshift: 1.04125, Systemstep: 0.000727732, Dloga: 0.00148658
Begin Step 640, Time: 0.490626, Redshift: 1.03821, Systemstep: 0.000728814, Dloga: 0.00148658

7.2 Performance statistics of the gravitational tree
In the file specified by TimingsFile, you get some statistics about the performance of thegravitational force computation by the tree algorithm, which is usually (but not always) the
main sink of computational time in a simulation. A typical output may look like this:
Step= 482 t= 1.29199 dt= 0.00292969
Nf= 10992 total-Nf= 0016981370
work-load balance: 1.10997 max=0.618782 avg=0.557475 PE0=0.618782
particle-load balance: 1.02133
max. nodes: 15202, filled: 0.422278
part/sec=9858.74 | 8881.96 ia/part=1004.71 (0)Step= 483 t= 1.29492 dt= 0.00292969
Nf= 60000 total-Nf= 0017041370
work-load balance: 1.01677 max=2.51368 avg=2.47221 PE0=2.43074
particle-load balance: 1.02133
max. nodes: 15202, filled: 0.422278
part/sec=12134.9 | 11934.7 ia/part=805.744 (0)36User’s guide for GADGET-2

The first line of the block generated for each step informs you about the number of the current
timestep, the current simulation time, and the system timestep itself (i.e. the time difference tothe last force computation). Nf gives the number of particles that receive a force computation in
the current timestep, while total-Nf gives the total number of force computations since thesimulation was started. work-load balance gives the work-load balance in the actual tree
walk, i.e. the largest required time of all the processors divided by the average time. The number
given by max is the maximum time (in seconds) among the processors spent for the tree walk,
while avg gives the average, and PE0 the time for processor 0. particle-load balance
gives the maximum number of particles per processor divided by the average number, i.e. it
measures the memory imbalance. An upper bound for this number is PartAllocFactor.
However, for technical reasons, the domain decomposition always leaves room for a few particles, hence this upper bound is never exactly reached. max.nodes informs about the maximum number of internal tree nodes that are used among the group of processors, and the number
behind filled gives this quantity normalised to the total number of allocated tree nodes, it
hence gives the fraction to which the tree storage is filled. Finally, part/sec measures theraw force speed in terms of tree-force computations per processor per second. The first number gives the speed that would be achieved for perfect work-load balance. Due to work-load
imbalance, the actually achieved average effective force speed will always be lower in practice,and this number is given after the vertical line. ia/part gives the average number of particlenode interactions required to compute the force for each of the active particles. The number inparenthesis is only non-zero if the Ewald correction is used to obtain periodic boundaries. It
then gives the average number of Ewald correction-forces evaluated per particle.
7.3 CPU-time statistics
In the file specified by CpuFile, you get some statistics about the total CPU consumption
measured in various parts of the code while it is running. A typical output looks like this:Step 13121, Time: 0.168501, CPUs: 2
44000.01
32984.59
6496.83
Step 13122, Time: 0.168504, CPUs: 244001.11
32985.30
6497.0152.40

1318.172744.38 ...

52.40

1318.172744.59 ...

For each timestep, there are two lines. The first line gives the current time step, corresponding
simulation time, and number of CPUs used. In the second line, the first number informs you
about the total cumulative CPU consumption of the code up to this point (in seconds), while
the other numbers measure the time spent in various parts of the code. The numbers givemeasurements for the following parts (in that sequence):
1. Total consumption.
2. Gravitational force computation (including tree walks, construction, communication).
3. Hydrodynamics (including neighbour search, density determination, and SPH communication).

37

V. Springel

4. Domain decomposition.
5. Potential energy computation (for the energy statistics).
6. Drifting the particle set.
7. Timestep determination and ‘kicking’ of particles.
8. Writing of snapshot files.
9. Pure time for tree walks (this is a sub-part of 1).
10. Tree construction (this is a sub-part of 1).
11. Communication and summation (this tries to measure the time spent in communicationand summation of force components in the gravitational tree part).
12. Imbalance of gravitational tree (this measures work-load imbalance losses directly).
13. SPH computations (finding of neighbours and evaluating SPH-sums with them).
14. Communication and summation in the SPH routines.
15. Losses due to work-load imbalance in the SPH part.
16. Neighbour adjustment (measures the time needed to readjust SPH smoothing lengths if
the number of neighbours obtained for the next guess smoothing length falls outside thedesired range).
17. Time for PM computation.
18. Time needed to establish Peano-Hilbert order.
19. Time spent in cooling and star formation routines. (Not present in the public version of
GADGET-2.)
All these times are wallclock times, provided WALLCLOCK is set in the makefile, which is
the recommended setting. Otherwise, they correspond to CPU cycles (expressed in seconds)
consumed on average per processor. The latter way of measuring things is more meaningful
if GADGET-2 has not exclusive access to all of the CPUs (i.e. other processes consume a nonnegligible fraction of the available CPU processes), but instead needs to share them with otherprocesses to a significant degree.

38

User’s guide for GADGET-2
7.4 Energy statistics
In the file specified by EnergyFile, the code gives some statistics about the total energyof the system. In regular intervals (specified by TimeBetStatistics), the code computes the total kinetic, thermal and potential energy of the system, and it then adds one line
to EnergyFile.
Each of these lines contains 28 numbers, which you may process by some analysis script.
The first number in the line is the output time, followed by the total internal energy of thesystem (will be 0 if no gas physics is included), the total potential energy, and the total kinetic
energy.
The next 18 numbers are the internal energy, potential energy, and kinetic energy of thesix particle types Gas to Bndry. Finally, the last six numbers give the total mass in thesecomponents.
While the meaning of kinetic and potential energy in non-cosmological simulations is straightforward (and also the test for conservation of total energy), this is more problematic in cosmological integrations.
For cosmological integrations, the kinetic energy you obtain in this file will be
T =1X
mi w 2 ,
2 i

(1)

√
where w is the peculiar velocity divided by a (see paper). The potential energy will becomputed as
1X
U=mi Φ(xi ),(2)2 iwhere Φ(xi ) is the peculiar potential when periodic boundaries are used.
If vacuum boundaries are used, and no cosmological integration is done, Φ(x i ) is simply the
Newtonian potential. However, for comoving integration with vacuum boundaries the potential
is computed asX
1(3)
Φ(xi ) = −G
mk g (|xk − xi |) − Ω0 H02 x2i2kFrequent outputs of the energy quantities allow a check of energy conservation. While thisis straightforward for Newtonian dynamics, it is more difficult for cosmological integrations,where the Layzer-Irvine equation is needed. (Note that the softening needs to be fixed in comoving coordinates for it.) We remark that it is in practice not easy to obtain a precise valueof the peculiar potential energy at high redshift (should be exactly zero for a homogeneousparticle distribution). Also, the cosmic energy integration is a differential equation, so a test ofconservation of energy in an expanding cosmos is less straightforward that one may think, tothe point that it is nearly useless for testing code accuracy.
Also note that for vacuum boundary conditions in cosmological integrations, the zero-pointof the potential will not be the one of the ordinary peculiar potential. Therefore, the potential
energy will be orders of magnitude larger than the kinetic energy, and the kinetic energy will
be dominated by the outer shell of the simulation volume in this case.

39

V. Springel

8 Differences between GADGET-1 and GADGET-2
8.1 Architectural changes
There have been many major changes in the internal workings between GADGET-1 and GADGET2, affecting nearly all parts of the code. In fact, basically all parts of the code have been completely rewritten, or were revised substantially. These modifications were done to improve theaccuracy and the performance of the code, to reduce its memory consumption, and to make itapplicable to a wider range of types of simulations. Detailed explanations of the reasoning behind each of these modifications is beyond the scope of this short set of notes. This information
can be found in the code paper on GADGET-2. However, below, I give a very brief summary ofsome of the most important changes in code design.
1. The internal organisation of the time integration has been changed substantially. The
timestep of a particle is now usually restricted (again) to be a power of 2 subdivisionof the total time span that is simulated. (It is however also possible to run the codewith a more flexible timestep hierarchy.) The integration proceeds then as an alternation
between ‘drift’ and ‘kick’ operations, with the leapfrog being a kick-drift-kick scheme
now, instead of a drift-kick-drift. There is no explicit prediction step any more, since thispart has been absorbed into the drift operation.
2. The quadrupole moments in the gravitational tree have been dropped, i.e. the new tree has
only monopole moments. To achieve the same force accuracy, this generally means that
more cell-particle interactions have to be evaluated. However, because each interaction
is relatively simple, this effect is largely offset by the increased performance of the treewalk in terms of interactions per second. This has become possible because the monopole
interactions are ‘simpler’ than the quadrupole ones, and in addition, the cache utilisationin the tree walk has been improved. The net result is that the new gravitational codeis faster than the old one at the same accuracy. The ultimate reason for dropping thequadrupole moments has however been that this simplifies the dynamic update (‘drift’)of the tree. This is now done fully consistently, i.e. the tree accuracy does not degrade
when the tree has not been updated for a while. (In the old code, the change of thequadrupole moments had been neglected. Also, in the Ewald correction their contribution
was neglected.)
3. For cosmological integrations, the integration variables have been changed: The velocity√
variable is now taken to be u = a2 ẋ, and not w = aẋ as before. This was adopted
because then the equation of motion for u does not depend on u any more, and only
then can the leapfrog be formulated in a symplectic way, which allows the Hamiltonian
structure of the system to be preserved. (Note however that the velocities in the snapshot
and initial conditions files are still in terms of w in order to have backwards compatability
with GADGET-1.)
4. SPH is implemented using the conservative ‘entropy formulation’ by default.

40

User’s guide for GADGET-2

5. The code may now be run optionally as a Tree-PM hybrid code, where the long-range
gravitational force is computed with discrete Fourier transforms. A parallel FFT is used
to this end. Both periodic and non-periodic boundary conditions are supported. Furthermore, an extension of the PM algorithm is available that is designed for “zoom” simulations, where a small volume is sampled with very many high-resolution particles. Here,the code can be instructed to place a second PM mesh on the high-res region, such that thefull force for the high-res particles is composed of three components, a long-range PM
force, an intermediate-range PM force, and a short range tree force. The time integration
between short-range and long-range forces is decoupled such that the PM timestep may
be larger than the short-range step. Nevertheless the integration scheme in principle stillpreserves its symplectic character (in an exact fashion only if the short-range steps do notchange and are all equal, and if force errors are negligible).

8.2 Notes on migrating from GADGET-1 to GADGET-2
If you have used the earlier version GADGET-1 before, you will be familiar with all the practicalities of running GADGET-2 already. This is because nothing has really changed here. In
particular, the default initial conditions file format has not changed (apart from some extensions), and the mechanisms for starting, interrupting, and resuming a simulation are still thesame.
However, as described above, there have been a number of changes in the parameterfile
that controls each simulation, and there are more compilation options in the makefile of thecode. If you want to reuse an old parameterfile with GADGET-2, you therefore have to make a
few modifications of it. If you simply start the new code with the old parameterfile, you will
get error messages that complain about obsolete or missing parameters. Delete or add theseparameters as needed (see also the information below). Note in particular that the meaning of
the parameter MaxSizeTimestep has changed.
As a result of the different time integration scheme used in the new version of the code,
considerably smaller steps are done by the code at high redshift than it used to be the case. Thisis necessary because even in the fully linear regime, the timestep may not exceed a certain small
fraction of the Hubble time in the new time integration scheme, otherwise the integration of thelinear evolution would become compromised. The old code was able to avoid this problem
by a trick – its integration step essentially mimicked the Zeldovich approximation, allowing it
to traverse linear evolution with pretty large steps. However, this integration scheme was not
symplectic, while the new one is. The advantages brought about by the new integration scheme
far outweigh the additional cost of having to do a few more steps at high redshift.
Note that one should also carefully review which makefile options are appropriate for a given
simulation. For many types of simulations, there are now different possibilities for running asimulation, e.g. with or without TreePM algorithm, with single or double-precision, etc.

41

V. Springel
8.3 Dropped makefile options
In case you wonder, GADGET-1 contained a number of makefile options that are not present any
more. The following table lists them, and briefly explains the reason why they were dropped.
-DVELDISPThe timestep criterion based on an estimate of thelocal velocity dispersion has been dropped because,
disappointingly, it did not do significantly better
than simpler timestep criteria based on the velocity
alone.
-DBMAX
Something equivalent to BMAX is always enforced in
the new tree walk.
-DRECTREECONS The monopole moments of the tree are always constructed recursively in the new code.
-DINLINE
The routines that could have benefitted from inlining
have been inlined manually, or have been replacedby macros.
-DDIAG
Timing diagnostics are now always collected by thecode.
8.4 List of source files of GADGET-2
In Table 5, you can find an updated list of the source files of GADGET-2. The source code iscommented and reasonably readable I hope. As an additional help to browse the sources, they
can be accessed via a cross-references hypertext documentation that can be found in the html
subdirectory (open the file index.html in your favourite browser). This documentation has
been generated directly from the annotated source code with the doygen tool. If you change
the sources and want to regenerate the sources, you can do so by executing the commanddoygen in the source directory of GADGET-2. This will only work if doygen has been
installed on your system, of course.
8.5 Notes on memory consumption of GADGET-2
Let Ntot be the total number of particles, and Np the number of processors. Then the average
particle load per processor is N = Ntot /Np . The code will allocate about
M1 = PartAllocFactor × N × (68 + TreeAllocFactor × 64)(4)bytes for particle storage, for the gravity/neighbour tree, and for the time integration scheme,on each processor. For typical values like
TreeAllocFactor ' 0.65 − 0.7, this implies a memory cost of ∼110 bytes per particle. For
SPH particles, additional memory of
M2 = PartAllocFactor × Nsph × 84

(5)bytes will be required. For an equal mixture of SPH and collisionless particles, the averagememory consumption per particle is therefore going to be about 152 byte/particle.

42

User’s guide for GADGET-2

accel.callocate.c
allvars.c
begrun.cdensity.c
domain.cdriftfac.c
endrun.cforcetree.c
global.c
gravtree.c
gravtree forcetest.c
hydra.cinit.cio.clongrange.c
main.cngb.cpeano.cpm nonperiodic.cpm periodic.c
potential.c
predict.cread ic.c
restart.c
run.csystem.ctimestep.c

Driver routine to carry out force computation
Routines for allocating particle and tree storage
Provides instances of all global variables
Initial set-up of a simulation run
SPH density computation and smoothing length determination
Code for domain decomposition
Computes look-up tables for prefactors in cosmological integrations
Terminates run upon severe errors
Gravitational tree code
Global energy computation
Main driver routines for gravitational (short-range) force computation
Routines for direct summation forces
Computation of SPH forces and rate of entropy generation
Code for initialisation of a simulation from initial conditions
Routines for producing a snapshot file on disk
Driver routines for computation of long-range gravitational PM force
Start of the program
Neighbour search by means of the tree
Routines to compute a Peano-Hilbert order
Code for non-periodic FFT to compute long-range PM force
Routines for periodic PM-force computation
Computation of the gravitational potential of particles
Drift particles by a small time interval
Read initial conditions in one of the supported file formats
Code for reading and writing restart files
Iterates over timesteps, main loop
Miscellaneous routines, e.g. elapsed time measurements
Routines for ‘kicking’ particles and assigning new timesteps

allvars.h
proto.htags.hDeclares global variables
This file contains all function prototypes of the code
Declares various tags for labelling MPI messages
Table 5: List of source files of GADGET-2.

43

V. Springel

Note however that it is not advisable to fill all the available memory of the machine. For thesake of performance one should leave room for memory-imbalance, i.e. PartAllocFactor
needs to be chosen larger than 1.0. In practice, values below 1.2 can lead to serious workimbalance, depending on the amount of clustering and on how many processors are involved.
Certain code options will also increase the amount of memory allocated per particle. Obviously, selecting DOUBLEPRECISION can increase the storage requirement substantially,
almost doubling it. Also, simulations with cooling and star formation require slightly morememory to allow the storage of additional quantities like ionisation fraction, metallicity, etc.
In practice, it is thus best to pay attention to the screen output that is produced in the log-filewhen GADGET-2 starts. The code will report the sizes of all major chunks of memory that areallocated. Note that these numbers refer to the allocation done by each individual processor. To
obtain the total amount of memory needed by the code, one needs to multiply the sum of thesenumbers by the number of processors that are used.
The communication buffers requires
M3 = BufferSize × 10242(6)

bytes, but this number is independent of the particle load.
Note that the TreePM algorithm can require substantial amounts of additional memory, depending on the choice of PMGRID, and the type of TreePM simulation done. In the simplest
case, a periodic box, the allocated amount of memory, summed over all processors, is about
M4 = PMGRID3 × 12 − 16(7)bytes, provided single precision is used. For double precision, twice this amount is needed. If
the mesh is made a factor of 2 finer than the mean particle spacing in each dimension, then therequired storage for the PM algorithm roughly matches that needed for the particle data and thetree, i.e. then the memory requirement of the code essentially doubles.
If the TreePM algorithm with non-periodic boundaries is selected instead, the memory requirements are substantially larger. They are then roughly
M5 = (2 ∗ PMGRID)3 × 16

(8)bytes. If a two level zoom simulation in a periodic volume is selected, then M 4 and M5 areboth allocated. If the zoom is done in a simulation with vacuum boundaries, then only M 5 isallocated, but the number 16 is replaced by 20.

9 Examples
To get some idea of the usage of the code, you may try a few of the simple examples I provide
with the code distribution (these are the same ones as in GADGET-1). The parameterfiles forthese examples can be found in the parameterfiles-subdirectory of the code-directory.
The first thing you have to do before you can run any of the examples is to edit the corresponding parameterfile and adjust the path of the output directory, and the filename of the initial44

User’s guide for GADGET-2

conditions file. Initial conditions (and snapshot files) of GADGET-2 are in binary format, so
its important to know the byte order of your system. Intel, AMD, and some Alpha machines(not the T3E, though) use little endian as opposed to big endian, which is the standard on mostother RISC machines (IBM, Sun). For each IC-File, I have prepared a corresponding file forlittle endian machines, designated by an extension ‘.intel’. These files are in the ‘ICs’
directory. (Note that you can reverse byte-order easily by writing a little C-routine yourself.)
The different examples are explained further below.

9.1 Galaxy collisionThis purely collisionless simulation runs two disk galaxies into each other, leading to a merger
between the galaxies. Each galaxy consists of a stellar disk, and a massive and extended dark
matter halo. This example uses plain Newtonian physics, with 20000 disk and 40000 halo
particles in total.
To allow you to get a first idea whether the examples have worked, I include a few IDL scriptsthat read snapshot files and diagnostics output. These files are in the directory ‘Analysis’.
You can use ‘energy galaxy.pro’ to plot the evolution of kinetic and potential energy, andthe variation of the total energy in the galaxy example. You may use the script ‘show galaxy.pro’to load the snapshot files of the galaxy collision one after the other and display the disk particlesof the galaxies.

9.2 Adiabatic collapse of a gas sphereThis simulation considers the gravitational collapse of a self-gravitating sphere of gas which
initially has a 1/r density profile at a low temperature. The gas falls under its own weight to thecentre, where a bounce-back with a strong shock wave develops. This common test problem of
SPH codes has first been described by Gus Evrard.
The simulation uses Newtonian physics in a natural system of units (G = 1). You can use‘energy gassphere.pro’ to display the evolution of thermal, kinetic and potential energy
for the collapsing gassphere. Note that this is a very tiny simulation of just 1472 particles.

9.3 Cosmological formation of a cluster of galaxies
This problem uses collisionless dynamics in an expanding universe. It is a small cluster simulation that has been set-up with Bepi Tormen’s initial conditions generator ZIC using vacuum
boundaries and a multi-mass technique. The simulation has a total of 276498 particles. In acentral high-resolution zone there are 140005 particles, surrounded by a boundary region with
two layers of different softening, the inner one containing 39616 particles, and the outer one
96877 particles. The IDL-script ‘show cluster.pro’ is an example for reading one of thesnapshots of the cluster simulation and showing the high resolution region.

45

V. Springel
9.4 Large-scale structure formation including gas
This problem consists of 323 dark matter, and 323 gas particles, following structure formation
in a periodic box of size (50 h−1 Mpc)3 in a ΛCDM universe. Only adiabatic gas physics is
included, and the minimum temperature of the gas is set to 1000 K. This simple example uses
grid initial conditions, where gas particles are put at the centres of the grid outlined by the dark
matter particles. The simulation starts at z = 10, and the code will produce snapshot files at
redshifts 5, 3, 2, 1, and 0.
You can use ‘show gas.pro’ to read a snapshot file of this simulation, to extract the gas
particles from it, and to plot their spatial distribution. You may use ‘plot hotgas.pro’ toshow the gas with temperature higher than 106 K at the present time.

10 Disclaimer
It is important to note that the performance and accuracy of the code is a sensitive function ofsome of the code parameters. Let me also stress that GADGET-2 comes without any warranty,
and without any guarantee that it produces correct results. If in doubt about something, reading(and potentially improving) the source code is always the best strategy to understand what isgoing on.

Please also note the following: The numerical parameter values used in the examples
contained in the code distribution do not represent a specific recommendation by the author!
In particular, I do not endorse these parameter settings in any way, nor do I claim that they will
provide fully converged results for the example problems, or for more general initial conditions.
I think that it is extremely difficult to make general statements about how convergence can beobtained, especially when one desires to achieve it with the smallest possible computational
effort. For this reason I refrain from making such recommendations. I encourage every simulator to find out for herself/himself what integration settings are needed to achieve sufficientaccuracy for the system under study. I strongly recommend to make convergence and resolution
studies to establish the range of validity and the uncertainty of any numerical result obtainedwith GADGET-2.
Volker Springel
Munich, April 2005

46