Cosmology Simulations
with Enzo

John Wise (Georgia Tech)
Enzo Workshop
北海道⼤大学 – 19 Nov 2014

Outline• Introduction to unigrid cosmology simulations• Introduction to nested grid cosmology simulations• Using diﬀerent non-equilibrium chemistry models• Including radiative cooling & star formation/feedback• Including radiative transfer

A universe in a box

Planck Collaboration (2014)

A universe in a box

Linearly evolve to astarting redshift (z ~ 50–200)

z ~ 1080

White et al. (1985)

Use simulations to evolve in
the non-linear regime

Unigrid Initial Conditions• Two programs are provided with Enzo to generate cosmology initialconditions: inits (src/inits) or mpgrafic (src/mpgrafic)• However, we will be using the program MUSIC (Hahn & Abel 2011) that ismore robust and eﬃcient.
• http://www.phys.ethz.ch/~hahn/MUSIC/index.html• https://bitbucket.org/ohahn/music• 10 initial condition files in HDF5 file format• Baryon Density: 1 x N x N x N• (x,y,z) Baryon Velocity: 1 x N x N x N• (x,y,z) Particle Velocity: 1 x N x N x N• (x,y,z) Particle Displacement: 1 x N x N x N

Unigrid Initial Conditions with MUSIC• MUSIC: MUlti-Scale Initial Conditions• Designed specifically for nested grid“zoom-in” simulations.
• Improves errors by two orders ofmagnitude compared to previouscodes.
• Zel’dovich approximation and 2LPT
• Multi-code support (e.g. GADGET,ART, Tipsy, RAMSES, Enzo)• Publicly available, OpenMP parallelUnigrid Initial Conditions with MUSIC• Requirements:• GNU Scientific Library (GSL)• FFTW 2 or 3
• HDF5

Installing MUSIC• Download the source code (on conival)• hg clone https://bitbucket.org/ohahn/music• cp ~guest01/music/Makefile music/• cd music• make• export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/home/
guest01/local/lib

Grab MUSIC and Enzo parameter files• On conival,• cd ~• tar xf ~guest01/sapporo_cosmo.tar• On your laptop• wget http://www.physics.gatech.edu/~jw254/scpics/sapporo_cosmo.tgz• tar xf sapporo_cosmo.tgz

Initial Conditions Parameters• Cosmology Parameters: box size, Hubble constant, mass-energy fractions(ΩΛ, Ωm, Ωb), initial redshift• Power spectrum parameters: power spectrum type, σ8, random seedunits of comoving Mpc/hlevel = 5 → 323 resolution

Initial Conditions Parameters – Grid parameters

Initial redshift
Base resolution = 2n cells across!

Finest resolution of zoom-in region
Generate baryon ICs
2LPT = yes; Zel’dovich = noCreate the initial conditions!
• export OMP_NUM_THREADS=4• ~/music/MUSIC sapporo_cosmo.conf

Set up Enzo parameter file• MUSIC creates a basic but non-functional parameter file, parameters.txt.
• The user must add the relevant AMR, hydro, chemistry, etc. parameters.
• See run/Cosmology/Hydro/AMRCosmology/AMRCosmology.enzo foran example.
• For this workshop, we have provided a set of example parameter files foreach lab exercise.

Run the simulation• Move parameter file and initial conditions into the same directory (seeAdiabatic directory)• Run enzo!
!

• Add the “-d” option for debugging information• Add the “-r” option for restarting from a snapshot.

Projected GasDensity

Nested Grid Cosmology
Simulations
aka Zoom-in calculations
1. N-Body Simulation
2. Locate region of interest, e.g.
most massive halo
3. Generate nested grid initialconditions
4. Run simulation!

N-body Simulation• Usually run at the same resolution as the production level-0 resolution• Create initial conditions with MUSIC and use_baryons=no• Key parameters

Halo finding• On the last output, we need to find the cosmological halos.
• Enzo has a built-in halo finder. Run it with!

• Will put the halo list sorted by mass in the FOF/ directory.

PositionMassz=3

Projected DM Density

Nested Grid Initial Conditions with MUSIC• Find halo of interest.
• We want to re-sample the Lagrangian volume of this region with a higherresolution.
• We can use yt to find this volume. Sample script can be found at• https://bitbucket.org/MatthewTurk/yt.lagrangian_volume• All particles have unique IDs, so we can find the particles in some region inthe final output.
• Then we find the bounding box of the same particles in the initial output.

MUSIC: Nested Grid Parameters

Resolution of the finest nested grid.
Effective resolution = 27 = 128
Center of the Lagrangian volume
Width of the Lagrangian volumeOptional: New random seeds for nested grids.

padding = 2

Nested Grid Initial Conditions with MUSIC• ./MUSIC sapporo_cosmo_nested.conf

1 Nested GridUnigrid
Smoothed DM Density SliceSetting up the Nested Grid Parameter File

Nested Grid Dimensions
and Boundaries

Refine region restricts AMR grids within these coordinates.
Should equal or be smaller than the finest nested gridOverdensity criteria• Very important: Must divide the baryon and DM mass refinement criteria by afactor of 8n, where n := maximum initial level• e.g. Refine on an overdensity of 3 with 2 initial grids → 3.0/64 = 0.046875• If not done, the hierarchy will remain under-resolved, compared with thedesired results.

Run enzo! (see directory: DarkMatter-Nested)• Running on a single core, outputting standard output to display.
!

• Running on 4 cores, piping standard output to the file, estd.out.
!

• This simulation takes significantly longer than the simulation with no nested
grids. On my laptop, it took 20 minutes on 2 cores.

Differences between simulations• Same spatial resolution but higher mass resolution!

z = 10Differences between simulations• Same spatial resolution but higher mass resolution!

z=3

z=3

Different chemistry and cooling models• MultiSpecies == 0: Equation of state with an adiabatic index ϒ• RadiativeCooling == 1: Cooling from a tabulated cooling curve (copyinput/cool_rates.in to working directory)

Non-equilibrium chemistry• MultiSpecies == 1: 6-species (H, H+, He, He+, He++, e–)• MultiSpecies == 2: 9-species (H, H+, He, He+, He++, e–, H–, H2, H2+); i.e.
+molecular hydrogen cooling• MultiSpecies == 3: 12-species (H, H+, He, He+, He++, e–, H–, H2, H2+, D, D+,HD); +HD coolingDifferent chemistry and cooling models• MetalCooling == 1: Tabulated cooling table from Glover & Jappsen (2007) &Sutherland & Dopita (1993) – input/metal_cool.dat.
• T < 104 K: fine-structure metal-line cooling• T > 104 K: collisional ionization equilibrium• MetalCooling == 2: Explicitly calculated metal cooling rates (Cen)• MetalCooling == 3: Tabulated cooling table from CLOUDY (Smith et al. 2008)

3 more parameters for large nested grid runs
• i.e. >2563 nested grids• ParallelRootGridIO = 1: Root grids are tiled before I/O, and eachprocessor reads/writes their own tile
• PartitionNestedGrids = 1: Same thing as above but for nested grids• UnigridTranspose = 2: More parallel-eﬃcient transpose for the FFT done
for the gravity solver.

Summary
You should now know how to:• Create cosmological unigrid and nested grid initial conditions with MUSIC. Formore, see• http://www.phys.ethz.ch/~hahn/MUSIC/index.html• https://bitbucket.org/ohahn/music• https://bitbucket.org/ohahn/music/downloads/MUSIC_Users_Guide.pdf• Run cosmological regular and nested grid runs (N-Body and N-Body + gas)
• http://enzo.readthedocs.org/en/latest/tutorials/RunCosmologySimulation.html• However, these instructions are for the inits initial conditions generator that isincluded with Enzo.