# Presentation: 376315

## Cosmic Microwave Background Data Analysis At NERSC

- Julian Borrill
- with
- Chris Cantalupo
- Ted Kisner

## What Is The CMB ?

- A snapshot of the Universe when it first became neutral 400,000 years after the Big Bang.

- Cosmic - filling all of space.
- Microwave - redshifted by the expansion of the Universe from 3000K to 3K.
- Background - primordial photons coming from “behind” all astrophysical sources.

## Why Do We Care About The CMB ?

- The CMB is a unique probe of the very early Universe.
- Its tiny (1:105-8) fluctuations carry information about
-  - the fundamental parameters of cosmology
-  - ultra-high energy physics beyond the Standard Model

## What Does The CMB Look Like ?

## CMB Past, Present & Future

- 1948: Gamow, Alpher & Herman - theoretical prediction.
- 1965: Penzias & Wilson - accidental discovery.
- 1966 - 1970: anisotropies predicted (S-W, R-S, S-Z).
- 1970 - 1990: are we nearly there yet (T) ?
- 1992: COBE - first T anisotropy detection.
- 1990 - 2000: are we nearly there yet (T peaks) ?
- 2000: BOOMERanG & MAXIMA - first 3 T peaks.
- 2002: DASI - first P anisotropy detection
- 2003 - ?: WMAP - confirms & refines all of the above.
- 2000 - 2010: are we nearly there yet (E-mode) ?
- 2008 - 2014: Planck
- 2010 - 2020: are we nearly there yet (B-mode) ?
- 2020 - 2025: Inflation Probe

## HPC And The Future Of CMB Research

- The new frontier in CMB research is polarization :
-  consistency check of temperature results
-  re-ionization history of the Universe
-  gravity wave production during inflation
*But* polarization fluctuations are up to 3 orders of magnitude fainter than temperature requiring :
-  many more detectors
-  much longer observation times
-  much more discriminating analysis
- Much larger datasets & much harder analyses => HPC

## CMB Work At NERSC

- Started in 1997:
- separate allocations for Maxima & Boomerang
- together 5 users & 30,000 CPU-hours
- Developed into premier world center for CMB analysis:
- single allocation shared by O(10) experiments
- O(100) users & O(1,000,000) CPU-hrs/year
- Now includes "Big Science" satellite mission
- split into two allocations
- mp107 - 13 sub-orbital experiments				     40 users & 500,000 MPP-hrs
- planck: Planck satellite						    60 users & 2,000,000 MPP-hrs*

## The Planck Satellite

- The primary driver for NERSC CMB work.
- A joint ESA/NASA mission due to launch in summer 2008.
- An 18+ month all-sky survey at 9 microwave frequencies from 30 to 857 GHz.
- O(1012) observations, O(108) sky pixels,		O(104) spectral multipoles.

## US Planck HPC Hardware Infrastructure

- 				NERSC

- NGF

- HPSS

- PDSF

- Franklin
- et al

- Scratch

- Ames

- US Planck
- Secure Zone

- IPAC

- E
- U
- R
- O
- P
- E

## US Planck HPC Software Infrastructure

- CMB
- Analysis
- Tool

- 							Berkeley++

- LFI DMC+pipeline

- HFI DMC+pipeline

- LFI
- data

- HFI
- data

- Any
- CMB
- data

- Data
- Abstraction
- Layer

- CMB
- Analysis
- Tool

- HPSS

- Data
-  Management

- DPC-provided

## HPC CMB Data Analysis Issues

- Data flow
- experiment to archive
- archive to disk
- disk to memory
- memory to memory
- memory to cache
- Project management
- shared repos
- group accounts
- data sharing/security
- code sharing/security
- reliable allocations

## Data Management

- 100s TB of data in 100,000s of files !
- no common format, distribution, naming, etc
- Storage
- HPSS, NGF, {scratch ... Franklin, Seaborg}
- WAN NGF (LBL, Ames)
- name-space mirroring & abbreviating !
- Database & scripting tools to manage
- storing
- inspecting
- retrieving
- purging
- XML/RunConfig building

## Data Abstraction Layer

- Data Abstraction library (M3)
- provides common interface to *arbitrary* CMB data
- any format or file-distribution
- full, compressed, virtual, on-the-fly sim
- XML RunConfig
- immediate inheritance of capabilities
- Generalized Compressed Pointing library (GCP)
- huge data compression
- on-the-fly re-calculation vs storage/reading
- Simulation library (SIM)
- on the fly simulation capability
- basic/generic & complex/specific

## Algorithms

- MADCAP (MAD-nes, -pre, -map, -ping, -spec, -bench)
- exact maximum likelihood analysis
- scales as O(Nt ln Nt) in time & O(Nl Np3) in pixels
- Data volume driving us to
- linear/log-linear algorithms
- less forgiving of inefficiencies
- approximation errors must be quantified
- data decomposition
- possibly hierarchy of scales
- de-/re-composition errors must be quantified

## Implementations

- Minimizing IO overhead
- asynchronous IO using MPI-2 (I:M3 & IO:Apps)
- compression & on-the-fly reconstruction
- Controlling memory management
- memory allocation/freeing at highest level
- tunable buffers
- Optimizing core operations (IPM !)
- fourier transform
- spherical harmonic transform
- sparse matrix-vector multiplication

## Project Management

- Single planck account with keyfob access
- UNIX owner file permissions on
- data
- proprietory codes
- planck module
- Self-managed modules in cmb project space for
- cmb: general CMB tools
- planck: planck-specific tools
- provide access to executables & libraries
- not possible through (finite) environment
- use config scripts & backtick operator
- `cmbconfig -I fftw healpix gcp`

## Project Management

- International satellite mission requires absolutely reliable resources - annual ERCAP not guaranteed.
- Planck data requires exceptional facilities - investment in NGF & PDSF,
- MoU under negotiation:
- DOE/HEP guarantees at least 2,000,000 MPP-hrs/yr through life of mission (2014), and Planck ERCAP may request more.
- NASA/Planck guarantees at least 2.5 FTE at Berkeley
- NASA/Planck buy in to 32TB NGF & 128 CPU PDSF with 3 year refresh and fractional FTE support TBD.

## Project Management

- mp107 shared by 13 experiments & 40 users
- each user has 5% cap initially, raised if justified
- problems:
- can submit any job provided quota in credit
- no-one but user can kill job
- each experiment has a filegroup
- very nice way of selectively sharing data/code
- problems
- quotas set by group not user
- conflates quota/permissions
- usergroup is silly
- new gpfs quota by directory - YES PLEASE !

## Overall

- HPC is essential for present & future CMB work.
- NERSC has become the premier world center.
- Recognized as such by DOE/NASA/NSF taskforce on future of CMB research.
- This has only happened because NERSC facilities are
- Capable
- Accessible
- Reliable
- YOU GUYS ROCK