# Astrophysics Data Handling Archiving and Mining Panel White Paper

Version 1, Dec. 11, 2008

# 1. INTRODUCTION: DATA INTENSIVE SCALABLE COMPUTING

The nature of high performance computing is changing. While a few years
ago much of high-end computing involved maximizing CPU cycles per second
allocated for a given problem; today it revolves around performing
computations over large data sets. This means that efficient data access
from disks and data movement across servers is an essential part of the
computation. Data sets are doubling every year, growing slightly faster
than Moore's Law\[1\]. This is not an accident. It reflects the fact
that scientists are spending an approximately constant budget on more
capable computational facilities and disks whose sizes that have doubled
annually for over a decade. The doubling of storage and associated data
is changing the scientific process itself, leading to the emergence of
eScience -- as stated by Gray's Fourth Paradigm of Science based on Data
Analytics\[2\].

Much data is observational, due to the rapid emergence of successive
generations of inexpensive electronic sensors. At the same time large
numerical simulations are also generating data sets with increasing
resolutions, both in the spatial and temporal sense. These data sets are
typically tens to hundreds of terabytes. The scientific community is in
a dire need of a scalable solution for data-intensive computing. Data
sets in High Energy Physics have already exceeded a PB\[3\], with much
more to come when the LHC accelerator starts operating. Astrophysics is
going to cross the PB boundary over the next 12 months\[4\]. The
commercial world has clearly recognized the need for processing and
analyzing large amounts of data. Several of the world's largest
companies are setting up their own facilities to store and analyze many
petabytes of data.

##### The next challenge: massive data

Due to these data intensive scientific problems a new challenge is
emerging, as many groups in science (but also beyond) are facing
analyses of data sets in tens of terabytes, eventually extending to a
petabyte since disk access and data-rates have not grown with their
size. There is no magic way to manage and analyze such data sets today.
The problem exists both on the hardware and the software levels. The
requirements for the data analysis environment are (i) scalability,
including the ability to evolve over a long period, (ii) performance,
both for streaming and random access, (iii) ease of use, including ease
of efficient programming, multi-purpose data representations and
interactive sharing of massive data, (iv) fault tolerance, including
efficient tolerance of failure during computation and high integrity and
persistence of valuable data, and (v) cost-effectiveness, including
continued storage density improvements and memory hierarchies that
decouple application performance from the least cost storage media.

##### Massive data analysis is different from conventional supercomputing

The applications that have driven today's supercomputer designs are
inter-process-intensive. Distributed memory simulation begins with
relatively small inputs, executes synchronized N-body models of complex
processes, generating periodic checkpoints of systems state for
visualization and fault tolerance. In contrast, the analysis of retained
state, for structure or abnormality, shares more with the
characteristics of observation. Inputs are huge, computation is less
intensive, inter-process communication rare but data-driven storage
access common. Data-intensive computing uses large computing clusters to
extract knowledge from data stores far larger than the combined memory
size of any communication-intensive
computer.![](media/image1.pct){width="5.165277777777778in"
height="3.875in"}

##### Bring analysis to the data, not vice-versa

Many of the typical data access patterns in science require a first,
rapid pass through the data, with relatively few CPU cycles carried out
on each byte. These involve filtering by a simple search pattern, or
computing a statistical aggregate, very much in the spirit of a simple
mapping step of MapReduce\[10\]. Such operations are also quite
naturally performed within a relational database, and expressed in SQL.
So a traditional relational database fits these patterns extremely well.

The picture gets a little more complicated when one needs to run a more
complex algorithm on the data, not necessarily easily expressed in a
declarative language. Examples of such applications can include complex
geospatial queries, processing time series data, or running the BLAST
algorithm for gene sequence matching. The traditional approach of
bringing the data to where there is an analysis facility is inherently
not scalable, once the data sizes exceed a terabyte due to network
bandwidth, latency, and cost. It has been suggested \[2\] that the best
approach is to bring the analysis to the data.

##### Building Balanced Systems: Amdahl's Laws

Amdahl has established several laws for building a balanced computer
system \[14\]. These were reviewed recently\[15\] in the context of the
explosion of data. The paper pointed out that contemporary computer
systems IO subsystems are lagging CPU cycles. In the discussion below we
will be concerned with two of Amdahl's Laws:

*A balanced system*

-   *needs one bit/s of IO for each CPU instruction/s*

-   *has 1 byte of memory for each CPU instruction/s*

These laws enumerate a rather obvious statement-- in order to perform
continued generic computations, we need to be able to deliver data to
the CPU, through the memory. Amdahl observed that these ratios need to
be close to unity and this need has stayed relatively constant.

The emergence of multi-level caching led to several papers pointing out
that a much lower IO to MIPS ratio coupled with a large enough memory
can still provide a satisfactory performance\[16\]. While this is true
for problems that mostly fit in memory, it fails to extend to
computations that need to process so much data (PB) that they must
reside on external disk storage. At that point having a fast memory
cache is not much help, since the bottleneck is disk IO.

It is revealing to consider some recent simulations and data analysis
projects. Figure 1a shows the Amdahl Number, the number of data in bits
divided by the number of CPU cycles, for a set of different projects and
for different hardware systems. The small purple arrows on the left
indicate typical supercomputer configurations ranging in an Amdahl
number from 0.001 (BlueGene) to 0.01 (XT-3). The green arrow shows the
Amdahl number of a typical BeoWulf cluster (0.08), while the blue arrow
corresponds to the GrayWulf, our IO oriented database cluster built at
JHU (0.5).

Aquarius (Springel etal 2008) GHALO (Stadel etal 2008), and Via
Lactea-II (Diemand etal 2007, 2008) are large cosmology simulations with
several 10 billion particles, generated using millions of hours of CPU
time on supercomputers. The BeoWulf-based computations are shown in
green, with different numbers of snapshots in the output (128 for
Millennium, 10 for Turbulence, 1024 for Turbulence 1K). SDSS represents
the pipeline processing of the Sloan Digital Sky Survey images,
resulting in the automated detection and measurement of 500 million
astronomical objects. The last few columns show the actual and projected
analysis usage of these large data sets at the JHU GrayWulf facility,
the Turbulence 1K, and estimates derived from the actual SDSS query
logs, corresponding to three different query sizes. It is important to
note (Fig 1b) that all the relevant data sets are between 10 and 100 TB,
due to disk space and data transfer limitations.

There are several interesting things to note: (i) the SC generated
projects have Amdahl numbers considerably below what the mainframe
hardware could deliver. The main reason is the limitations on the data
set sizes: if the simulations could have written a 500TB output instead
of 20-50TB, their Amdahl number would have been 10-20 times larger, and
(ii) the much higher Amdahl numbers on the right panel represent the
quite different needs of the data analysis phase. It is clear that these
require a different balance of CPU and IO than the main computational
facility. ![](media/image2.pct){width="5.152777777777778in"
height="1.8555555555555556in"}

##### Raw sequential IO 

For very large data sets the only way we can even hope to accomplish the
analysis if we follow a maximally sequential read pattern. Over the last
10 years while disk sizes have increased by a factor of 1,000, the
rotation speed of large disks used in disk arrays has only changed a
factor of 2 from 5,400 rpm to 10,000 rpm. Thus random access times of
disks have only improved about 7% per year.

The sequential IO rate has grown somewhat faster as the density of the
disks has increased by the square root of disk capacity. For commodity
SATA drives the sequential IO performance is typically 60MB/sec,
compared with 20MB/sec 10 years ago. Nevertheless, compared to the
increase of the data volumes and the CPU speedups, this increase is not
fast enough to conduct business as usual. Just loading a terabyte at
this rate takes 4.5 hours. Given this sequential bottleneck, the only
way to increase the disk throughput of the system is to add more and
more disk drives and to eliminate obvious bottlenecks in the rest of the
system.

##### Scale-up or scale-out?

Today's supercomputers typically dedicate a few percent of the nodes as
IO nodes, restricting the bisection bandwidth between compute and
storage to a small fraction of the computer's capability. 
Data-intensive scalable computers are primarily concerned with storage
processing, and need much higher storage bisection bandwidth.  So
traditional models attaching a few Fibre Channel disk arrays to a few
nodes and requiring all data to flow through these nodes before any
processing is ineffective.  Storage needs to be more widely distributed
in data-intensive scalable computers.

Today's data-intensive systems emphasize replicated commodity
components, modeled on the success of BeoWulf and Google clusters. 
Where simple filtering dominates data processing, for example, a cluster
of inexpensive nodes with a few inexpensive disks each can deliver
surprisingly high data rates inexpensively. Understanding the
limitations of such distributed storage architectures for future
generations of data-intensive applications is an important goal for
data-intensive systems \[Riedel98\], as well as the complexities of data
partitioning and management. Moreover, physical space efficiency and
power efficiency may benefit from a deeper exploration of
storage-in-node architectures exemplified by BeoWulf and Google
experience.

##### The Challenges of Building a Data-Intensive Exascale System

Adding more disk spindles per CPU is not the only change required.
Extremely large systems come with many other challenges, not seen in
smaller scale systems. To address these challenges the software dealing
with data-intensive applications must be rethought from ground up.

-   Unavoidably, petabytes require thousands of disks. Even though a
    mean-time-to-failure of each individual disk might be measured in
    tens of year, when combined, the system will experience hardware
    failures daily, or often hourly. Graceful, transparent recovery from
    such failures is therefore a must to ensure uninterrupted job
    execution. Data-intensive system software will need to meet its
    performance requirements while simultaneously recovering from many
    concurrent multi-TB disk failures -- this implies a revolution in
    the software design of the data storage system.

-   At extremely large scales, administrative costs that increase with
    the size of the data are unsustainable. Systems must be
    self-sustaining, self-healing, self-load-balancing and
    self-adjusting to avoid requiring a large number of administrators.

-   Large fraction of the computational challenges in astronomy in the
    next decade will be driven by complex analytics. An example of a
    modern analytics can be time series analysis, or complex spatial
    analysis involving densities. Such analysis often requires
    multi-joins of entire data sets, or cross-correlating entire
    catalogs. Performing such operations at extreme scale requires new
    approaches to analytics tuned for the scale.

-   Disk IO dominates the cost in data intensive systems. For this
    reason, aggressive data compression will start playing an increasing
    important role and will serve as a way to reduce IO and use
    frequently under-utilized CPUs. Given that scientific data sets are
    almost always write-once-read many, compression techniques such as
    delta compression will likely become increasing important.

-   In data intensive systems, where IO cost dominates and compute
    cycles are cheap, it is often the case that reconstructing
    intermediate data is preferred over storing it. In order to perform
    this reconstruction, accurate recording and management of the
    lineage and provenance of each data item is required. Capturing
    provenance in large scale systems is also essential in order to
    capture entire lineage of produced data.

-   At the extreme scale, it is essential to move computation to the
    data. This easy to understand paradigm often requires major
    rethinking of the entire architecture. This must be combined with
    parallelization. Scanning a single petabyte of data at a speed
    delivered by a single disk would take several years, thus reading
    and processing data in parallel becomes de facto a must.

-   At the extreme scale, it becomes worthwhile considering relaxing the
    accuracy of the answer - often obtaining a 100% precise answer can
    cost several orders of magnitude more than obtaining a 99.9% precise
    one. Taking into the account such precision, as well as accuracy of
    data measurements can lead to huge performance gains and cost
    reduction. Lossy compression techniques can similarly reduce storage
    and IO bandwidth costs.

-   System flexibility is becoming increasing important. At smaller
    scales, it was relatively easy to change the schema of the
    underlying data. At extreme scale, a system must be flexible to
    allow schema changes without too much cost.

# 2. SCIENCE GRAND CHALLENGES

Several complementary probes of dark matter and dark energy are planned
with the next generation surveys. Stage III surveys such as DES, PS1,
BOSS will be carried out in the next five years or so, while the more
ambitious Stage IV surveys (LSST, JDEM, SKA) will be completed in the
10+ year timescale. The science goals of these and other surveys can be
roughly categorized as:

-   Cosmic Microwave Background (CMB)

-   Extreme Environments

-   Large-Scale Structure, Galaxy Clusters and Baryon Acoustic
    Oscillations (LSS)

-   Weak Gravitational Lensing

### CMB Temperature and Polarization

The computational challenges posed by the analysis of a Cosmic Microwave
Background data set can be quantified by its numbers of observations in
the time domain (N~t~) and in the pixel (map) domain (N~p~). The first
is set by the duration of the mission and the numbers and sampling rates
of its detectors at each of its observing frequencies, and largely
drives the computational requirements of the data analysis. The second
is set by the angular resolution of the detectors and the fractional sky
coverage of the mission and primarily impacts the communication
requirements. The ongoing quest to measure ever fainter signals at ever
higher angular resolution has driven a steady and continuing increase in
both of these parameters.

Current satellite missions: WMAP is a NASA satellite mission that has
been surveying the full sky with 20 detectors at 5 frequencies at
moderate angular resolution. Launched in 2001, after its scheduled 9
years of operation WMAP will have gathered 2 x 10^11^ time samples and
mapped them into 5 x 10^7^ sky pixels. Planck is a joint ESA/NASA
satellite mission to survey the full sky with 74 detectors at 9
frequencies at high angular resolution. Scheduled to launch in spring
2009, over its two-year mission Planck will gather 6 x 10^11^ samples
that will be mapped into 8 x 10^8^ pixels.

Current/future suborbital missions: with Planck set to make the
definitive measurement of the CMB temperature anisotropies, the focus of
suborbital missions has shifted to CMB polarization, and especially the
lensing-induced B-mode polarization signal at small angular scales.
Since they can only observe part of the sky, suborbital experiments have
much lower pixel counts. Examples of these missions include the EBEX
long-duration balloon experiment, intending to fly in Antarctica in late
2010 and gather 3 x 10^11^ samples with 1,400 detectors at 3
frequencies, and the PolarBear and QUIET ground-based experiments to be
deployed in the Atacama Desert in the next 2-3 years and each use
several thousand detectors to gather 3 x 10^13^ and 10^14^ samples
respectively.

Future satellite missions: the large-scale CMB B-mode signal predicted
to be induced by gravity waves from the inflationary epoch will require
an exquisitely sensitive all-sky mission such as the CMBpol satellite
called for in NASA's "Beyond Einstein" program. Although this will
likely not launch before the end of the next decade, mission concept
studies typically envisage several thousand detectors observing at a
large number of frequencies for 2 years or more, gathering a data set of
a few x 10^14^ samples over a few x 10^9^ pixels.

#### Computational Challenges in CMB Data Analysis

Since CMB temperature and polarization signals are so faint, their
precise measurement requires 10^3^ - 10^5^ observations of each point on
the sky, so the analysis of a CMB data set is dominated by operations on
the N~t~ time samples. Each such sample includes CMB signal, foreground
contamination, and instrument noise, and these signals are correlated in
the multipole-, pixel- and time-domains respectively. Since we have to
account precisely for each of these correlations (and indeed the CMB
correlations are the fundamental measure of the data we are seeking) the
entire data set has to be treated as a single data object, precluding
the kind of divide-and-conquer analysis approaches used in other
domains.

The most computationally challenging operation on a CMB data set is the
mapping of the time-ordered data to the sky pixels, and in particular
properly accounting for the temporal correlations in the noise. Using
preconditioned conjugate gradient techniques, and exploiting the block
band Toeplitz structure of the time-time noise correlation matrix,
solving for the maximum likelihood map scales as O(N~t~) with a
pre-factor that includes the number of iterations (typically around 50
for a white-noise pre-conditioner). Such analyses are not just used to
make maps of the actual data but also of the tens of thousands of
simulated data sets required both for mission design and deployment and
for the estimation of the CMB power spectra and their uncertainties
using Monte Carlo techniques. Finally these analyses are inevitably
iterative, with each analysis uncovering systematic errors in the model
of the data and/or the instrument that have to be accounted for in the
subsequent re-analysis; the need for these iterative Monte Carlo
techniques will be even greater with the increased sensitivity to
systematic effects in the forthcoming generation of multiplexed
polarization experiments. Overall we can expect a combined
iteration/realization/reanalysis pre-factor of O(10^5^).

Over the next 15 years we expect the size of CMB time-ordered data to
grow by 3 orders of magnitude; coincidentally this exactly matches the
projected growth in computing power over the same period assuming a
continuation of Moore's Law. Since today's CMB data analyses are already
pushing the limits of current HPC systems, this implies that our
algorithms and their implementations will have to continue scaling on
the leading edge of HPC technology for the next 10 Moore-foldings if we
are to be able to support first the design and deployment of these
missions and then the scientiﬁc exploitation of the data sets they
gather.

### Extreme Environments

A number of facilities over the coming decade and beyond are targeting
extreme environment physics - black holes, pulsars, relativistic
outflows, gamma-ray bursts, the origin of the highest energy cosmic
rays, sources of gravitational waves, and cosmic neutrinos. The key
current and medium term future facilities are probably Chandra, XMM,
Fermi, and Auger; the most important future facilities under development
are IXO (X-ray), SKA (radio), LISA (gravitational waves), and ICE CUBE
(neutrinos); but many other facilities with a time monitoring capability
will play a role - notably Pan-STARRS and LSST, but also many smaller
telescopes with an alert-followup capability. As well as the intrinsic
astrophysical topics, these studies will have direct bearing on
fundamental physics - testing strong field GR, testing our understanding
of black holes, making the first measurement of gravitational waves, and
probing neutrino physics.

These projects all have significant data volume and processing
requirements, which add to the general points made elsewhere in the
report; however the special feature in this area is the fact that the
sky changes, on every timescale from milliseconds to decades. There is a
need to detect and recognise transients; to separate the rare
interesting ones from the millions of daily false alarms (flaring M
stars, moving asteroids, unexpected data \"junk\" etc); to mine the
evolving data base for more subtle variables; and to construct time
histories post hoc as required. These problems are with us now but
become far more challenging in very high rate data flows, partly of
because of the processing requirement and partly because the
contamination problems become much more severe with deeper data.

The prime need is for development of fast automated algorithms for
detection and classification of transients. The second need is improved
methods for trawling the very large databases that will result from
multiple time slices of the sky, with searches being run by many
simultaneous science users, often returning to raw pixel data. This will
probably require new more efficient techniques for structuring and
searching incremental databases, which will find commonalities with many
other areas of science and commerce.

### Baryonic Acoustic Oscillations

Oscillations of the coupled photon-baryon ﬂuid in the early Universe
imprint a scale on the clustering of matter. This baryon acoustic
oscillation (BAO) scale, set by the sound horizon scale at the epoch of
recombination, can be calculated from straightforward physics and
calibrated by its imprint in the CMB. In the galaxy correlation function
the BAO signature is a sharp local peak at 150 Mpc. In the galaxy power
spectrum, the Fourier transform of this peak appears as a series of
oscillations, analogous to but much smaller than the acoustic
oscillations in the CMB power spectrum. Measuring the BAO scale from
galaxy clustering in the transverse and line-of-sight directions yields
estimates of the angular diameter distance D~A~(z ) and Hubble parameter
H (z ), respectively. Determining these quantities with high precision
requires enormous survey volumes.

BAO measurements can be carried out from both spectroscopic and
multi-color imaging surveys. For the latter, accurate photometric
redshifts must be estimated from multi-color information. This is an
inversion of an inherently noisy system (from measured to physical
properties) that results in non-Gaussian errors. Propagation of the full
covariances (as a function of redshift and spectral type) is required to
fully model the uncertainties on the cosmological parameters. This
requires new computational and algorithmic approaches to inverse
processes and error propagation.

The measurement and analysis issues for BAO are addressed below since
they overlap significantly with large-scale structure.

### Galaxy Clusters and Large-Scale Structure

While BAO measurements use the standard ruler test to probe dark energy,
the full distribution of galaxy groups, clusters and large-scale
filaments constrains a wealth of information on dark energy and dark
matter. The structure of the halos of galaxies and galaxy clusters
provide particularly good tests of dark matter.

Galaxy clusters with masses in the range 10^14^-10^15^ solar masses are
useful probes of dark matter and dark energy. Clusters are the largest
virialized objects in the mass distribution and are a particularly
tractable target for observations of structure and its evolution over
cosmic time. They are observed at redshifts between 0\<z\<1 and are
sensitive to the geometry of the universe and the amplitude of mass
fluctuations. For any set of cosmological parameters, the growth of
cluster-sized dark matter haloes as a function of redshift and mass can
be precisely predicted from N-body simulations. Comparing these
predictions to observations of the real universe provides constraints on
cosmology. This depends critically on understanding the mapping between
predicted properties such as halo mass and the observed properties of
clusters, such as galaxy counts/velocities, X-ray emission,
Sunyaev-Zeldovich ﬂux decrement, or weak lensing shear.

Correlation function (or Fourier space spectra) measurements of the
galaxy distribution are sensitive to cosmology on large-scales and on
the physics of clusters and galaxies on small scales. With appropriate
parameterization, information on the cosmology and small-scale physics
can be extracted (see below).

### Weak Gravitational Lensing

Upcoming major WL surveys will be performed by Pan-STARRS, the dark
energy survey (DES), and ultimately by LSST from the ground and by JDEM
from space. Data from these surveys will accumulate over the next 20
years and will dwarf past and current efforts in the field by many
orders of magnitude.

These surveys will use the coherent distortion of images of distant
galaxies to determine the structure in the dark matter, which is
determined by the early universe, and also the growth of this structure
which is influenced by the current equation of state of the background,
i.e the dark energy, and is also affected by possible interactions in
the dark sector. Additional tests of cosmological theory come from the
dependence of the geometric path lengths on redshift.

With high precision and high accuracy measurements of cosmic shear it
will be possible to tightly constrain cosmological parameters; measure
evolution of the equation of state; test gravity on large scales;
provide clues to the nature of the dark matter through the effect of
self-interactions or interactions between the dark matter and dark
energy; and test theories of inflation from evolution of dark matter
structures.

Weak lensing, like other precision cosmology probes, is limited by the
understanding and correction of systematic errors. Weak lensing
measurements require exquisite understanding of the point spread
function (PSF) as the effects of any PSF anisotropy must be corrected
for to high precision. The PSF can be measured from stars, but the devil
is in the details of how this depends on the spectra of the galaxies,
which are different from the stars. These effects are the limiting
factor in current ground based surveys. Tackling this problem will
require the development of techniques for characterizing and modeling of
the instrumental PSF via interaction with data and simulations that
incorporate propagation of radiation through the atmosphere (for ground
based surveys) and through the instrument optics and the detectors. The
next generation of surveys will also require the development of
techniques for simultaneous multi-image object detection and shape
analysis. This poses a significant computational challenge as the
upcoming surveys will be generating peta-pixels of image data per year.

In addition to the challenge of making accurate measurements, there are
challenges in development of the theoretical predictions with which the
data will be confronted. Linear theory is not adequate on the scales of
interest, so simulations will be needed, and techniques will need to be
developed for propagation of light to generate synthetic images and
synthetic shear catalogs; this will also require simulations that
accurately model the sources. Generating and analyzing such high
precision and high dynamic range simulations spanning the range of
theoretical parameter space promises to be a greater computational
challenge than making the measurements. A further computational
challenge is in developing techniques for high dimensional model
parameter fitting.

Data Analysis Challenges

#### Systematic Error Issues in Survey Data Analysis 

Several of the science challenges discussed above are currently
dominated by systematic errors in the measurements. The challenge for
ground-based astronomy is due to the inherently uncontrollable
interference of the atmosphere. While ground-based facilities are
improving their characterization of the physical instrumentation, the
time-varying nature of the atmosphere makes precision measurements
extremely difficult. The stochastic contributions from the atmosphere
affect both the spectral response of the system, thus the photometry,
and also the point-spread function (PSF).

#### Reducing Systematics by Improved PSF Measurements

Perhaps the most critical source of systematic errors is the accuracy of
the determination of the point spread function (PSF) for real images.
Poor PSF models impact the quality of photometric, astrometric, and
morphological measurements of real objects in astronomical images.
Current state of the art PSF modeling consists of empirical measurements
from individual images using the available stars, coupled with modeling
of the effects of the telescope optics and atmosphere to constrain the
range of likely PSF models. The future missions will require more
advanced techniques, and these will be extremely data intensive. One
promising technique currently in consideration improves the PSF model of
individual images, and simultaneously measures the shapes and
brightnesses of images by performing a single global solution to all of
these parameters for a collection of overlapping images. To perform this
type of analysis on a reasonable timescale (months), on the PS4 /
LSST-scale datasets, will require 1-10 Petaflops operating on \~100 PBs
of data.

#### Precision Photometry and Modeling of the Atmosphere 

The photometric precision ground-based observations will continue to be
limited by our limited knowledge of the photometric response of the
atmosphere. Current techniques using the empirical measurements of the
impact of the atmosphere are reaching the limit of their effectiveness.
Future improvements will require coupling the on-site monitoring of the
atmospheric observables (temperature, pressure, humidity, temp vs
altitude, etc) to detailed modeling of the atmosphere. Observatories are
already forming partnerships with climate research groups; Future
efforts will require closer interaction between local climate modeling
and measurements from astronomical observatories. Modeling efforts
needed by the astronomical community will include both the modeling of
the spectral response of the atmosphere given the observable
constraints, but also constraints on the seeing and scattered light due
to the atmosphere.

#### Computational Challenges in Clustering Statistics and Parameter Estimation

The computational problems in several of the large-scale science probes
discussed above relate to the estimation of two-point and higher
correlations, inverting their covariance matrices, and in estimating
parameters from them. Upcoming surveys will provide rich information in
the redshift direction that spans a large fraction of the horizon
volume, as well as the two dimensions on the sky. We consider here the
generic analysis of data from imaging surveys that provide photometric
redshifts and angular positions for millions of galaxies (exceeding a
billion galaxies for Stage IV surveys) over a tenth to half the sky.

In the five year time-scale, Stage III imaging surveys will provide
auto- and cross-spectra (at least 5) for lensing and angular clustering.
These power spectra can be binned in at least 10 bins each in redshift
and angular scale. Counting all cross-spectra in redshift bins, we have
at least 5x50x10\~2500 binned spectra. Thus the calculation of
covariances is feasible as it requires inversion of matrices of order
10,000x10,000. However with Stage IV surveys, we must consider finer
measurements in both redshift and angular scale. This problem rapidly
reaches exascale proportions: with 100 bins each in redshift and
lengthscale, one has 5x50,000x100\~2.5 million binned spectra! So
calculation and inversion of covariances, common to the analysis of all
clustering probes, will requires major algorithmic and computational
advances. For higher order correlations this problem is significantly
harder.

Beyond two-point statistics, higher order correlations scale naively as
Nk where k is the order of the correlation function. For current data
sets 3-point statistics have been measured on 50-100,000 sources in
10,000 CPU hours (to estimate the covariance of these measures requires
multiple realizations of these or simulated data sets). Scaling to the
size of angular surveys in BAO and lensing surveys and to 4 and higher
order statistics is out of the range of current computational
facilities. To achieve these orders requires not just new computational
resources but also the development of approximation techniques where the
accuracy of the measure is limited by the desired accuracy on the
physical cosmological rather than the underlying shot noise.

The simultaneous estimation of cosmological parameters, small-scale
physics and systematic uncertainties in the measurement (such as shear
calibration errors in lensing) or model (scale and redshift dependent
biases of galaxies) is essential in making rigorous dark energy
measurements. So far the estimation of systematic uncertainties has been
done using approximate models, but upcoming surveys offer the
opportunity to estimate them from the data with minimal assumptions.
This however requires the ability to carry out the equivalent of Monte
Carlo Markov Chain parameter estimation over parameter spaces of very
high dimension. Current analyses work with order 5D parameter spaces.
Even with Stage III surveys, the goal will be to fit for up to 10
cosmological parameters and order 100 "nuisance" parameters: a
computational challenge that requires vast increases in computational
power and new algorithms. The number of nuisance parameters would be
even larger depending on one's degree of conservatism and with Stage IV
surveys.

# 3. EMERGING NEW SURVEYS AND COLLABORATIONS

The table below shows the trends in astrophysical data over the last few
decades. The exponential growth in data is quite obvious. In the
following section we discuss in some detail the main data sets relevant
to the current discussion that will emerge over the next decade and will
provide unique science opportunities, but will also require extreme
scale computational
capabilities.![](media/image3.png){width="6.059027777777778in"
height="3.2944444444444443in"}

## 

## Upcoming Optical Surveys

**BOSS, a part of SDSS-3,** is a next generation spectroscopic redshift
survey following up on the Sloan Digital Sky Survey (SDSS); it is
intended to measure structure in the distribution of baryonic matter via
direct redshift measurements of 1.5 million Large Red Galaxies (LRG) to
a redshift of 0.7 and via measurements of the "Lyman alpha forest" in
front of 160,000 QSOs to obtain the structure between redshifts of 2.3
and 3.3. BOSS observations are expected to begin in mid 2009, following
the completion of a new spectrograph for the APO/SDSS telescope. The
survey will span 2009-2014. BOSS is an order of magnitude larger than
existing surveys. The distribution of baryons, as seen in the LRG
distribution, shows a feature in its "correlation function," this sets a
scale which is directly correlated with the fluctuation power spectrum
of the early universe. At low redshift this scale was measured by SDSS
and 2dF to be 110 Mpc (Perceval, et al 2007MNRAS.381.1053P). Changes in
this scale as a function of redshift will directly show the growth of
these fluctuations, leading to greatly improved estimates of the
cosmological parameters.

The spectroscopic redshifts from BOSS will provide fundamental
calibrators for the photometric redshifts used by imaging based dark
energy experiments, such as PanStarrs and LSST.

**The Pan-STARRS Project PS1 survey** is a single 1.8m telescope with a
7 degree field of view imaged by a 1.4 Gpixel camera. This telescope
will perform a near-all-sky survey covering a 30,000 degree field over
3.5 years, with 60 visits distributed over 5 filters *(grizy).*
Including data from several deep fields with frequent re-visits, the
total raw data volume for this survey will be 1.5PBs. The Pan-STARRS
Project longer-term goal is to replicate the PS1 optical system and
camera multiple times. The project is on track to complete a 4-telescope
system (PS4) in the 2012 timeframe which will perform similar wide-area
and deep surveys over a 10 year mission lifetime. The PS4 data volume
will approach 20PB, and will result in \~500 repeated observations of
the 30,000 degree region distributed across the 5 filters. The PS1
survey depths are *g*=24.1, r=23.6, i=23.5, z=22.5, y=21.0. The PS1
telescope is currently located on the summit of Haleakala on Maui and
will begin survey operations in early 2009, while the PS4 telescope is
planned for the summit of Mauna Kea on the Big Island of Hawaii. The
science goals of the survey cover the full range of science from
cosmology to the structure of the solar system. Dark Energy and Dark
Matter studies will be enabled by the combinations of large-scale
structure observations, weak-lensing observations on the large-area
survey, and the observation of Type Ia supernovae.

**The Dark Energy Survey (DES)** is a Type III dark energy survey
comprising a wide field (5000 sq degrees) survey and narrow field (9 sq
degree) time-domain survey in four passbands g, r, i and z passbands.
With a 500 Mpixel camera it will achieve a data rate approaching 1 TB
per night and, over the 5 years of survey operations, will catalog and
archive approximately 1 PB of data with a depth of *g*=24.6, *r*=24.1,
*i*=24.3, and *z*=23.9. The science drivers are the study of dark energy
and dark matter using four primary probes: the evolution of the galaxy
cluster density with redshift (100,000 clusters to a redshift of
z\<1.3), Type Ia SNe as standard candles (1000 SNe at z\~1), weak
lensing at redshifts of z=1 and z=4 and Baryonic Acoustic Oscillations
at z\<1. Situated at the Blanco 4m telescope at Cerro Tololo
Inter-American Observatory (CTIO) science operations are expected to
begin in 2011 and run for 5 years.

**The Large Synoptic Survey Telescope (LSST)** is an 8.4m telescope with
a 3.5 degree field of view and a 3.2 Gpixel camera that will generate a
6 band (0.32-1.1 microns) survey of 20,000 sq degrees of the sky over a
period of 10 years. Returning to each patch of the sky approximately
1000 times in 10 years it will reach a limiting magnitude of r\~27.5.
The data rate for the LSST will be approximately 30 TB of imagery per
night that will be analyzed in real time and will result in an archive
of 30 PB of data in the form of catalogs and 60 PB of imagery and 270
Tflops for archive operations (after 10 years). A Type IV dark energy
survey the primary science goals of LSST are the study of dark energy
and dark matter using gravitational weak lensing (with a density of 55
sources per square arcmin at a median redshift of z=1.2), Type Ia
supernovae (10^6^ SNe) , Baryon Acoustic Oscillations. The size of the
LSST data set will, however, enable a broad range of ancillary science
ranging from a census of the Solar System, studies of Galactic structure
and probes of the high energy transient universe. LSST is slated to
begin early science operations in 2015 and enter its survey mode in
2016.

**The Joint Dark Energy Mission (JDEM)** is an Einstein probe that is
being developed in a partnership between DOE and NASA. Currently in the
process of defining a mission statement a number of proposals have been
made for JDEM to study dark energy using Type Ia SNe, Baryon Acoustic
Oscillations and gravitational Weak Lensing. These include a
near-infrared slitless spectroscopic survey (targeting H-alpha) of 100
million galaxies at 1\<z\<2 covering 28,000 sq degrees, optical and
infra-red imaging over several thousand square degrees for weak lensing
to a depth of approximately J=25(30 galaxies per sq arcmin) and a slit
or slitless spectroscopic survey of Type Ia SNe to z\<1.7. The imaging
data volumes and resulting archives present similar challenges to those
from DES and LSST. As a space-based mission the telemetry data rate and
on-board processing present significant computational and algorithmic
challenges. JDEM is exected to begin operations around 2016

##### Radio Surveys and The High Redshift Universe

Two major international facilities - the International X-ray Observatory
(IXO) and the Square Kilometre Array (SKA) - being developed over the
next decade are aimed at bridging the gap between our knowledge of the
very early universe and the current epoch. The challenge is to find and
study the very earliest objects to condense out of the cosmic soup - the
first stars, protogalaxies and accreting black holes - and to map out
the physical and magnetic structure and evolution of the Inter Galactic
Medium out of which these objects are forming. This will connect first
principles physics to observable structures and so constrain that
physics and the cosmic ingredient list - e.g. the type of dark matter -
and for the first time attempt to explain the origin of cosmic
magnetism.

The biggest computational challenge comes from SKA, because of the
requirement to collect a high bandwidth time series from each of many
elements in the array, and to produce a skymap by Fourier Transforming
these \"fringes\". The challenge will build up gradually as SKA is being
conceived as a sequence of pathfinders and intermediate projects (e.g.
LOFAR and PREPSKA) starting in 2009 and continuing through to 2021. By
2018, SKA estimates data volume of 100PB/day, and a processing
requirement of 20Tflops, figures that are only just plausible with
Moore\'s Law type extraopolations. The processing requirement would be
many times worse if, as is traditional in radio astronomy, users were to
be given the facility to re-compute on demand many different skymaps
from the fringe data, using different assumptions. It may even be more
cost effective to throw away the fringe data and re-observe the sky
rather than re-compute. There is therefore an urgent need to understand
how to make the FT calcns as fast as possible, either by algorithm or by
dedicated machine architecture, how to provide compute-on-demand, and
where the cost-performance drivers are.

# 4. PRIORITY RESEARCH DIRECTIONS/COMPUTATIONAL CHALLENGES

## Algorithms and Architectures

  -----------------------------------------------------------------------
  3yrs                    6yrs                    10yrs
  ----------------------- ----------------------- -----------------------
  Bring more community    Provide better          Have community
  codes to Petaflop scale characteri-zation of    
  of 10^5^ way            extreme scale           
  parallelism             programming models with 
                          simulators              

  Develop methods capable Implement 10^6-7^ way   codes ready to exploit
  of using 10^6-7^ way    parallel community      
  parallelism             codes                   

  Develop best practices  Learn from best         Extreme-scale
  for PB scale analysis   practices, develop,     
  systems, encourage      build and test          
  diversity               solutions scalable to   
                          10PB                    

  Develop scalable        Run these algorithms on computing
  statistical algorithms  extreme datasets        
  for the astro                                   
  challenges (100D                                
  parameter estimation,                           
  clustering analysis)                            
  -----------------------------------------------------------------------

## Action Items

  -----------------------------------------------------------------------
  3yrs                    6yrs                    10yrs
  ----------------------- ----------------------- -----------------------
  Expand DoE CSGF         Develop                 Support a wide range of
  program. Engage DoEd    Lab/Uni/Industry        extreme
  (GAANN) and NSF in      partnerships for        
  supporting graduate     extreme scale era       
  training grants for                             
  Computational Science                           

  Expand Computational    Develop National        scale applications,
  Science Summer schools  Extreme scale graduate  
  with focus on extreme   program linking labs    
  scale in participating  and universities        
  labs                                            

  (Re)start broad         Build shared large DISC tools
  inter-agency program in system(s) to be used    
  Extreme scale computing for science analyses    
  similar to HPCC of a                            
  decade ago                                      
  -----------------------------------------------------------------------

# 5. CROSS-CUTTING RESEARCH DIRECTIONS

### Developing a Common Data Access Platform: SCIDB

Existing off-the-shelf data management systems no longer meet the
extreme requirements of data-intensive scientific and industrial users.
As a result, majority of existing data intensive systems both scientific
and industrial are currently relying on custom, home grown solutions.
\[ref: xldb1, xldb2\]. Due to increasing complexity and cost, building
such systems from ground up per project is becoming impossible, and
developing a common scalable data management system, defining common
data models, standards and interfaces become essential. Involved tasks
include

> 1\) exploring commonalities and improving collaboration
>
> 2\) understanding new technologies and emerging systems.

Re 1: we should explore commonalities and differences between large
scale commercial systems and scientific systems from different domain.
We should work closely with researchers and academics focusing of DMS
issues, as well with vendors including database vendors. This
exploration and collaboration has already started through the series of
Extremely Large Databases Workshop, organized by SLAC.

Re 2: we should understand and evaluate applicability of the new
technologies and emerging systems, such as the SciDB project \[ref:
scidb CIDR paper\]. SciDB is a new project which aims to build a new
open source science database, with main focus on complex analytics,
scalability, new data model, improved interfaces, provenance and data
uncertainty.

### The Next Generation of Computational Scientists

The success of computational science within DOE is critically dependent
on the development of a workforce that can exploit the new generation of
experiments and surveys. This new generation of scientists needs to be
as comfortable in Computer Science, Statistics and Applied Mathematics
as they are in Physics or Astrophysics. To accomplish this requires a
concerted and sustained effort to develop and train a workforce that is
computationally and mathematically literate. As an initial focus,
graduate programs in cross-disciplinary research (such as computational
graduate fellow CSGF) need to be created, supported and expanded as well
as the development of cross-disciplinary summer schools and workshops.
This should be undertaken as a multi-agency initiative (including DOE,
NASA and NSF) and should be in partnership with high-tech companies
(Google, Microsoft, Yahoo, Amazon, E-bay) who require a work force with
data intensive and compute intensive skills.

It should be noted that this is not just a question of increasing the
skills of the next generation of scientists. If they are taught to think
about data and computing new ways they will likely uncover new physical
relations within the data that were not envisaged when the experiments
were defined. New discoveries often come at the edges.

An ongoing concern is the lack of career paths for scientists who work
at the boundaries between fields. Departments are often reluctant to
embrace scientists whose expertise is not easily compartmentalized (a
Physics department may see the researcher as a computer scientist while
computer science may regard them as a Statistician and Statistics as a
Physicist). Departments can be encouraged to view these computational
cross-disciplinary scientists as integral to their departments if grant
programs exist to explicitly support there research effort.

# 6. REFERENCES

Erik Riedel, Garth Gibson, Christos Faloutsos, \"Active Storage For
Large-Scale Data Mining and Multimedia\" Proceedings of the 24th
International Conference on Very Large Databases (VLDB \'98). New York,
NY. August 1998.

The BOSS web site is:
[www.sdss3.org/cosmology.php](http://www.sdss3.org/cosmology.php)

Bell, G., Gray, J. & Szalay, A.S. 2006, \^?Petascale Computational

Systems: Balanced Cyber-Infrastructure in a Data-Centric World\^?, IEEE
Computer, 39, 110

Diemand, J., Kuhlen, M., & Madau, P. 2007a, ApJ, 657, 262

Diemand, J., Kuhlen, M., & Madau, P. 2007b, ApJ, 667, 859

Diemand, J., Kuhlen, M., Madau, P., Zemp, M., Moore, B., Potter,D., &

Stadel, J. 2008, Nature, 454, 735

Kuhlen, M., Diemand, J., & Madau, P. 2007, ApJ, 671, 1135

Kuhlen, M., Diemand, J., & Madau, P. 2008, ApJ, 684

Springel, S.D.M. White, C. S. Frenk, et al. 2008, Nature, 456 (7218) 73

Szalay, A.S., Gray,J. 2006, Nature, 440, 23