Taming theALMA data avalanche
Felix Stoehr
Subsystem Scientist of the ALMA Science Archive

IAU General Assembly, SpS15

Beijing, 28.8.2012

Outlinel

ALMAl

The data challengesl

Outlookl

Who? Too many to name them on a slide. The credit goes tothe whole ALMA team.

IAU General Assembly, SpS15

Beijing, 28.8.2012

ALMA

IAU General Assembly, SpS15

Beijing, 28.8.2012

ALMA First LightIAU General Assembly, SpS15

Beijing, 28.8.2012

ALMA in a nutshelll

66 antennas at 5000 m elevation in the Atacama Desertl

Interferometry with baselines up to 16 kml

Wavelength range from 3 mm to 400 µm (84 to 702 GHz)l

Built by ESO, NRAO, NAOJ in cooperation with Chilel

Timeline:l

Early Science Cycle 0 observing is under wayl

Early Science Cycle 1 Call for Proposals very successfullCycle 1 observing around beginning of 2013lCompletion of the full array: second half of 2013

IAU General Assembly, SpS15Beijing, 28.8.2012

The data challenges
1 Data rate: 200TB/yr with peak rates of 10x average
2 Data storage: “All data taken have to be stored forever”
3 Archive copies: full copies on four continents
4 Operation: general-purpose telescope
5 Pipeline: science-grade products for all scientific data
6 Support: computing power and hands-on help for scientists

IAU General Assembly, SpS15

Beijing, 28.8.2012

1 Data ratelll

The theoretical maximum data rate of ALMA correlator is 15PB/yr = 512MB/s = 4096 Mb/s (currently 1/8th but upgrade is easy)
Based on the predicted data rate requirements of theexpected type of science observations the official averagedata rate has been set to 200TB/yr = 6.6MB/s = 50Mb/s
The NA ARC (M. Lacy, D. Halstead) updated these values withthe experience gained through Cycle 0. The expected datarates are higher: up to 700TB/yr = 23MB/s = 190Mb/s

https://science.nrao.edu/facilities/alma/naasc-memo-series/naasc-memos/110.naasc-data-ratesl

The peak data rate is 10 times the average rate and the system
was designed to support this rate for long periods

IAU General Assembly, SpS15

Beijing, 28.8.2012

1 Data ratelll

Choice: the system was designed to contain no hardbottlenecks. Most systems are build out of commodityhardware. It can grow naturally to support higher rates ifneeded.
Lessons learned: so far, this seems to work out very well
Way forward: monitor the actual usage very closely andupgrade as required

IAU General Assembly, SpS15Beijing, 28.8.2012

2 Data storagel

ALMA storage solution requirements:ll

Cost-effective, scalable, future-proof, PB-scale
High read/write performance despite large variation in filesizes (a few kB up to hundreds of GB)l

Large I/O speeds to the pipeline processingl

Built-in file management layer (database)l

Automatic consistency checksl

Support globally distributed archives

IAU General Assembly, SpS15

Beijing, 28.8.2012

2 Data storagel

Choice: NGAS http://www.eso.org/projects/dfs/dfs-shared/web/ngasl

Developed by ESO for the VLTlCombined hardware/software solutionlCommodity-hardware approachl

Provide online processing capabilitiesl

Portablel

Powerful plugin architecture (archiving, processing, ...)IAU General Assembly, SpS15

Beijing, 28.8.2012

2 Data storagellLessons learned: ALMA was able to reuse the existing NGAS
solution and ESO had substantial experience running it.
However, improving the I/O performance for small files, writing
the multi-stream mirroring and making it robust to network
failures was considerably more work than anticipated.
Way forward: continuously increase storage capacity matching
the actual data rate. Avoid (external) vendor lock-in to allow toadapt smoothly to new technologies.

IAU General Assembly, SpS15

Beijing, 28.8.2012

3 Archive copies

IAU General Assembly, SpS15

Beijing, 28.8.2012

3 Archive copiesl

The main archive is in Santiago de ChilelFull copies in Charlottesville (NA), Mitaka (EA), Garching (EU)ll

The copies serve as backup of the main archive and allow theARCs (ALMA Regional Centre) to provide user-support,reprocessing, quality control, phase 3
P2P proposal of the NA ARC (M. Hatz, M. Lacy, D. Halstead) toreduce network transfer cost

IAU General Assembly, SpS15

Beijing, 28.8.2012

3 Archive copies

1/3
1/3

IAU General Assembly, SpS15

1/3Beijing, 28.8.2012

3 Archive copieslll

Choice: VPN network transfer instead of disk shipping
Lessons learned: also from past experience at ESO we knowthat wherever possible, network transfer should be used.
Individual files arrive much faster, it is much less work-intensive,
much more robust and in the end more cost-effective.
Way forward: implement the NA proposal to reduce network
cost in Chile

IAU General Assembly, SpS15

Beijing, 28.8.2012

4 Operationl

l

ALMA is a general purpose telescope with standard calls for
proposals once a year
ALMA will support a large variety of different types ofobservations (continuum, line, spectral sweep, mosaic, solar)
with different observing settings (12m array/ACA/TP) in different
configurations

IAU General Assembly, SpS15Beijing, 28.8.2012

4 Operationl

ALMA work-flow (courtesy Erich Schmid)
Phase I

Observing
ProgramPhase II

Scheduling
BlocksSchedulingControl

Principal
Investigator

ArchiveScience
Pipeline/
CASA

Correlator

Quality
Assurance

Telescope
Calibration
Quicklook

Operator/AoD

IAU General Assembly, SpS15

Beijing, 28.8.2012

4 Operation
l

l

l

Choice: fully integrated system with the Archive in the centerusing all the same interfaces has been built. A sophisticatedobserving tool and science data model (ASDM = ALMA Science
Data Model) have been developed.
Lessons learned: so far, we think that the choices were the rightones. The performance of the XML metadata database turns
out to be rather low.
Way forward: The ALMA Science Archive including VO services
is being built. Metadata will be harvested into denormalized,
search-optimized relational tables. In the longer term we will
move at least parts of the metadata to a relational database.

IAU General Assembly, SpS15

Beijing, 28.8.2012

5 PipelinelllThe amount of data of a single project can easily exceed thedata-reduction resources available to a typical user today
A fully automatic pipeline is required for ALMA deliveringscience-grade products (a first in radio astronomy)
Policy: All ALMA science data will be reduced by the project

IAU General Assembly, SpS15

Beijing, 28.8.2012

5 Science grade?
l

What do PIs do?
TodayTimel

As telescopes will become more complex and deliver moredata, telescopes will move to deliver science-grade products

IAU General Assembly, SpS15

Beijing, 28.8.2012

5 Pipelinelll

Choice: An analysis software package CASA (CommonAstronomy Software Applications) was written (nearly) fromscratch. A pipeline (heuristics + CASA) was developed.
Lessons learned: so far this approach looks very good
Way Forward: Commission the pipeline and bring it intooperations.

IAU General Assembly, SpS15

Beijing, 28.8.2012

6 Supportl

l

l

l

l

If users want to re-analyze the data e.g. an optimized reduction
for their particular science case, they need help.
ALMA is meant to be a telescope that can be used by all
astronomers, not only those that have a radio background
Choice: the three ARCs provide face-to-face support.
Lessons learned: already now, despite the small data sizes, theface-to-face support is highly appreciated.
Way Forward: provide a visualization solution (developmentplan). Provide the possibility for users to submit pipeline jobs withdifferent parameter settings on the ARC processing clusters.

IAU General Assembly, SpS15Beijing, 28.8.2012

Long-term evolutionll

ALMA operations are expected to last many decades.
Scalability and flexibility in hardware and software areextremely important.
For the areas that do not scale with Moore-type laws,
parallelization/clustering is needed.
Moore-type law?
Hard disksyesBandwidthyesSingle TCP stream

noCPU power

yesSingle-core speed

noDatabase rows

noIAU General Assembly, SpS15Beijing, 28.8.2012

Impact of science on data rates (and vice versa)l

Science influences data ratesll

We do not know which data rates we will get: They
depend hugely on the science that gets accepted
The new analysis shows that scientists are requesting more
wavelength ranges to be observed simultaneously than
anticipated and on-the-fly interferometry which is very
data-intensive is not yet implemented

IAU General Assembly, SpS15

Beijing, 28.8.2012

Impact of science on data rates (and vice versa)lData rates influence sciencelllProbably many/most scientists will not reduce the datathemselves any more or at maximum tweak the imaging
Extremely large amounts of 3-4 dimensional data
available (6mas resolution at 675GHz, up to 7680
channels, full polarization, up to 32 spectral windows
simultaneously)
The data is very well described which (hopefully) will allow
for very powerful searches (data mining)

IAU General Assembly, SpS15

Beijing, 28.8.2012

Summarylll

High data rates and the long-term nature of ALMA are a real
challenge and a huge effort was deployed at all ends to builda system that can handle the data intake.
Hard and software solutions have been built to be scalable andflexible to allow to adapt to change.
The goal of ALMA is to help the scientists wherever possible fromproposal preparation over the science-grade data products
and data-reduction to archival research and to work towards agreat end-to-end user-experience

IAU General Assembly, SpS15Beijing, 28.8.2012