The ALICE computing upgrade
project andnetwork in Asia
Tatsuya Chujo(for the ALICE collaboration)
AFAD 2015
6th Asian Forum for Accelerators and Detector
January 26, 2015, NSRRC, Taiwan

LHC experiments @ CERN
Lake Leman

Geneva airport (GVA)

Point 8LHCb

Point 1ATLAS

Point 5CMS

CERN

Point 2ALICE

~27 km tunnel,
4 major experiments

ALICE Experiment16m x 16m x 26m, 10,000 tons

>1400 scientists from 149 Institutes
in 40 Countries
The dedicated experiment in LHC experiments to study
Quark Gluon Plasma (QGP) by using heavy ion beams.

18 different
sub-detectors:
tracking, particle
identification, energy
measurement,
event trigger
3

What is Quark Gluon Plasma (QGP)?
gluonquark••

De-confined state of quarks and gluon inside hadrons under the extremely
high temperature and energy density•

New and still unknown state of matter.

Lattice QCD calculations:••

Critical temperature: Tc = 150-200 MeV
Crossover phase transition from hadronic phase to parton phase.

QuarkGluon
Plasma（QGP）
Time: few μ sec after the bigbang.
Temperature: 2 Trillion K
Energy density: > 1 GeV/fm35ALICE data collection (current)
Nominal LHC beam crossing at 40 MHz
ALICE:
Multi-level trigger system needed:
40 MHz → a few kHz

Single Pb-Pb collision events(√sNN = 2.76 TeV)

Online:1) Reject background2) Select most interesting
interactions
3) Custom computer toreduce the total datavolume
6

Data rate and data volume
Beam type

Recording(Events/s)

Recording(MB/s)

Data archived(PB per year)

ALICE

Pb-Pb
(one month)

200

1250

2.3

ATLASp-p(~8 months)

100100

6.0

CMSp-p(~8 months)

100100

3.0

LHCbp-p(~8 months)200401.0•

Pb-Pb events are much heavier than pp ones.

•ONLINE selection + OFFLINE processing →Final data volume archived.

•Each experiment has its own online-offline strategies.
7

Data archiving

LHC start taking data•

Experiments production
data in CASTOR: LHC
contribution is the mainsignificant one (data takingstarts at the end of 2009)•LHC contributed with ~12
PB/year between 2010 and2013•Nearly 80 PB ofexperiments data currently
stored in CASTOR's tapes

8Computing model (Run1 and Run2, -2018)slide by T. Sugitate9

ALICE upgrade (2018-)
ALICE upgrade; high rate capability✓GEM-TPC continuos high rate readout✓ ITS Silicon high rate readout✓DAQ (RCU etc.)

For LHC high luminosity upgrade, Pb-Pb@50kHzRecord all MB events, x100 statistics（Unique capability in ALICE）

Standard GEM
Pitch=140μm
Hole φ=70μm→Access to high precision
measurements and rare probes
Physics Goals:
Measure- heavy quarks, photons, lepton pairs
azimuthal anisotropy- Jet w/ PID hadron simultaneously
10ALICE Upgrade Plans: (2018-)•Current: reducing the event rate from 40 MHz to ~ 1 kHz– Select the most interesting particle interactions– Reduce the data volume to a manageable size•

After 2018:– Much more data (X 100) because:• Higher interaction rate
• More violent collisions → More particles → More data (1 TB/s)• Physics topics require measurements characterized by;– Very small signal/background ratio → large statistics– Large background → traditional triggering or filtering techniques very
inefficient for most physics channels• Read out all particle interactions (PbPb) at the anticipated interaction rateof 50 kHz• No more data selection– Continuous detector read-out– Read-out and process all interactions with a standard computer farm.
– ~1,500 nodes with the computing power expected by then➡ Total data throughput out of the detectors: 1 TB/s

11Expected data bandwidth (after 2018-)
Detector

Input to Online
System (GB/s)

Peak Output toLocal Data
Storage (GB/s)

Average Output
to Computing
Center (GB/s)

TPC

1,000

50.0

8.0

TRD

81.5

10.0

1.6

ITS

40

10.0

1.6

Others

25

12.5

2.0

TOTAL

1,146.5

82.5

13.2

Note: LHC luminosity variation during fill and efficiency taken into accountfor average output to computing center
12

The ALICE Online-Offline (O2) Project• From Detector Readout to Analysis:• What is the “optimal” computing architecture?
-Handle >1 T Byte /s detector input
Support for continuous readout
Online reconstruction to reduce data volume
Common hardware and software systemdeveloped by the DAQ, HLT, Offline teams
13

Functional Requirements of the O2 system✓ Data fully compressed before data storage.
✓ Reconstruction with calibrations of better quality.
✓ Grid capacity will evolve much slower than the ALICEdata volume.
✓ Data archival of reconstructed events of the current year
to keep Grid networking and data storage within ALICEquota.
✓ Needs for local data storage higher than originally
anticipated14Basic idea of the O2 system
Detectors electronics
Continuous and triggered streams of raw data

1.2 TB/s

Readout, chopping, and aggregation
Pattern recognition and calibration
Local data compression
Compressed Sub-Timeframes
Data aggregation
Synchronous global reconstruction,
calibration and data vol. reduction
Compressed Timeframes

60 GB/s

Data storage
and archival
Compressed TimeframesAsynchronous and refined calibration,
reconstruction
Quality control - Event extraction

Reconstructed	  events

15

The ALICE O2 Hardware Architecture

16Computing model (Data flow)
AcronymDescriptionRAW

Raw data as it comes fromthe detector.

CTF

Compressed Time Frame
containing the history ofOM(100 ms) of detector
readout information in theform of identified clusters
that belong to identified
tracks.
Event Summary Data.

ESDAOD

Analysis Object Data forphysics analysis.

HISTO The subset of AOD

information specific for agiven analysis.

MCMontecarlo simulation

by Pierre Vande Vyvre
17

Computing model (O2 processing flow)
Facility

Function

O2

ALICE Online-Offline Facility at
LHC Point 2.
Online reconstruction during the run.
Provides data storage capacity. Afterdata taking: runs the calibration andreconstruction tasks.

T0CERN Computer Center facility
providing CPU, storage and archiving
resources.

T1

Grid site connected to T0 with highbandwidth network links (100+ Gb)
providing CPU, storage and archiving
resources. Reconstruction andcalibration tasks

T2

Regular grid site with good network
connectivity (10+ Gb); runningsimulation jobs.

AFDedicated Analysis Facility of HPC
type that collects and stores AODs
produced elsewhere and runs theorganized analysis activity.

by Pierre Vande Vyvre- Maintain the advantages of the Grid and theanalysis trains- Make it more open and more effective18

Detectors	  electronics
TPCRaw	  data	  input

…

ITS

…

Trigger	  and	  clock

TRD

Detector	  data	  samples
interleaved	  with	  synchronizedheartbeat	  triggers

FLPs
FLPs Buﬀering

O(100)

Local	  processing

O2 system (1)QA

Local	  aggregaDon

Synchronous data flow
and processing

Time	  slicing

Frame	  dispatch

Global	  processing

SynchronousSub-­‐4meframes
Data	  ReducDon	  0	  CalibraDon	  0	  	  	  e.g.	  clusteringon	  local	  data,ie.	  par@al	  detector

TaggingPar4ally	  compressedsub-­‐4meframes
EPNs

O(1000)Timeframe	  buildingSub-­‐4meframes
Timeframes
Compressed	  4meframes
AOD
Data
Quality	  Control

QA	  data

Full	  4meframe
Detector	  reconstrucDon
e.g.	  track	  ﬁnding

CalibraDon	  1	  	  on	  full	  detectors
e.g.	  space	  charge	  distor/on

CCDB	  Objects

Data	  ReducDon	  1
QA

CondiDon	  	  &	  CalibraDon	  Database

Compressed	  4meframes

Storage

O2/T0/T1

T0/T1
Storage

Archiveby Pierre Vande Vyvre

19

The ALICE O2: Data Inputs
Detectors	  electronics

Raw	  data	  input
Detector	  data	  samples
interleaved	  with	  synchronizedheartbeat	  triggers

FLPs

by Pierre Vande Vyvre (modified)

TPC

…

ITS

…

Trigger	  and	  clock

TRD

O(100)

Buﬀering

QA

Local	  aggregaDon

QA	  data

Data
Quality	  Control

Time	  slicing

First	  Level
Local	  ProcessingSub-­‐%meframes

Data	  ReducDon	  0	  	  e.g.	  TPC	  clustering
Tagging

Calibra@on	  0	  	  on	  local	  data,ie.	  par@al	  detector

to Event Processing Nodes (EPNs)

Condi7on	  	  &	  Calibra7on	  Database- Handle > 1 TByte/s input from detectors- Data Links, Receiver Card, First Level Processor
20

The ALICE O2: Data Reduction (I)from	  First	  Level	  Processors	  (FLPs)

Par%ally	  compressedsub-­‐%meframes

EPNs

by Pierre Vande Vyvre (modified)

Timeframe	  building

O(1000)

Full	  %meframe

Global	  processing

Detector	  reconstrucDon

CalibraDon	  1	  	  on	  full	  detectors

e.g.	  track	  ﬁndinge.g.	  space	  charge	  distor/on

Data	  ReducDon	  1Condi7on	  	  &	  Calibra7on	  DatabaseSub-­‐%meframes
compressed	  %meframes
AOD	  etc.

QA
Compressed	  %meframesto	  Storage- Reduce data from > 1 TB/s to ~80 GB/s for storage- Only possible with online reconstruction

21

The ALICE O2: Data Reduction (II)slide by A. Uras (IC3INA 2013)
22

The ALICE O2: Data Reduction (III)•Data compression factors ranging from 2 to 20 according to the detector•

TPC still accounts for 60% of the total event size
slide by A. Uras (IC3INA 2013)
23

The ALICE O2: Data Storage

by Pierre Vande Vyvre (modified)from	  Event	  Processing	  Nodes	  (EPNs)
Compressed	  %meframes

O2/T0/T1

Storage

T0/T1

Storage

Archiveto	  Reconstruc7on	  Pass•Data in “intermediate” formats (not directly usable for physics analysis):- 80 GByte/s peaks to be handled, distributed over ~1,250 nodes- Average load of 15 GByte/s- Local storage in O2 system- Permanent storage in computing center•Data in “final” formats (usable for physics analysis):- GRID storage, accessible by experiment's users

24

O2/T0/T1
Storage

T0/T1
ArchiveStorageO2 system (2)Compressed	  *meframes

Reconstruc7on	  passes	  	  and	  event	  extrac7on

Analysis

Asynchronous

O2/T0/T1

O(1)

AOD

Globalreconstruc@on

Calibra@on	  2

Event	  extrac@on
Tagging
AOD	  extrac@on

Sub-­‐4meframes
Timeframes
Compressed	  4meframes
AOD

QA

Analysis	  Object	  Data	  	  	  (AOD)
Analysis	  Facili7es

O(1)Analysis

Histograms,trees

T2

Reconstruc@on
Event	  building
AOD	  extrac@on

Compressed	  *meframes

Simula@on

QA	  data

Data
Quality	  Control

Storage
CCDB	  Objects

AOD

Simula7on

Asynchronous data
flow and processing

ESDO(10)

CondiDon	  	  &	  CalibraDon	  Database

Storage
QAby Pierre Vande Vyvre(modified)25

Network in Asia andJapan for ALICE26

Hiroshima T2 Approaching to LHCONE

core node/DC
40-10Gbps
edge node/DC
40-2.4Gbps

slide by T. Sugitate

Present in SINET-4 (-2015)n Hiroshima DC: 40Gbps Core noden T2 to KEK via Hiroshima DC on1Gbps-MPLS of HEPNet-Jn Internat’l connection via a default
SINET routing
New SINET-5 announced by NII (2016-)n Domestic nodes at 100Gbps, andupgrade to 400Gbps/1Tbps latern Direct links to US/EU at 100Gbps
n Approach to LHCONE at 10Gbps

Possible scenarios on ALICE network• 2015-2017:– LHC Run2 data taking, x2 more heavy ion data.
– Data analysis on Run-2 (+Run-1)– Almost no change from Run-1 scheme. Due to data larger data volume,
network traffic in Asia will increase, at least x2.

• 2018- beyond:– LHC Run-3 data taking, x100 more data.
– Architecture change (O2) applied.
• O2, AF, and T0/T1/T3 scheme.
– Significant data reductions, reconstruction in O2 mainly, and analysis →reduce data volume.
1. Can keep the similar network traffic as Run-2?
2. Or if have AF in Asia (using HPC), then it will need more network traffic
than that in Run-2 → Accelerate local physics analysis in Asia.

➡ Direct links to US/EU at 100Gbps may be necessary in case of Japan?
28

Summary and Outlook• ALICE computing upgrades on online-offline for thedata taking after 2018 is ongoing.
• Continuous minimum bias event readout at 50 kHz in Pb-Pb collisions.
• 1 TB/s raw data from detector, need a significant data reduction down to80 GB/s to storage, and make a physics outputs timely.

• O2 Scheme: Online reconstruction and calibration by O2 (near ALICE) &T0/T1, organized analysis at Analysis Farm (AF), and simulation at T2.

• Designing based on Physics requirements is completed.
• Intensive works on, modeling, Technologies (processingplatform & network), O2 prototyping.
• Technical Design Repot (TDR) is progressing. It will be
submitted to LHCC in April 2015.

29

ALICE O2 Project
Project Organization
PLs: P. Buncic, T. Kollegger, M. Krzewicki, P. Vande Vyvre
Computing Working Group (CWG)
1.
Architecture2.
Tools & Procedures
3.
Dataflow
T. Breitner →4.
Data Model
5.
Computing Platforms
6.
Calibration
7.
Reconstruction
8.
Physics Simulation
9.
QA, DQM, Visualization
10.
Control, Configuration, Monitoring
11.
Software Lifecycle
A. Grigoras →12.
Hardware13.
Software framework

Chair
S. Chapeland
A. Telesca
I. Legrand
A. Gheata
M. Kretz
C. Zampolli
R. Shahoyan
A. Morsch
B. von Haller
V. Chibante
D. Berzano
H. EngelP. Hristov

Editorial Committee
L. Betev, P. Buncic, S. Chapeland, F. Cliff, P. Hristov, T. Kollegger, M.
Krzewicki, K. Read, J. Thaeder, B. von Haller, P. Vande Vyvre
Physics requirement chapter: Andrea Dainese
ALICE O2 project | January 2015 | Pierre Vande Vyvre

O2 CWGs

O2
TechnicalDesignReport30

Back up31

Computing model (Data flow)
AcronymDescriptionPersiste
ncy

RAW

Raw data as it comes from the detector.

Trans.

CTF

Compressed Time Frame containing the Persist.
history of OM(100 ms) of detector
readout information in the form ofidentified clusters that belong toidentified tracks.

ESD

Event Summary Data. Auxiliary data to
CTF containing the output of thereconstruction process that assigns
tracks to vertices and identifies theindividual collisions.

AODAnalysis Object Data containing the final Persist.
track parameters in a given vertex andfor a given physics event. AODs arecollected on dedicated facilities forsubsequent analysis.

HISTO

The subset of AOD information specificfor a given analysis. Can be generated
during analysis but needs to be
offloaded from the Grid.

Temp.

MC

Simulated energy deposits in sensitive
detectors. Removed once thereconstruction of MC data is completedon the Worker Node.

Trans.

Trans.

by Pierre Vande Vyvre

32

Computing model (O2 processing flow)

FacilityFunction

O2

ALICE Online-Offline Facility at LHC Point 2. During
data taking: run the online reconstruction in order toachieve maximal data compression. Provides data
storage capacity. After data taking: runs thecalibration and reconstruction tasks.

T0CERN Computer Center facility providing CPU,
storage and archiving resources. Herereconstruction and calibration tasks are carried out
on a portion of the archived CTF data, plus
simulation if required.

T1

Grid site connected to T0 with high bandwidth
network links (100+ Gb) providing CPU, storage andarchiving resources. It runs the reconstruction andcalibration tasks on its portion of archived CTF datawith simulation if needed.

T2

Regular grid site with good network connectivity(10+ Gb); running simulation jobs.

AF

Dedicated Analysis Facility of HPC type that collects
and stores AODs produced elsewhere and runs theorganised analysis activity.

by Pierre Vande Vyvre- Maintain the advantages of the Grid and theanalysis trains- Make it more open and more effective33