ALICE computing andnetwork in Japan
Tatsuya Chujo
University of Tsukuba
AFAD 2016
February 1, 2016
Kyoto University, Uji campus

Outline

1. Introduction– ALICE computing in Run-1 and Run-2

2. ALICE computing in Run-3 and Run-4 (2021-)
3. Current ALICE O2 project status
4. T2 site(s) in Japan and network
5. Summary2

QuarkGluon
Plasma（QGP）
PlasmaTime: few ! sec after the bigbang.
Temperature: 2 Trillion K
Energy density: > 1 GeV/fm

3ALICE Experiment16m x 16m x 26m, 10,000 tons

>1400 scientists from 149 Institutes
in 40 Countries
The dedicated experiment in LHC experiments to study
Quark Gluon Plasma (QGP) by using heavy ion beams.

18 different
sub-detectors:
tracking, particle
identification, energy
measurement,event trigger
4

Pb-Pb 5.02 TeV (One PeV collisions , Nov. 2015) !!

Current ALICE computing (some numbers)▶ 80 computing centers around the world• T0, T1, T2▶ Up to 100K concurrently running jobs
▶ 600M executed jobs
• 1600 users▶ 10 Tape + 56 Disk storage elements• 25 + 25 PB of data, 1B+ files▶ Up to 40 GB/s read rates (10GB/s avg.)• Writing at 1/10th the read rate

6

Distributed resources on Grid

" Federated computing and storage resources• Users interact with the entire Grid through AliEn" Tightly coupled central task queue and file catalogue• Tasks are typically sent to where a copy of the input data is,but one can also read from anywhere in the world

7

The ALICE grid keeps growing*	  10	  Gbs	  line	  from	  KISTI	  Tier-­‐1	  opera7onal

8ALICE Grid evolution (10 years history)

Successfully using all available capacity9

Grid jobs after the end of Run-1Maximum:• 96K parallel jobs
! "##$%&'()&*(+%)*%,--%(#./01*2%(&*34&5%6789%#*%,:2;<%! =#*1*0&%3>)5%(#./01*2%.#$&-%
0*1-%3>&%&*$%#?%@0*AB%6CDBE%
B8FGAB8B8<10

ALICE computing in
Run-3 and Run-4 (2021-)

11

LHC schedule
89:2;"1"<5=-6>+":?1@;A"?9@&"/6B.-"C5=-6>+":3?:2A"@D2"E/)*.-"C5=-6>+89:2;"11"<5=-6>+":3?:2A"@D2"/6B.-"C5=-6>+

9+6FG"1.*"?C/)*.')HG"",-./"#$IJ"H."J"K#$IJ

9?4?9@A"55"(C/)*.')HG",-./"#$LMN5+6OP"H."Q"K#$LMN(+F+((+>P12

Resource needed in LH-LHC (WLCG)

Integrated luminosity
collision energy

13

The ALICE Online-Offline (O2) Project-Handle >1 T Byte /s detector input
Support for continuous readout
Online reconstruction to reduce data volume
Common hardware and software systemdeveloped by the DAQ, HLT (High LevelTrigger), Offline teams

ALICE O2 project* O2 TDR Approved in September 2015
(with some conditions)14New O2 facility▶ 463 FPGAs• Detector readout and fast cluster finder▶ 100K CPU cores• To compress 1.1 TB/s data stream 14x▶ 5000 GPUs
• Reconstruction speed-up• 3 CPU + 1 GPU == 28 CPU▶ 60 PB disk space• Buffer space to allow for a more precise calibration▶ The current Grid and more in a single computing center• Heterogeneous computing capacity▶ Identical software should work in both Online and Offline
environments

15

Computing Strategy for Run-3 and 4O2

Data of all interactions shipped from detector
to online farm in trigger-less continuous mode
mod91"-C*"#R#"3SGH+T'Data volume reduction by cluster finder
No event discarded
Average factor 2.2(factor 2.5 for the TPC data)
Q$$"%SGH+T'

Data volume reduction by online calib. and reco
All the events go to data storage
Average factor 5.5 (factor 8 for the TPC data)

Tire 0

I$"%SGH+T'Tiers 1 and AF
A

U$"%SGH+T'

Compressed Timeframes (CTF)

Data Storage: 1 year of compressed data
• Bandwidth: Write 90 GB/s Read 90 GB/s
• Capacity: 60 PB

Asynchronous event reconstruction with final calib(delay of few hours)
hours

16

Roles of Tiers in Run-3
Reconstruction
CalibrationArchivingAnalysis

Simulation

DAQ
Reconstruction
Calibration

Analysis17

Analysis facilities (AF)! Analysis is still an I/O bound operation, even after adopting the analysis trains! Merging stages could be sped up on well connected, high memory machines• Leading to shorter turn-around time for entire trains! Solution is to have dedicated analysis facility/facilities• Sites optimized for fast processing of large local datasets• Run organized analysis on local data, similar as today's Grid
• Requires 20-30K CPUs and 5-10 PB of very well connected persistent storagespace• Could be any of the T1s or T2s,
• But ideally this would be a purpose build facility optimized for such workflow
18

Storage requirements* Computing Resources Scrutiny Group (CRSG)

19

Hardware O2 facility
Computing rooms- CR1: reuse existing room (adequate power and cooling for the detector read-out).
- CR0: new room for the computing part of the farm.

20

Computing room
CR0: new room and infrastructureneeded on the surface
Commercial Container Data Centers•Used by several Internet giants but at a lower
power density

CR1: existing room adequate21

T2 site(s) in Japanand network

22

3'COC&6":(1@;"3I"'H6HC'

Slide by N. Nakamura (NII) @ AFAD2015

Members:
T. Chujo (responsible), S. Kato (technical staff)
Status:Infrastructures, MW (EMI 3.1), and service have been
set-up.
Setting up T2 for the test job submission by ALICE.
16 WN’s (X5355; 4 cores x 2 cpu,@2.6GHz) in a rack.
Used IP: HepNet-J.
Connected to SINET-4/(5) (via HepNet-J).
Plan:
Sign-up WLGC (2016).
Will use University’s IPs for head nodes for the futureconnection to WLCG (and LHCONE), with thesupport by U.Tsukuba info. center and KEK.

"1616 WNs (provided by Hiroshima U)
as a prototype of T2 in Tsukuba(marked by yellow labels)
2015, May24

SINET5 (2016-)

SINET5 (2016-)

Slide by M. Nakamura (NII) @ AFAD2015

SINET 5 (April 2016-), by NII
Domestic: 200Gbps backborn,
International: 100 Gbps direct link JPN ⇄US & EU

BH

Summary• ALICE computing usage in Run-1 & 2.
• ALICE Run-3 &4 (2021-)• Continuous trigger-less readout at 50 kHz in Pb-Pb

collisions.
• 1 TB/s raw data from detector, need a significant data
reduction down to 90 GB/s to storage → O2 project• Computing model (O2, T0, T1/AF, T2; re-defined theroles)• Status of O2• Japanese T2 site(s) status & networking (SINET-5).
• Japanese involvement for O2.
26