Overview of the Belle II computing

Y. Kato (KMI, Nagoya)1

Belle → Belle II
KEKB→SuperKEKB

・~40 times luminosity (8×1035 cm-2s-1)
・~50 times integrated luminosity (50 ab-1)

Belle → Belle II・ Fine segmentation.
・ Waveform sampling.

Huge data sample→ Huge computing resource
for large collaboration2Computing resource for Belle IICPU (kHEPSpec)**1core~=10 HepSpec~104 cores

・ Estimation until 2021 (~20 ab-1).
・At the end of data taking (50 ab-1), more than
- 100000 core CPU- 100 PB storage
are expected to be needed to store and analyze
data in a timely manner.

Tape (PB)

3

More than 100 PB?

Big data in 2012
LHC(2012)
Census Bureau 15PB

Business e-mail3000PB/year

3.7PB

Google search index
100PB/year

・Similar to Google search index orContents uploaded to Facebook (per year).
・Impossible to be hosted by a single institute.
→ Distributed computing

Each institute prepare the resources.
Connect by network.

Climate DB

6PB

NASDAQ
3PB

YouTube video
15PB/year

Digital Health
30PB

Contents of Facebook
180PB/year

4

Library of Congress
3PB

https://www.wired.com/2013/04/bigdata

Belle II computing modelFirst 3 years

5Belle II computing system

6

What we need is..
ResourceInterface
Extensionfor
todifferent
meet experimental
resource type=DIRAC
requirements(DistributedInfrastructure
Remote Agent Control)-Grid:- Automationof MC withproduction,-Originallydeveloped
Relatively
bigprocessing
sites. by LHCb
raw data-Thesource,
good (andextensibility
Storeopen
data/MCsampleproduce).
→
Productionsystem-Usedin many experiments:-Cloud-LHCb,UserBES
interface.
III,resource”
CTA, ILC, and so on“Commercial- Analysis
framework
MCproduction-Local- etc computing cluster
Relatively small resource in universities.
MC production

GridCloudLocal cluster….

Production system and its development 7Definition
・ MC prod / data process
・ Type (BB, ττ, ccbar..)・ # of events・ software version
・ etc..

BelleDIRAC

DIRAC

PS

-Production-Distribution-MergeProduction manager (human)- Define “Production”

Distributed data management system
・Gather outputs to major storage(and distribute over the world)
・Check status of storages
・Define “Transfers”

output info
Monitor

DIRAC Transfer management

Fabrication system
・ Define jobs
・ Re-define failed job
・ Verify output filesDIRAC Job management
Submit job on site

Resource Destination storage

Temporary storage

Computing siteMC production campaigns

8・ Test the validity of the computing model/system.
・ Provide simulation samples for the sensitivity study.
Auto job submission
Automatic Issue detection (5th)20 kjobs

4th

5th

Data distribution
by DDM

6th 7th(ongoing)

3rdnd2
1st・ ~50 computing sites join in the latest campaign.
・ More than 20k jobs can be handled now.
・ Gradually automating the production procedure.
・ Belle II colleagues take computing shifts from 4th campaign as an official service task.

KMI contributions
Significant contribution from KMI
・ Resource

Belle II dedicated resource in KMI・ 360 (+α) CPU cores.
・ 250 TB storage.
・ Grid middleware (EMI 3) installed.
・ DIRAC server.
・ Operation by physicists→ Learned a lot on operation ofa computing site.

・ Development of monitoring system- To maximize the availability of resources- Automatic detection of the problematic sites- Operation and development of the shift manual9

Contribution of KMI resources
CPU usage during MC7~7%

Data transferred during MC7

・ Serving as destination storage.
・ Executed more than 1×105 jobs in MC 7
・ We will purchase ~200 cores CPU in this year (~1.5 times).

10

Monitoring system

11・Many interfaces → Need to identify “where the trouble happens”

・ Store and process information of each step in database.
・ Analyze log file to identify the origin of problem further.
・ Show the list of problems on the web, if detected.

Automatic issue detector

Monitoring system (active way)

12

Actively collect site status by submitting diagnosis job.
SiteCrawler:

Check the site environment to execute Belle II job

Job Submission check:

As DIRAC does not record failure reason,
job submission is tried and record the result.

Our activities maximize the availability of the resource !

Production progress monitoring・Submitted

・Under job・Transfer

・Finished13

Future prospects and coming events 14
・ Continue to improve the system- Maximize the throughput.
- More automated monitoring/operation.
・ Cosmic ray data processing (2017)- First real use case to try raw data processing workflow.
・ System dress rehearsal (2017): before Phase-2 runs
- To try the full chain workflow from raw data to skim.
・ Start of the phase 2 run in 2018.

Summary

15・ Belle II adopted the distributed computing model to cope withrequired computing resource (first experiment hosted at Japan).
・ "BelleDIRAC" is being developed to meet experimental requirements
and validated at the MC production campaigns.
・ KMI has a huge contribution on distributed computing:- Resources- Development of the monitor
and upgrade the resource in this year.
・ In 2017, the processing of comic ray data and System dress rehearsal
will be performed.

16Backup

Distributed computing

17Local computer:-Interactive-Data on local disk.

Computer cluster (Belle case):- Send “job”- “Homogeneous” resource in single place.
- Data on shared disk

Distributed computing:- Send Job (to central service)-“Heterogeneous” resource
distributed over the world.
- Data distributed over the world.

Network data challenge result

Network challenge results

18

User analysis in Belle II19

Physics gurus perform bulk MDST Production
Skimming group makes skims from MDST

Almost all user analysis will originate from skims(Data samples > 1 ab-1 require this)We should plan for at least 200 simultaneous analyses

Skim production will require an automated Fabrication system••

Options for Physics skims from bulk MDST
– Output skimmed MDST
– Output Physics Objects + MDST (μDST) <= Baseline solution• Direct copy to WN using DIRAC• Stream data using XRootD/http– Output Index files which point to skimmed events <= Under investigation• Access bulk MDST directly via root on large cluster• Stream from bulk MDST using XRootD/httpLocal Cluster/Workstation root
– Validated by Physics group

μDSTE20E

TT
T

BB
ETE

TT

MDST, Keep TEE

TTTE

T

T

T

E

E

T

T

T

E

T

T

μDST, Keep

B

21

22

23

24

25

26