A short history ofHigh Energy Physics activity
between Japan and Korea
Soh Y. Suzuki (soh.suzuki@kek.jp)
Computing Research Center
High Energy Accelerator Research Organization (KEK)2014/10/08(水)High Energy Physics• is study of sub-atomic particles to understand the Universe• Category of experiments‣ Accelerator‣‣ Accelerate protons or electrons toproduce sub-atomic particles

KEKB‣‣ accelerated bunch of particles is called as "beam"‣ Non-Accelerator‣‣ e.g. Cosmic-Ray observation• Most of experiments are driven byeach international collaboration
Super-Kamiokande2014/10/08(水)Facilities for experiments• Accelerator to provide beam
• Detector to take data using thebeam• Computer to analysis the data
‣ Huge data from accelerator experiments
need computer for analysis‣ Once accelerator starts the operation,
computer must not stop recording data by
their problem

2014/10/08(水)Major international
collaboration in Japan
J-PARC@Tokai
T2K, g-2, etc
KEK@Tsukuba
Belle, K2K (past)
ICRR@Kamioka
Super-Kamiokande
T2K, K2K (past)

Hiroshima-Univ.
ALICE Tier2

KEK is 60km awayfrom Tokyo

ICEPP@TokyoATLAS Tier2
4Tokyo2014/10/08(水)AS2505 HEPnet-J• HEPnet-J connects many HEP groups of universitiesin Japan• Their IP prefixes are independent from theircampus network.
‣ but the primary network of their campus network is SINET• Gate to Internet is placed in KEK‣ operated by KEK staff‣ Formerly, it has several international links. (to US, TW, CN, SU),but most of them are already shutdowned or replaced by VCs by NREN.

2014/10/08(水)Really brief history• ~2000: KR-JP link of IMnet through SINET

Dedicated Link age‣

Priority is keeping reachability‣

but frequently the route from SNU to KEK went to US• 2000~2003: ATM VC between APAN-JP and KEK‣priority is keeping better path for short RTTATM age• 2003~2005: routed as ordinary Internet Unfortunate age• 2005~2014: VC by JGN-X between APAN-JP and KEK• 2014~: VC by SINET between APAN-JP and KEK

2014/10/08(水)Formerly, it was centralized• The experiment site should provide all of them.
‣All of the data are stored and preserved at there.

‣Collaborators may use computer remotely from their home institute,‣‣ Requires world wide network‣‣ Protocol was DECnet (~1990s) + Internet (~ now)

• Or copy skimmed (most important) part of data to their institute.
‣by tape cartridges• Network was mainly for communication of collaboration and remote access.

Computer

Come here forbetter computing
experience :-PDetector

Interactive use over 100ms RTT
is very uncomfortable!

2014/10/08(水)Wide-band network has changed it• Skimmed data can be copied via network‣ Collaborator can timely and quickly analyze using the skimmed
data at their institute.
• Abroad collaborators predicted they can analyze by computers nearby
them, no need to wait for the long batch queue on the host site!
‣ Soon after that, the long latency of trans-pacific networks destructed
the beautiful dream.
‣ There was no handy application to transmit data over shared
networks whose RTT is longer than 300 msec.
‣ It was solved by Grid systems such as LHC Computing Grid orOpen Science Grid.

ComputerDetector

We can taste data easily after mass-data transmission!

2014/10/08(水)De-centralization• The scale of experiments becomes larger an larger‣

Needs of computing resources also

• Single institute can't provide sufficient computing resources only byitself.
‣Data will be scattered and preserved to many collaborator sites.
‣‣‣ CPU power + Large storage‣‣‣High speed network connection to Internet

Tier0

Primary site‣‣‣ Operational human power• By the LHC experiment which is a largest
collaboration in HEP, it’s tier-structure
becomes popular.

Tier1
National level

National level

Tier2
Institutional

Institutional
Institutional
Institutional• KISTI has a Tier-1 for ALICE of LHC experiment

Institutional

Tier3

local

local

local

local
local

2014/10/08(水)National level

local
local

local

local

local

corresponds to about 1TB of raw data per day. The
amount of the raw and processed data accumulated so far
exceeds 1.2PB. Belle’s computing model has been atraditional one and very successful so far. The computing
has been managed by minimal number of people using
cost effective solutions. Looking at the future, KEKB/Belle plans to improve the luminosity to a few times 1035cm2s1, 1024 times as much as we obtain now. This
paper describes Belle’s efficient computing operations,struggle to manage large amount of raw and physics data,and plans of Belle computing for Super KEKB/Belle.

BELLE COLLABORATIONThe Belle collaboration consists of more than 400
physicists from 58 institutions from 13 countries asshown in Figure 3; Universities and HEP institutions fromRussia, China, India, and universities from Japan, Korea,
and Taiwan, Australia, US and Europe. KEK dominates
in one sense as 30~40 staffs work on Belle exclusively
and most of construction and operating costs are paid byKEK. However, universities dominate in another sense as
young students from the universities are to stay at KEK,
to help operations and to do physics analyses. Shortagesof human resources, in particular, in the area ofcomputing and software, are our problem as Belle is
STATUS OF BELLE EXPERIMENT
physics analyses and planning the major
Research on violationof thesymmetry
anddoing
anti-particles
Belle started
takingdata in 1999between
and has nearlyparticles
been running,doubling its data size every year. Figure 1 shows the upgrade.

Belle Experiment in KEK•

using large number of B-mesons. (1999~2010)
history of its accumulated integrated luminosity.

May 1999:first• 13 countries, 57 institutes, 400
collaborators
collision
~1 fb /day !

July 2001:30fbOct. 2002:100fb
July 2003:150fb
July 2004:287fb
July 2005:550fb

• Centralized computing model

Aomori U.
BINP
Chiba U.
Chonnam Nat’l U.
Chuo U.
U. of Cincinnati
Ewha Womans U.
Frankfurt U.
Gyeongsang Nat’l U.
U. of Hawaii
Hiroshima Tech.
IHEP, Beijing

IHEP, Moscow
IHEP, Vienna
ITEP
Kanagawa U.
KEK
Korea U.
Krakow Inst. of Nucl. Phys.
Kyoto U.
Kyungpook Nat’l U.
U. of LausanneJozef Stefan Inst.

• Data rate was 15MB/s ~ 30MB/s in average

Super KEKB
1~10ab/year!

>1TB raw data/day
more than 20% is hadronic events
1km

Figure 1: History of Belle integrated luminosity
KEKB now runs with the continuous injection mode.
No need to stop run
Always at ~max. currents and therefore maximum luminosity~30% more ³Ldt

[CERN courier Jan/Feb 2004]

both KEKB & PEP-II

continuous injection (new)normal injection (old)40

HER current

0m

Luminosity
12~1 fb-1/day- !
(~1x106 BB)24

Time

2014/10/08(水)Seoul National U.
Shinshu U.
Sungkyunkwan U.
U. of Sydney
Tata InstituteToho U.
Tohoku U.
Tohuku Gakuin U.
U. of Tokyo
Tokyo Inst. of Tech.
Tokyo Metropolitan U.
Tokyo U. of A and T.
Toyama Nat’l CollegeU. of Tsukuba
Utkal U.
VPI
Yonsei U.

13 countries, ainstitutes, ~400 members

Figure 3: The Belle collaboration

SOFTWARE
Core software
Belle software is supported on two operating systems,
Solaris 7 on Sparc CPUs and RedHat 6/7/9 on Intel CPUs.
Belle uses the GNU compiler collection of version from
2.95.3 to 3.3. Belle code compiles with SunCC as well.
Belle software requires no commercial software. Belleuses QQ from CLEO, EvtGen from CLEO and BaBar,CLHEP, GEANT3 and CERNLIB from CERN, andpostgresql. Parts of Belle software (parts of detector
simulation code and calibration code) are written in
FORTRAN. For I/O, Belle uses a home grown software
package called panther and compression using zlib.
Panther is the only data format for all stages from DAQ tofinal user analysis skim files. Index file (pointer to eventsin the panther data files) is used for final physics analyses.

3.5PB tapes were consumed
at the exp. end

LER current

0U. of Melbourne
Nagoya U.
Nara Women’s U.
National Central U.
Nat’l Kaoshiung Normal U.
Nat’l Lien-Ho Inst. of Tech.
Nat’l Taiwan U.
Nihon Dental College
Niigata U.
Osaka U.
Osaka City U.
Panjab U.
Peking U.
Princeton U.
Riken-BNL
Saga U.
USTC

Figure 2: Continuous injection improves int. lum.

History of computing
facility for Belle
1999

2001

2006

2009

CPU (speccint2000)~100(SPARC WS)~1200(WS+PC)~42500(PC)~115200(PC)

Disk (TB)

4910001500

Tape (TB)

1606203500

3500# of Servers

41196

96

Servers forInteractive use

28

23128

128

11
2014/10/08(水)2014/10/08(水)Belle collaborators from Korea• Chonnam National Univ.
• Gyeongsang National Univ.
• Hanyang Univ.
• KISTI• Korea Univ.
• Kyungpook National Univ.
• Seoul National Univ.
• Soongsil Univ
• Sungkyunkwan Univ.
• Yonsei Univ.

2014/10/08(水)Belle II experiment

5

9 months/year
20 days/month

60
40
30

• institutes will store a certain
amount of data.

2target instantaneous luminosity
352

8 x 10 /cm /s

20
100
2011

Physics run starts in 20172013

Experiment• Currently, existing CPUpower is almost exhausted.

34

2.1 x 10 /cm /sryBelle-1~1 ab

50target integrated luminosity
50 ab-1in 2022

ina70

SuperKEKB Commissioning starts in 2015m• Impossible to provide
sufficient resources only byKEK,80

Preli• Accelerator will be
upgraded to provide the 40
times higher luminosity.

Integrated Luminosity (ab-1)

Luminosity Prospect

Belle II
ALICE (Pb-Pb)
ALICE (p-p)
ATLAS
CMSLHCb2015

2017

Event size

2019

2021

2023 calendar year

Rate @ Storage Rate @ Storage[kB][event/sec][MB/sec]

300

6,000

1,800 (@ max. luminosity)

50,0002,000~700~1,000
55100
100600500
4,500

4,000200
400
several 100s
250

(LHC experiments : as seen in 2011/2012 runs)

23 countries/regions
97 institutes
587 colleagues

2014/10/08(水)Belle II Raw Data Distribution

Raw data distribution

A few PB storage
area isMeetingneeded10%of rawdata
LHCOPN-LHCONE
University of forMichigan(US)September 15-16, 201415
2014/10/08(水)9Interactive use and file transfer• In the case of JP-KR, the problem is not RTT, but bandwidth.
• In the former model, the activity between JP-KR was just interactive
use of computer facility and most of the problem was just reachability.
‣ Sometimes route from KEK is mis-filtered and inaccessible fromKorea‣

e.g. Seoul National Univ.

KREN

KREONET

SINET

Best

IMnet

SNU
KOREN

APAN-JP

UsableBad

RENATERGEANT• Busan and Fukuoka 1Gbps line opened at 2004, the situation wasdramatically changed.
‣ Center of HEP (CHEP) in KNU tasted the performance of file transfer.
‣ CHEP - APAN-KR - QGPOP - SINET - KEK

2014/10/08(水)KEKRelation reported at
APAN2003
Participation of Institutions in the HEP Data Grid Project!
1. CMS!
Tier-1!
Regional Center!
(CERN)!

4. Belle!
Exp!
(Japan)!

Konkuk U!

CHEP!
Kyungpook N U!
Gyeoogsang N U!
Korea U!

2. AMS!
Regional Center!
(CERN)!

Seoul N U!

5. K2K!
Exp!
(Japan)!

Seongkyunkwan U!
Chonnam N U!
Ewha W U!

Yonsei U!
3. CDF!
Grid-KC
AF!
(USA)!

Dongshin U!
KBSI!
…!
Other users!

17
2014/10/08(水)Data Grid!
Cluster!

6. PHENIX!
Grid!
(USA)!

By Youngdo Oh,at APAN2003

CHEP(KNU) - KEK performance• RTT:KEKKNU Kyushu

7ms
20ms
• Netperf showed the necessity of window size extension• Multi-stream application achieved better speed.
‣ bbcp achieved 110Mbps by 10 streams although it wasn't
popular.
Throughput by netperf‣ HPN patch for SSH didn't yet appear.

• Already major labs in EU,US wasmoving to use grid-ftp.

Throughput (Mbps)‣ Manual and occasional file transfer

60
4530
1500100
socket buﬀer size (kB)

2014/10/08(水)200scp is not so fast, but...
• Still scp and rsync are popular for manual and occasional file transfer, butthey are slow over international links.
• Why?
‣ Many users can't expose their own host to Internet, so unusual new
application are not available at the login server.
‣‣ filtered by firewall controlled by IT section in their institutes.
‣ If both users have grid resources, grid-ftp is usable. But not always so.
• Few people know HPN patch is effective, but most of sites do NOT apply
the patch to their production system to avoid security risks related to SSH.
• The Belle experiment was NOT in the grid world at that time.
LAN2014/10/08(水)InternetLAN

KISTI• A HEP group born in KISTI and joined to Belle in 2008
• Belle group planned a data replication• KISTI is in KREONET and route from KEK went to HKIX
‣ relatively farther than sites in KOREN• Route is KISTI - KREONET - APAN-JP - KEK• HPNed SCP achieved ~ 420Mbps• Never reached to 1Gbps, so precise QoS was indispensable. Itis quite un-welcome for the server managers.
• We observed severe packet
loss higher than that, butwhere is the bottle neck?

KISTI
KREONET

Universities

KOREN
QGPOP

HKIX
2014/10/08(水)SINET
APAN-JPKEKIt was in Japan• The congested link is Tsukuba-Tokyolink.
• This link is used for the peer for KEK andAPAN-JP.
• Traffic to Universities in KOREN havesame problem.
• Situation becomes worse year by year,
recently we could achieve only
400Mbps.

Jammed• Traffic to KOREN is also damaged

2014/10/08(水)

KNU

KOREN

APAN-JPKISTIKREONETHKIX

JGN-Tokyo

JGN-Tsukuba

KEKL
iperf test at 2010 from
KEK to a node over
KREONET2

Limit to 400Mbps

No QoS
Limit to 450Mbps

2014/10/08(水)Link replacement• At May 2014, We noticed KREONET will be reachable via JP-KR link.
‣Strong request to fix this jam condition• The jammed link is superseded by a virtual circuit provided by SINET.
• The situation dramatically improved.

KEK - KREONET2KEK - APAN-JP
2014/10/08(水)Careless tuning is not good

Default installation of perfSONAR-PS
Window size extension for trans-pacific tests
Peak speed may be better, but average speed decreases.
24
2014/10/08(水)Future prospects• Belle II will start the operation and transmit rawdata to the data
center in Korea.
• KISTI in KR and Hiroshima Univ. in JP are Tier-1 and Tier2 of the
ALICE experiment in LHC respectively.
‣ Data source site is automatically chosen by Grid middleware, files
are transferred non-interactively.
‣ Total bandwidth limited by facilities inHiroshima University, upgrade is needed.
‣They are planning to join into LHCONE tobypass firewall after few years.

KEKKISTI
Hiroshima Univ.

2014/10/08(水)TokyoSummary• Current bandwidth between JP and KR is very
helpful for our collaborations.
• Still main activity is interactive use and occasional
file transfers, but several experiments are planningmass-data transfer.
• We deeply appreciate efforts for many years toimprove the connection between Japan and Korea.

2014/10/08(水)