H I G H E N E R G Y A C C E L E R AT O R R E S E A R C H O R G A N I Z AT I O NCOMPUTINGRESEARCHC E N T E RInter-University Research Institute Corporation

High Energy Accelerator Research Organization

Computing Research Center

Computing Research Center

Computing Research Center

KEK Network

3

▲
▲
▲

P

Grid System

7

▲
▲
▲

P

Supercomputer System

9

▲
▲
▲

P

1

High Energy Accelerator Research Organization Computing Research Center

13

P

▲
▲
▲

Central Computer System

IT Service
▲
▲
▲

P

15

Hadron Experimental Facility
Materials and Life Science
Facility (MLF)

Main Ring

Circumference: 1,568m

Neutrino Beam

J-PARC
▲
▲
▲

17

P

3 GeV RCS

Neutrino Near
Detector

Circumference: 348m

Linear Accelerator
330m

Research Activities

19

▲
▲
▲

P

High Energy Accelerator Research Organization Computing Research Center

2

Central Computer System

Central Computer System
KEK uses the Central Computer System (KEKCC) for various research activities. The Data Analysis System in KEKCC is primarily used to analyze experimental data. KEKCC also provides IT infrastructures such as the E-mail system and different Web services.

Work / Computing Server
The Data Analysis System provides
high-performance computing servers anda large-capacity storage system to accumulate and analyze a large amount ofexperimental data. The computing servers,
which have approximately 15,000 CPU
cores, are used for interactive and batch
analysis. The storage system has a 25.5 PB
disk system and a tape library with a maximum capacity of 100 PB. These high-spec
storage systems enable to process experimental data with high-throughput.
Lenovo ThinkSystem SD530/SR630 15,000 cores

3

High Energy Accelerator Research Organization Computing Research Center

Network connecting the Systems

External connection: Cisco Nexus 9516 / Juniper SRX4100

Interconnect fabric: Mellanox CS7500 4xEDR

Disk Storage System

Mail System · Web System

IBM Elastic Storage Server GL4C (25.5PB)

F5 BIG-IP Local Traffic Manager / WAF
Lenovo SR650 / SR630
IBM Storwise V5030E
IBM System Storage TS4300

Tape Library

IBM TS4500 Tape Library (100PB)

High Energy Accelerator Research Organization Computing Research Center

4

User Groups of the Data Analysis System

User Groups of the Data Analysis
Belle / Belle ⅡExperiment
The Belle experiment is conducted to detect small differences among the reactions between matter andantimatter in B meson decays. It contributes to verification of the Kobayashi Maskawa theory, for which
Kobayashi and Maskawa were awarded the Nobel Prize in Physics in 2008. The next generation of the Belleexperiment, Belle II, will detect data 50 times as large as the Belle experiment.

Dr. Makoto Kobayashi and Dr. Toshihide Maskawa

5High Energy Accelerator Research Organization Computing Research Center

The Nobel Prize medal awarded to Makoto
Kobayashi, Professor Emeritus of KEK

System
Tokai-to-Kamioka (T2K) Experiment
The objective of the T2K neutrino oscillation experiment is to elucidate the entire aspects of the originof a neutrino’s mass and neutrino oscillation (mixing). In the experiment, a beam of high-intensity muon
neutrinos produced at the J-PARC accelerator in Tokai travels 295 km to the Super-Kamiokande located 1,000m below the town of Kamioka in Gifu. As the neutrinos travel underground, they change flavors. This phenomenon is called neutrino oscillation (mixing). Detailed and comparative analyses of observation results
between J-PARC and Super-Kamiokande are used in this experiment.

International Linear Collider (ILC) Project©Rey.Hori

Consisting of two linear accelerators that face each other, the ILC
would collide electrons with positrons at a collision energy of 250
GeV. Each collision creates many particles that enable us to examine
the Higgs particle and to search for new particles such as dark matter.
The international team is making an effort for realization of the ILC.

High Energy Accelerator Research Organization Computing Research Center6

Grid System

Grid System
Accelerator physics experiments, such as Belle II, T2K, and ILC, collect the order of hundreds ofpetabytes data these days. KEKCC has introduced a system for the high-speed data share amongcollaborating institutes around the world, as well as the Grid system for the efficient and safe
use/operation of the computer system even in environments where data is distributed globally.

Grid Middleware
The software for the distributed systems is usually called Middleware, particularly in Grid computing.
The Grid System in KEKCC consists of two types of middleware: Integrated Rule-Oriented Data System(iRODS) and Unified Middleware Distribution (UMD).

Integrated Rule-Oriented Data System (iRODS)
iRODS is a software specialized for the data sharing. It is actively utilized to transfer data produced in
J-PARC (KEK Tokai Campus) to KEKCC (KEK Tsukuba Campus).

Unified Middleware Distribution (UMD)
UMD is the integrated set of various middleware deployed in the KEKCC Grid System. It enables not
only for the data sharing but also transparent use of distributed computing resources. UMD is widely usedin the Worldwide LHC Computing Grid (WLCG), the world's largest computing infrastructure. A lot of accelerator physics experiments, including LHC, Belle II, and ILC, are designed to analyze the huge amounts ofdata leveraging that computing infrastructure.

7

High Energy Accelerator Research Organization Computing Research CenterCertificate Authority
We have maintained a certificate authority since 2006 to support various projects using grid computing
conducted at KEK and in Japan, such as the Belle experiment at KEK, the ATLAS and ALICE experiments at
CERN, the ILC project, and lattice Quantum Chromodynamics (QCD). Our certificate authority is an accredited member of the International Grid Trust Federation (IGTF), which plays a major role in establishing anauthentication infrastructure for international large-scale computing grids. The certificate authority issues
digital certificates, using SHA-2 signature algorithms, to users and hosts. The issued certificates and associated keys are used for grid authentication and secure communication. The server of the certificate authority
distributes a Certificate Revocation List (CRL) and operates the Online Certificate Status Protocol (OCSP) responder, with an iPv4/iPv6 dual-stack solution.

WLCG – the world’s largest computing infrastructure – consists of 1 million CPU cores and 1 exabyte of storage capacity as of 2016.

High Energy Accelerator Research Organization Computing Research Center8Supercomputer System

Supercomputer System
The KEK Supercomputer System is used for large-scale numerical simulations. The current system is NEC SX-Aurora TSUBASA. The system is provided for researchers under “Particle, Nuclear, and Astro Physics Simulation Program” promoted by Institute of Particle and Nuclear Studies(IPNS), KEK.

NEC SX-Aurora TSUBASANEC SX-Aurora TSUBASA is a heterogeneous system equipped with thevector engines that accelerate arithmetic operations. The large memory bandwidth of the vector engines is particularly efficient for large-scale numericalcomputations.
System: NEC SX-Aurora TSUBASA
		
A500-64 2 rack
Number of vector hosts: 16
Number of vector engines: 128
Total peak performance: 313.6 TFlops(double precision)
Total main memory: 6 TB

*1: Flops (Floating point operations per second) is a measure of the performance on four arithmetic operations in a second.
G (giga), T (tera), and P (peta) represent 10 to the power 9, 12, and 15, respectively.

9High Energy Accelerator Research Organization Computing Research CenterScientific Research on the Supercomputer System

Scientific Research on theSupercomputer System
Lattice Quantum Chromodynamics
The dynamics of quarks and gluons,
which are elements of nucleons, are described by Quantum Chromodynamics(QCD), the fundamental theory of thestrong interaction. Numerical simulationsof QCD have become possible with latticediscretization of four-dimensional spacetime. It enables prediction of the masses
and decay form factors of nucleons andmesons, and exploration of vacuum structures without model assumption.

Guido Cossu © 2011

Numerical simulations on supercomputers can elucidate various characteristics ofparticles and atomic nuclei that are intractable by analytical calculation. These characteristics include the following:• Automated computation of reaction processes of particles• Quantum chromodynamics at finite temperature/density• Space-time reduced models of the gauge theory• New approaches to understand structure of field theories

Simulation of Supernova Explosions
The mechanism of a supernova explosion caused by gravitational collapse of a massive staris still shrouded in mystery. KEK
seeks to solve this mystery bymeans of large-scale simulations
incorporating the neutrino effectthat plays an essential role in the
explosion mechanism.

High Energy Accelerator Research Organization Computing Research Center

10

HPC Servers for Joint / Collaborative Research

HPC Servers for Joint /Collaborative Research
KEK is engaged in the development of a next-generation architecture and corresponding applications to exploit the maximum performance of the architecture.

Suiren2 and Suiren BlueThe Suiren2/Suiren Blue systems are supercomputers equipped with many-core Pezy-SC processors (byPezy Computing K.K.) in a compact manner via a submersion liquid cooling technology that enables low
power consumption and high-density mounting, which was developed by ExaScaler Inc. KEK is engaged in
the development of high-performance applications in collaboration with Pezy/ExaScaler.

Suiren (Water Lily) 2 : “ZettaScaler-2.2” 12 Bricks• Total peak performance: 1,082 TFlops• Main memory: 26 TB (2-TB CPU + 24-TB Pezy-SC2)• 384 Pezy-SC2 processors
Because high-performance computers generate significant quantities of heat, an efficient cooling method is key for their optimizationand downsizing. Suiren can be efficiently cooled by complete immersion in a special coolant. Suiren needs neither air-cooling fans nor
water-cooling tubes, and therefore saves space. Suiren was upgradedto Suiren2 by installation of new Pezy-SC2 processors.

Suiren Blue (Blue Water Lily) : “ZettaScaler-1.5” 16 Bricks• Total peak performance: 428.3 TFlops•	Main memory: 16 TB (8-TB CPU + 8-TB PezySC)• 256 Pezy-SC processors
Suiren Blue gives the same arithmetic performance as Suiren, but it is a more compact implementation that occupies a quarter of the installation area and consumes even less power than
Suiren.

Suiren Blue and Suiren2 achieved the world ’ s second ranks, respectively, in“ The Green 500 List ”ofsupercomputer power consumption performance ranking published in July 2015 and in Novembar 2017.

11 High Energy Accelerator Research Organization Computing Research Center

Data Grid for Computational Physics

Data Grid for Computational
Physics
Japan Lattice Data Grid (JLDG)
Lattice QCD simulations produce large amounts of data. To analyze these data using supercomputers
across Japan, it is necessary that the data be transferred at a high speed and handled as if they are on asingle file system.
JLDG is a data grid for computational particle physics, which is constructed on the high-speed network
SINET, operated by the National Institute of Informatics. It uses Gfarm, a grid file system developed by the
National Institute of Advanced Industrial Science and Technology in cooperation with University of Tsukuba.
Status in May 2020:
13.0 PB total storage, 12.3 PB used (95 %)141M files

http://www.jldg.org/
gfarm provides:
a network shared ﬁle systemthat supports scalable I/O
performance in distributed
environment
Kanazawa

KEKpart of JLDG data can beaccessed from ILDG
CCS

fast parallel ﬁle copy

HPCI Shared Storage

Tsukuba
Hiroshimagfarm

Riken (Wako)

Kyoto

Tokyo (Kashiwa)

Osaka

Tokyo(Kashiwa) Riken-CCS

Backbone: SINET
L3-VPN (NII with KEK support)

High Energy Accelerator Research Organization Computing Research Center

12

KEK Network

KEK Network
KEK is connected to SINET, an academic network by 120 Gbps bandwidth including backup circuit. Most of
the devices in KEK are connected via 1Gbps link, but some systems that require high-throughput network are connected to core switches via multi links of 10Gbps.
There are about 210 buildings in campus, and 50 of them are directly connected to the Computing ResearchCenter (CRC) through optical fibers. In the case of populous building, each floor is directly connected to CRC.
These optical fibers are connected to authentication switches that manages connections to network outlets ineach room. The authentication switch automatically connects devices to proper networks using the network registration database. Unregistered devices can’t use our campus network.
There are about 250 wireless network access points in Tsukuba Campus, and these are controlled centrally.
Each access point services four different networks and user can choose one of them according to their needs.

Wavelength division multiplex

Border switch

XGMC2016

Core switch

Brocade MLXe4

External router
FirewallExternal router: ARISTA 7280SR
Firewall: Paloalto 5250

Nexus9508

13 High Energy Accelerator Research Organization Computing Research CenterResearchers in high-energy physics have collaborated over international networks since the 1970s. Although the wide spread of the internet has changed applications and technology for them, their aims are still datatransfer and communication such as teleconference and e-mail. The KEK network has 10 Gbps redundant bandwidth firewalls for high-speed data transfer.

LHCONEInternet

The LHC experiment at CERN
produces extremely large amountsof data, and those data are processed by distributed computing.
LHCONE is a closed network specialized for mass data transmission
among LHC grid computing sites.
As most of the grid computing sites
joining the Belle II experiment in
KEK are already connected to LHCONE, KEK is also connected.

LHCONE

SINET

KEKCC

VPN

The KEKCC system operated
by CRC has a firewall that is independent of the campus network at
KEK because KEKCC provides highspeed data transfer not only for KEK
staff, but also for collaborators at
remote sites. This dedicated firewall
enables high-speed data transfer using the grid middleware of up to 20
Gbps.

VPN
30,00025,00020,00015,00010,0005,000

Mar-20

Sep-18

Sep-19Mar-19Sep-17Mar-18

Sep-16

Mar-17

Sep-15Mar-16

Sep-14Mar-15Mar-14

Sep-12

Sep-13Mar-13

Sep-11Mar-12

Sep-10Mar-11Mar-10

Sep-08

Sep-09Mar-09Sep-07Mar-08

Sep-06Mar-07

Sep-05Mar-06

Sep-04Mar-05

Sep-03Mar-04Mar-03

Sep-01

Sep-02Mar-010
Mar-02

CRC has provided a Virtual Private Network(VPN) connection service since 2001. The firewall prohibits any access from the internet toservers in the KEK campus network, except toservers in the demilitarized zone(DMZ) network.
May researchers at KEK have their own servers
on the KEK campus network; the VPN enables
access to those servers from the internet. The
number of VPN connections has increased annually, and currently there is close to 20,000
connections a month.

High Energy Accelerator Research Organization Computing Research Center

14

IT Service

IT Service
E-mailThe e-mail system sends and receives emails for more than 1,600 persons at KEK. The
system also operates over 900 mailing lists associated with research activities and operationsat KEK. The figure shows the daily throughput
of our e-mail system in April 2020. It deals withapproximately 100,000 e-mails per day and hasenhanced security measures for dealing withspam and computer viruses.

Daily throughput of e-mail system
180000160000140000120000100000
80000
60000
40000
20000
0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

The DMZ User’s Portal:
A vulnerability management portal
Servers receiving access from outside KEK,
such as internet connections, are required tomaintain a high security level. In KEK, thoseservers are connected to the DMZ network.
CRC developed a vulnerability* management portal, called the DMZ User’s Portal, toprovide a vulnerability management environment for managers who have their servers in
the DMZ. The portal site is linked to the vulnerability diagnosis devices and other systems toprovide a variety of security services on DMZ.
Server managers can not only confirm periodic vulnerability diagnosis reports for security
maintenance but also can prepare and submit
self-assessment reports of security inspection
conducted by KEK.

Internet
NetworkswitchDiagnosis
deviceSW
Diagnosis
deviceFirewallDMZ User’s
Portal Operation
Preparation/Submissionof a diagnosis reportServer Diagnosis

FW
Diagnosis

SW

DMZ

Diagnosis

Diagnosis
deviceIntra-KEK
network* Vulnerability refers to a defect in information security that developed owing to program malfunction and/or design error in the computer OS and/or software. A vulnerability is also called a security hole. Using a computer thathas a vulnerability can lead to unauthorized access to that computer and/or viral infection.

15 High Energy Accelerator Research Organization Computing Research CenterWeb System
The Computing Research Center provides the KEK official
websites (www.kek.jp, www2.kek.jp) and the websites that
support research activities at KEK. These websites include theresearch information web (research.kek.jp) to publish research
activities and achievements, and Wiki (wiki.kek.jp) to provide
a place for exchanging information between researchers. We
operate the conference web system (www-conf.kek.jp, conference-indico.kek.jp) that supports the hosting of conferences
and workshops and the publishing of results, and also the KDS
system (kds.kek.jp) that supports the group meetings.

Video Conference System
We have been providing a video conference
最大同時接続数
Maximum numberof simultaneous connections
service with an MCU in a multipoint videoconferencing system for more than 20 years. In October
2018, we have replaced an MCU with a new videoconference service “Vidyo”.
The client software “VidyoConnect” supports
multi-platforms such as Windows, macOS, iOS, andAndroid. For Linux, we open a generic communication channel via WebRTC, which allows users to attend the video conference on common web browsers with no specific software.
The number of registered users reachednearly 1,200 in about 17 months after the service
launched. The figure in the right shows the numberof simultaneous connections to the system, which is load-balanced by two machines. Roughly 110 of connections is the upper limit in the current contract of the system. 70-90% of maximum capacity has been used forthe last few months.

High Energy Accelerator Research Organization Computing Research Center

16

J-PARC

J-PARCThe Japan Proton Accelerator Research Complex (J-PARC: https://www.j-parc.jp) consists of themain facilities on the Tokai Campus, and is run jointly by KEK and the Japan Atomic Energy
Agency (JAEA). In J-PARC, which is a multi-purpose quantum beam facility open to users fromaround the globe, cutting-edge experiments and research are conducted. These experiments
and research are across a wide range of scientific fields, including particle physics, nuclear
physics, material science, life science, and nuclear technology, that make use of the variety of
secondary-particle beams such as those of neutrons, muons, K mesons, and neutrinos produced by one of world’s highest intensity proton beam. The J-PARC network infrastructure,
JLAN, connects more than 5,000 information systems in J-PARC, providing a LAN environment
and an internet access speed of 10 Gbps. It also connects the Tokai area, where accelerators
exist, to the Tsukuba area, where the main data analysis computers exist, supporting research
activities at J-PARC. The internet and Tokai-Tsukuba connections use SINET (https://www.sinet.
ad.jp), which is provided by the National Institute of Informatics.

6000

350

Edge
Switches
エッジスイッチ

250

WLAN
Access
無線Lan
AP数 PointsNo.
of hosts
端末数

5000

No. of Hosts 端末数
connected to JLAN

SW and WLAN AP
スイッチ，無線Lan
AP数

300

4000

200

3000
150

100

50

0

J-PARCofLan上のエッジスイッチ、無線AP、端末数
NumberHosts , edge SW and wireless AP on JLAN

17 High Energy Accelerator Research Organization Computing Research Center

2000

1000

0

Hadron Experimental Facility
Materials and Life Science
Facility (MLF)

Main Ring

Circumference: 1,568m

Neutrino Beam
3 GeV RCS

Neutrino Near
Detector

Circumference: 348m

Linear Accelerator
330m

High Energy Accelerator Research Organization Computing Research Center18

Research Activities

Research Activities
GRACE
GRACE is a software system for automatic computation of
scattering cross section in particle physics. It was developed collaboratively by theoretical, experimental, and computer specialists. The system is not only used for precise theoretical particle
physics analysis but also utilized for high-energy physics experiments through creation of programs for event generators toanalyze experiments. By utilizing software technologies such as
numerical computation techniques including integration, formula manipulation techniques for dealing with Feynman diagrams,
and program/parallelization techniques for high-speed computation, as well as a system dedicated to high-speed multipleprecision calculation, this system continues to provide the highspeed, precise, and large-scale computing required for the ILC.

Manyo-Lib: Analysis software frameworkfor neutron scattering experiments“Manyo-Lib” is a software framework for developing analysis software at the Materials and Life Science
Facility (MLF) in J-PARC. MLF provides one of the mostintense pulsed neutron and muon beams for various
materials and life sciences experiments.
Manyo-Lib comprises a C++ class library with a python interface, and provides data analysis operators,
data containers, network distributed data processing
modules, user interface modules, etc.
Manyo-Lib has been applied to develop data analysis environments operating on 20 neutron scattering
instruments in MLF. The figures presented are screenshots of these instruments.
The user manual and the source code of ManyoLib can be obtained at its web page:https://wiki.kek.jp/display/manyo/Manyo+Library+Home

19 High Energy Accelerator Research Organization Computing Research CenterGeant4
Geant4 is software for simulation of the interactions between elementary particles and materials inhigh-energy physics experiments. It is developed and maintained under the international collaboration. Geant4 is widely used in other fields such as medicine and astronomy. CRC collaborates with medical users
and extends Geant4 applications to the simulation of particle therapy.

DAWN: Software for high-quality rendering ofdetectors and particle trajectories. The figure
shows ATLAS detector.

Opacity curve and color map editor

gMocren: Visualization of CT-scanned images
and simulation result.

MPEXS

The graphic processing unit (GPU) is generally used forcomputer graphics. It is a particular device that has several
thousands of parallel-processing cores. In recent years, different scientific applications, including AI, make big leaps
in accelerating applications using GPU as general-purpose
processors. CRC is developing a GPU-based radiation simulator called "MPEXS." The main target of MPEXS in thefields of medicine and biology. Taking advantage of GPU's
high computing power, we work on the acceleration oftreatment planning in particle therapy and the study of
biological effect by radiation. So far, we realize 400x speed
up comparing Geant4 simulation. In the future, we plan
MPEXS extensions to the simulation in high-energy physics.

NVIDIA A100 GPU

High Energy Accelerator Research Organization Computing Research Center20https://www.kek.jp

Inter-University Research Institute Corporation
High Energy Accelerator Research Organization

Computing Research Center
1-1 Oho, Tsukuba, Ibaraki 305-0801
Phone +81 29-864-5473
Fax +81 29-864-4402