HPCI Computing Resource Handbook©

Registered Institution for Facilities Use Promotion / The Representative for HPCI Operation

Research Organization for Information Science and Technology (RIST)©

2024.01

CPU Architectures
Hokkaido U.



Tohoku U.

High-Performance Computing Infrastructure (HPCI)The HPCI initiative established by Japanʼs Ministry of Education, Culture, Sports, Science, andTechnology (MEXT) promotes supercomputing resources for a shared computational environment.
The HPCI initiative provides an innovative shared computing environment that meets diverse userneeds by connecting world-class, advanced supercomputers and storage devices installed at
Japanese universities and research institutions, including Fugaku, through a high-speed network.

The organizations operating HPCI systems
RIKEN Center for Computational Science

Academic Center for Computing and Media Studies, Kyoto University

National Institute of Informatics

Cybermedia Center, Osaka University

Information Initiative Center, Hokkaido University

Research Institute for Information Technology, Kyushu University

Cyberscience Center, Tohoku University

Center for Earth Information Science and Technology, Japan Agency for Marine-Earth

Center for Computational Sciences, University of Tsukuba

Science and Technology(JAMSTEC)

Joint Center for Advanced HPC（JCAHPC）

Center for Engineering and Technical Support, The Institute of Statistical Mathematics(ISM)

Information Technology Center, The University of Tokyo

Information Technology Research Institute, National Institute of Advanced Industrial

Global Scientiﬁc Information and Computing Center, Tokyo Institute of Technology

Science and Technology (AIST)

Information Technology Center, Nagoya University

RIKEN Information R&D and Strategy Headquarters (R-IH)

 

 

 

U. of Tsukuba AIST

R-IH

HOKUSAI BigWaterfall2 

Nagoya U.

Flow Ⅰ Flow Ⅱ ­

Kyoto U.

RIKEN
Kyushu U.

 

 

Tokyo Tech ISM- - JAMSTEC



­



 ­ ­

U. of Tokyo ­

Osaka U.



JCAHPC

 ­

 





­

­

Supercomputer System for Data Assimilation 

Earth Simulator (ES4)VE nodeCPU node 

The main supercomputers at each institution are shown above. For more details, please refer to the website below.

https://www.hpci-office.jp/en/using_hpci/hardware_software_resource


Processors with x86-64 instruction sets manufactured andsold by Intel for servers or workstations
Processors with x86-64 instruction sets designed anddeveloped by AMD based on the Zen microarchitecture









x86-64 host processors with NVIDIA HPC GPUs foracceleration
x86-64 host processors with NEC's Vector Engine foracceleration

Fujitsu Arm microprocessors compliant with Armv8.2-A
SVE

1

Supercomputing Resources by SoftwareName of Software

RIKEN


Molecular Dynamics

Quantum Chemistry

Condensed Matter
Physics2

Hokkaido U.
   







Tohoku U.
 
 


 






U. of Tokyo
U. of Tsukuba Tokyo Tech
U. of Tokyo

Data is current as of January 2024, but subject to change.
Some institutions have restrictions on the use of commercial software.
Please contact our help desk for the latest information

Data is current as of January 2024, but subject to change.
Some institutions have restrictions on the use of commercial software.
Please contact our help desk for the latest informationNagoya U.

     ­ Flow     
      Flow 

 Kyoto Osaka
U.
U.

 





Kyushu U.





JAMSTEC
Earth
Simulator  
AIST

ISM



R-IH

Name of Software

Supercomputerfor Data HOKUSAI  System
BigWaterfall2
Assimilation









   

  




Φ
­
  
   


RIKEN

Computational Biology

Fluid Analysis

Structural / Collision
AnalysisElectromagnetic Field
Analysis

Multi-physics

Particle Systems

Weather / Climate


Hokkaido U.
   







Tohoku U.
 
 


 






U. of Tokyo
U. of Tsukuba Tokyo Tech
U. of Tokyo


Nagoya U.

     ­ Flow     
      Flow 

 Kyoto Osaka
U.
U.

 








  Kyushu U.

JAMSTEC

AIST





Earth
Simulator

ISM

R-IH

Supercomputerfor Data HOKUSAI  System
BigWaterfall2
Assimilation 
­


  (※1)
 ­­ 


 
­

 
 

  (※2)



 

 
※1…Particle-based Casting Simulator※2…Magnetic Simulator

3

Supercomputer Fugaku

RIKEN Center for Computational Science

Fumiyoshi Shoji

The supercomputer Fugaku oﬃcially became available for use on
March 9, 2021. Fugaku features CPUs based on the Arm architecture
widely used in smartphones and other devices, along withhigh-speed CPU-to-CPU interconnects. Taking a “co-design” approach, software and hardware engineers worked closely with eachother to develop a system versatile enough to eﬃciently executeapplications with diverse requirements across a range of ﬁelds.
The hardware conﬁguration is shown in the ﬁgure on the right.
Fugaku consists of compute nodes and IO nodes (storage, IO, boot),which are connected by an interconnect called “TofuD”. Each set of
16 compute nodes is equipped with a compute/storage node (approx.
1.6 TB of SSD storage). These compute/storage nodes constitutethe ﬁrst-tier storage. The ﬁrst-tier storage is used as a cache for thesecond-tier storage, and as a local ﬁle system for compute nodes
and a shared ﬁle system for jobs. The second-tier storage provides
a total of 6 volumes with a Lustre-based shared ﬁle system and a
total capacity of about 150 PB. The third-tier storage provides
servers for external cloud storage services.

Architecture

For more details, please refer to the website below.
https://www.r-ccs.riken.jp/en/fugaku/

SIMD）＋Fujitsu extensions

48 cores for compute and 2/4 for OS activities
Double-precision ﬂoating-point arithmetic : 2.7+ TF

CoreSingle-precision ﬂoating-point arithmetic: 5.4+ TF

Half-precision ﬂoating-point arithmetic: 10.8+ TF

L1D/core: 64 KiB, 4way, 230+ GB/s（load）, 115+ GB/s（store）

Cache
MemoryInterconnect
I/O
Technology

L2/CMG: 8 MiB, 16way
L2/node: 3.6+ TB/s
L2/core: 115+ GB/s（load）, 57+ GB/s（store）
HBM2 32 GiB, 1024 GB/s
TofuD（28 Gbps x 2 lane x 10 port）
PCIe Gen3 x 16 lane
7nm FinFET

Main system
Compute node
& BIO nodeCompute node
& SIO nodeCompute nodeCompute nodeCompute node
& BIO nodeCompute node
& SIO node

1st tier
storage

1st tier
storage

Compute nodeCompute node

& BIO nodeCompute node

For greater convenience, Fugaku is also expanding its cloud capabilities to include REST API access, a container execution environmentand orchestration tools, object storage, and high-speed connectivity
to commercial clouds.

Armv8.2-A SVE（512 bit& SIO nodeCompute nodeCompute node

& BIO nodeCompute node
& SIO nodeCompute node

Login and ﬁle transfer node (Intel)
Login and ﬁle transfer node (Arm)Login node with extra memory

Compute nodeCompute node
& GIO node

1st tier
storage

1st tier
storage

Front-end

Compute node

Visualization node
Cloud storage gateway
Web API node
Web API container node
Web API DB node

2nd tier storage
RIKEN network

Internet

3rd tier storage
Commercial cloudservices

Hardware conﬁguration

4©

5

Information Initiative Center, Hokkaido University

Interdisciplinary Large-scale Computing System
Masaharu Munetomo

In December 2018, Information Initiative Center of Hokkaido University
updated their “Interdisciplinary Large-scale Computing System”, which iscomprised of supercomputer and cloud systems. Performance of the newsupercomputer system is 20 times larger than that of the previous
system. Moreover, the cloud system involves an advanced nationwide
distributed computing environment, which is operated as the “Hokkaido
University High-Performance Inter-Cloud”.
The supercomputer system consists of two computing subsystems
named “Grand Chariot” and “Polaire”, and a storage subsystem of 16
PB. These subsystems are connected via a high-speed network basedon Omni-Path architecture. The total computing performance of the supercomputer system is 4PFlops. Multi-core and many-core processors
based on the x86 architecture are used for the system. The operating
system of the system is based on Linux. High performance compiler,
numerical libraries, and various tools for programming are available.
Furthermore, various open-source software can be used for the system.
These hardware and software stacks are expected to be used for
cutting-edge computational science and HPCI collaborative research.
Moreover, an originally designed job scheduler is used for the operationof the system, which provides the eﬃcient use of computing resources.
The “Inter-Cloud” system consists of high-performance cloud servers,
including baremetal (physical), virtual and GPU-equipped servers.
These servers are managed by the OpenStack cloud computing platform, which provides the users with a cloud system environment that
ensures performance and convenience.
In the “Inter-Cloud” system, an “Inter-Cloud” package is available. In thispackage, the user can use remote sites established at the University ofTokyo, Osaka University, and Kyushu University. Servers at the remotesites and Hokkaido University are connected via Japanʼs ultra-high speed
6

Science Information Network: SINET6. The users can simply apply and use
this wide-area distributed system immediately, eliminating the need to coordinate between sites.
Moreover, a remote backup site has been set up at the Kitami Instituteof Technology. Important data is regularly backed up to tape archiving
equipment there, ensuring that research data is safely preserved in theevent of a disaster.
For more details, please refer to the “Interdisciplinary Large-scale Computer System” page on the Hokkaido University Information Initiative Centerʼs website below.
https://www.hucc.hokudai.ac.jp/en/overview/ilcs/

Public cloud
100Gbps
RouterlaaS/HaaS

FW/VPNHINES（Campus LAN）

Administrativeand businessservers

laaS/
HaaS

（Physical/
virtual servers）

RemotebackupRemote sites(U. of Tokyo, Osaka U., Kyushu U., Kitami Inst. of Tech.)

Total computing
performance:
4 PFlops
Interconnect

High-Speed Network

laaS/ laaS/
HaaS HaaS（Physical/
virtual servers）

laaS/
HaaS

App

Supercomputer
Grand Chariot

WebDAV（Physical servers）

Cloud
storage

Supercomputer
PolaireSupercomputerstorage7

Cyberscience Center, Tohoku University

Supercomputer AOBA
Hiroyuki Takizawa

Tohoku Universityʼs Cyberscience Center has operated the Supercomputer AOBA since October 2020. In August 2023, this system underwent
major enhancements. AOBA consists of three subsystems. In addition to
the AOBA-A and AOBA-B subsystems adopting NEC SX-Aurora TSUBASA
B401-8 and NEC LX406Rz-2, respectively, the latest subsystem of
AOBA-S adopts NEC SX-Aurora TSUBASA C401-8. AOBA-A and AOBA-S
are vector supercomputers that oﬀer a good balance of computing power
and memory performance. They are particularly eﬀective for scientiﬁcand technical calculations, which are often memory intensive. For thisreason, AOBA-A and AOBA-S are expected to mainly run user-developed
code. AOBA-B is an x86 server using AMDʼs EPYC processors. It mainly
runs open-source software and commercial applications. The nodes of
AOBA-A and AOBA-B are connected via a high-speed InﬁniBand HDR
network, sharing a ﬁle system with a total capacity of 2 PB. The nodes of
AOBA-S are connected via a high-speed InﬁniBand NDR200 network,
sharing a ﬁle system with a total capacity of 4.5 PB.
Since 1997, Tohoku Universityʼs Cyberscience Center has worked
closely with users and computer vendors to optimize user-developed programs. This eﬀort resulted in many valuable user-developed applications
for the NEC SX-ACE system, which was the previous generation system.
Because the new AOBA system architecture is signiﬁcantly diﬀerent, wehave been working with users from the outset to help them migrate theirapplications to the new system. Also, the operating system is now a standard Linux environment. This makes it easier for new users to take fulladvantage of the performance of vector processors (especially its highmemory bandwidth) for scientiﬁc computing. We also help accelerate
new usersʼ computer programs.

simulations to predict potential damage from ensuing tsunamis. These
predictions are immediately forwarded to the Japanese government andother agencies to enable prompt and accurate response. Thus, AOBA is not
only for academic research. It also plays a critical and prominent role in mitigating damage to societal infrastructure during emergencies.
For more details, please refer to the Cyberscience Center page on Tohoku
Universityʼs website below.
https://www.tohoku.ac.jp/en/news/university̲news/tohoku̲university̲unveils̲new̲supercomputer̲aoba.html

Subsystem:

Vector Host: (VH)+8VE

C401-8User
Vector Engine (VE)
Type 30A (16 cores, 96 GB memory)

Storage system: (Lustre)
Working capacity: 4.5 PB

Subsystem:
Vector Host: (VH)+8VE

B401-8Vector Engine (VE)
Type 20A (8 cores, 48 GB memory)

Seminar terminal

Storage system: (ScaTeFS)
Working capacity: 2 PB

Subsystem:
Large format printers

AMD EPYC 7702(64 cores) ×2
256GB memory

In the event of a magnitude 7.0+ earthquake, AOBA can run emergency
8

9

Center for Computational Sciences, University of Tsukuba

Pegasus & Cygnus
Taisuke Boku

The University of Tsukuba's Center for Computational Sciences (CCS)
began operating the “big memory” supercomputer Pegasus in April
2023. Pegasus consists of 120 nodes with one NVIDIA H100 PCIe GPU
per node, with an overall theoretical peak performance of 6.5 PFlops.
Equipped with DDR5 memory and non-volatile memory, it can be usedas large capacity memory or ultra-high speed storage. Its greatly improved computational performance, memory size, and memory bandwidth are major advances for computational science and the ﬁelds ofbig data analysis and extremely large-scale AI.
We also operate Cygnus, a massively parallel GPU/FPGA cluster
system. Cygnus consists of a total of 78 nodes with 4 GPUs (NVIDIA
Tesla V100) per node. The theoretical peak performance excluding
FPGA is 2.34 PFlops. The compute nodes are connected by an InﬁniBand HDR100 4-port (400 Gbps) “fat tree” network topology with fullbisection bandwidth. In addition, 32 nodes are equipped with two Intel

10

Stratix10 FPGAs per node, connected by 100 Gbit optical links in an
8-by-8 2D torus.
At the University of Tsukuba's CCS, we promote a “Multidisciplinary Coo p e r a t i v e R e s e a r c h P r o g r a m” ( h t t p s : / / w w w. c c s . t s u k u b a . a c . jp/eng/use-computer/mcrp/ ) to help expand interdisciplinary computational science work throughout Japan. By providing Pegasus and Cygnus
as computing resources that can be used free of charge under this program, we are contributing to the advancement of computer science andcomputational science.
For more details, please refer to the website below.
https://www.ccs.tsukuba.ac.jp/eng/supercomputers/

Supercomputer Cygnus(PACS-X)

11

Information Technology Center, The University of Tokyo

Wisteria / Big Data & Extreme Computing
Kengo Nakajima

Bringing about the dawn of new science: The “Wisteria/BDEC-01”
supercomputer system, operated since May 14, 2021 by the Universityof Tokyoʼs Information Technology Center, combines computer Simulations with big Data and machine Learning (S+D+L). This is a hybrid
system with two types of compute nodes: simulation nodes (Odyssey)and data/machine learning nodes (Aquarius). With the human-centeredvision of Japanʼs “Society 5.0” initiative in mind, the “S+D+L” approach
incorporates expertise in data science and machine learning into computational science and computational engineering.
The Simulation nodes cluster (Odyssey) is equipped with 7,680 Fujitsu A64FX processors, the same CPUs used in supercomputer Fugaku,
for a peak performance of 25.9 PFlops. The Data & Learning nodes
cluster (Aquarius) is equipped with 90 Intel Xeon Platinum 8360Y (IceLake) CPUs, and 360 NVIDIA A100 Tensor Core GPUs, for a peak performance of 7.2 PFlops. Odyssey and Aquarius are connected by an
InﬁniBand EDR 100 Gbps network with a bandwidth of 2 TB/s. In addition, some Aquarius nodes can directly access various external resources, including servers, storage, and sensor networks, via networks
such as Japanʼs SINET, and can record data in real time for analysis
and simulations.
The Information Technology Center provides libraries, tools, and applications for a wide range of ﬁelds such as computational science,
data science, artiﬁcial intelligence, and machine learning. The centerhas also created open source software to make developing high-performing S+D+L applications easier. These include “ppOpen-HPC”, anapplication development and execution environment with automatictuning functionality, and “h3-Open-BDEC”, a novel software platform
for S+D+L applications.

12

The Wisteria/BDEC-01 supercomputer is the world's ﬁrst heterogeneous large-scale system that combines computer Simulations with big
Data and machine Learning. It is expected to play an important role in realizing the goals of Japanʼs “Society 5.0” initiative.
For more details, please refer to the website below.
https://www.cc.u-tokyo.ac.jp/en/supercomputer/wisteria/system.php

Platform for Integration of (S+D+L)
Big Data & Extreme Computing䝅䝭䝳䝺䞊䝅䝵䞁䝜䞊䝗⩌Simulation
Nodes
Odyssey
Fujitsu/Arm A64FX
25.9PF, 7.8PB/s

Shared File
Shared File
SystemSystemඹ᭷䝣䜯䜲䝹
(SFS)
䝅䝇䝔䝮
䠄SFS䠅
��.� PB,
���GB/s
25.8 PB, 500GB/s

2.0TB/s䝕䞊䝍䞉Ꮫ⩦䝜䞊䝗⩌
Data/Learning
Nodes
Aquarius
Intel Ice Lake + NVIDIA A100

7.20PF, 578.2TB/s

Fast File
Fast File
SystemSystem㧗㏿䝣䜯䜲䝹
(FFS)
䝅䝇䝔䝮
䠄FFS䠅
� PB,
�.�TB/s
1PB, 1.0TB/s

800 Gbps

External Network
External Networkእ㒊䝛䝑䝖䝽䞊䜽

External
Resources
External
Resourcesእ㒊䝸䝋䞊䝇

13

Global Scientiﬁc Information and Computing Center, Tokyo Institute of Technology

TSUBAME 4.0 (from April 2024)
Toshiya Itoh

Since 2006, the “TSUBAME” supercomputer series has been in use at the
Global Scientiﬁc Information and Computing Center, Tokyo Institute ofTechnology. TSUBAME 1.2 achieved a milestone in 2008, becoming a globally pioneering supercomputer with integrated GPUs, and then providing
GPU-based supercomputer resources as a service. In April 2024, the upcoming TSUBAME 4.0 will replace the current TSUBAME 3.0, operational
since 2017.
TSUBAME 4.0 has a theoretical performance of 66.8 PFlops (64-bit double-precision) used in scientiﬁc and technical calculations. It also boasts
an impressive theoretical performance of 952 PFlops (16-bit half-precision), particularly relevant in artiﬁcial intelligence applications. The system
consists of 240 interconnected compute nodes linked by a 200 Gbps x 4
port high-speed network. Each compute node is equipped with two AMD
EPYC 9654 processors, four NVIDIA H100 Tensor Core GPUs, 768 GB ofmain memory, and 1.92 TB of local SSD storage. Collectively, these nodes
share 44.2 PB of HDD storage and 327 TB of SSD storage.
Expanding on the TSUBAME concept of a "Supercomputer for Everyone",TSUBAME 4.0 allows for innovative use cases, including access throughweb applications. These user-friendly enhancements make TSUBAME 4.0
increasingly suitable for regular use by students and researchers across a
range of ﬁelds. Equipped with 960 of the latest GPUs, using its GPU logicalpartitioning and Linux resource partitioning contributes to improved performance and usability compared to its predecessor.
From April 2024, TSUBAME 4.0 computing resources will be provided to
HPCI initiative. We are also accepting industrial and apprentice users to
broaden the scope of supercomputer users. We also support challenging
projects, such as through our Budding Researchers Support program,which encourages young and female researchers to use TSUBAME, and14through our Grand Challenge Large-Scale Computing program, which encourages research results that can only be obtained by using all of TSUBAMEʼs nodes at once, thereby broadening the range of supercomputer users.
For more details, please refer to the TSUBAME Computing Services website below.
https://www.t4.gsic.titech.ac.jp/en

HPE Cray XD665 Server × 240
CPU

AMD EPYC 9654 (96 cores, 2.4GHz) × 2 Socket

GPU

NVIDIA H100 SXM5 × 4
FP64 33.5TFlops, FP64 Tensor 66.9TFlops, FP32 66.9TFlops
TF32 Tensor 494.7TFlops, FP16/BF16 Tensor 989.4TFlops,
INT8 Tensor 1978.9Tops
Memory 94GB HBM2e 2395.87GB/s

Memory768GiB (DDR5-4800)

Local storage
Network

1.92TB NVMe U.2 SSD
InﬁniBand NDR200 × 4

TSUBAME4.0 compute node conﬁguration

Rendering of the TSUBAME 4.0

New data center on the Tokyo

supercomputer

Tech Suzukakedai Campus
exclusively for TSUBAME 4.0

15

Supercomputer Flow

Information Technology Center, Nagoya University

Takahiro Katagiri

The Flow supercomputer marked the world's ﬁrst use of a Fugaku-type
node. It is available to any qualiﬁed user through an application process. It
is particularly useful for preliminary development of Fugaku-targeted software, allowing for a seamless transition to Fugaku itself. Flow is also
well-suited for the rapidly growing ﬁeld of data science. Subsystem II,with its powerful GPUs for machine learning and a massive 30 PB of “hotstorage” is particularly eﬀective for data science work. Each node of Subsystem II is equipped with 6.4 TB of NVMe SSD storage (1.4 PB in total),
and another 50 nodes (up to 320 TB) can be used to create a shared ﬁlesystem using BeeGFS. These features allow the high-speed ﬁle accessneeded for machine learning.

On July 1, 2020, Nagoya Universityʼs Information Technology Center
began operations of the supercomputer “Flow”, which consists of threesubsystems and a cloud system.
- Subsystem I (Fujitsu) features 2,304 Fugaku-type A64FX-based
nodes like those used at the RIKEN R-CCS.
- Subsystem II (NVIDIA) features 221 nodes, each equipped with 4
Tesla V100 Volta GPUs.
- Subsystem III (Hewlett Packard Enterprise) features a sizeable 48
TB of memory.
- The cloud system (Intel) features 100 nodes with 4-socket Xeon
Gold 6230 CPUs.
With a theoretical performance of 15.88 PFlops, it is one of Japanʼs
top supercomputers for numerical computations and data science. It
also oﬀers 6 PB of “cold storage” optical disc archiving in operation forthe ﬁrst time.

For more details about Flow, please refer to the website below.
https://icts.nagoya-u.ac.jp/en/sc/

Internet΢ϱνʖϋρφTypeγϔεητϞ
Type II subsystem

FUJITSU
FUJITSUPRIMEHPCPRIMEHPCFX1000
FX1000CPUʁFUJITSU
CPU: FUJITSU A64FX×1
૱όʖχ਼ʤαΠ਼ʥ
2,304όʖχʤ110,592αΠʥ
Nodes (cores): 2,304 (110,592
cores)
૱ԍࢋ੓೵
7.782 PFLOPS
Total ﬂops: 7.782
૱ϟϠϨ༲ྖ
Total memory:72
72TiB
TiB

SINET

Front-end servers (login nodes)ϓϫϱφΦϱχγʖώʤϫή΢ϱόʖχ‫܊‬ʥ
FUJITSU PRIMERGYFUJITSUPRIMERGYRX2530M5×18nodes
RX2530M5 ×18όʖχ
FUJITSU PRIMERGYFUJITSUPRIMERGYRX2530M5×18nodes
RX2540M5 ×7όʖχ
CPU: Intel Xeon
Gold
6248×2CPUʁIntel
Xeon
Gold
6248×2
GPU: NVIDIA Tesla V100 Tesla(PCIe)V100(PCIe൝)×2౧ࡎ
x 2 (only RX2540M5)
RX2540M5͹ΊNVIDIAόʖχ͍ͪΕ
3.2
TFLOPS,
384 GiBϟϠϨ
3.2 TFlops/node, 384 GB/node

NICE
಼ָLAN
(NICE)
CampusLAN(NICE)ηʖϏʖαΠη΢ρο
Supercore Switch

10G

InfiniBand
HDR/EDR
Type
Type II
II γϔεητϞ
subsystem

FUJITSU
FUJITSUCX2570M5
CX2570M5
Intel6230×2
Tesla
V100×4
IntelXeon
XeonGoldGold
6230×2+ +NVIDIA
NVIDIA
Tesla
V100×4૱όʖχ਼ʤαΠ਼ʥ221όʖχʤ25,656αΠʥNodes (cores): 221 (25,656
cores)
૱ԍࢋ੓೵
Total ﬂops: 7.489
7.489PFLOPSPFLOPS૱ϟ΢ϱϟϠϨ༲ྖ
Total main memory:82.875
82.875TiB
TiB֦όʖχͶ6.4TB͹NVMe
SSD౧ࡎɼSSDΝཤ༽ͪ͢
6.4 TB/node NVMe SSD,
shared ﬁle system‫༙ڠ‬ϓΟ΢ϩεητϞʤBeeGFS/BeeONDʥΝ఑‫ڛ‬(BeeGFS / GeeOND) on SSD

Hot storage
Cold storage
ϙρφηφϪʖζαʖϩχηφϪʖζ

DDN SFA18KE
Sony ODAPetaSitescalable library
DDN
SFA18KESony
ODA PetaSite֨௃‫ܗ‬ϧ΢ϔϧϨ
Total: 30PB
Installed౧ࡎ༲ྖʁ6PBʤ5.5TB×1091‫ת‬ʥ
Capacity: 6 PB (5.5 TB x 1091 cartridges)૱ࣰް༲ྖʁ30PB
Maximum
Capacity: 10.89 PB (5.5 TB x 1980 cartridges)࠹୉༲ྖʁ10.89PBʤ5.5TB×1980‫ת‬ʥDrives: ૱χϧ΢ϔ਼ʁ20χϧ΢ϔ
20

RemoteՆࢻԿ
Remote visualization

tiled display
8K 185-inch
185inchiν΢ϩχυΡηϕϪ΢
Samsung 46inchi
46inch Display×16
Samsung

Local
Localvisualization
ՆࢻԿImage processing clientժ଀ॴཀྵૹ஖

HDMIέϧΤχεητϞ
Cloud System

HPE
HPEProLiant
ProLiant DL560
DL560
CPUʁIntel
XeonGold
Gold6230×4
6230×4
CPU: Intel Xeon૱όʖχ਼ʤαΠ਼ʥ100όʖχʤ8,000αΠʥNodes (cores): 100 (8,000
cores)
૱ԍࢋ੓೵
Total ﬂops: 537.6
537.6 TFLOPSTFLOPS૱ϟϠϨ༲ྖ
Total memory:37.5
37.5TiB
TiB

16

TypeⅢIIIsubsystemγϔεητϞ
Type
HPE
HPE Superdome
SuperdomeFlex
Flex
Intel
Intel Xeon
XeonPlatinum
Platinum8280M×16
8280M×16
NVIDIA
×4
NVIDIA Quadro
QuadroRTX6000
RTX6000×4૱όʖχ਼ʤαΠ਼ʥ
Nodes (cores): 2 (896 2όʖχʤ896αΠʥcores)
૱ԍࢋ੓೵
77.414 YFLOPSTFLOPS
Total ﬂops: 77.414
૱ϟϠϨ༲ྖ
Total memory: 48
48 TiB
TiB
51.2TB/node
SSD+
1008TB
Type III༽‫༽ڠ‬ηφϪʖζ
51.2TB/node SSD + 1008TB Sharedstorage for Type III

Sony ODAχϧ΢ϔODA drive

Lenovo P920
P920
LenovoCPU: Intel
5122 (3.6GHz/4αΠʥ(3.6GHz/4 cores)
CPUIntel Xeon
Xeon Gold 5122
GPU: NVIDIA
RTX8000GPU
NVIDIA Quadro
Quadro RTX8000
Memory:512GiB
512GiBϟϠϨ
Storage:2TB
2TBHDD,
HDD,SonySony
ODA drive
Storage
ODAχϧ΢ϔ
Connected to NICE via 10G
֐෨ϋρφϭʖέʤNICEʥͳ10GͲંକ
On-siteclientΨϱγ΢φཤ༽ૹ஖
FUJITSU PRIMERGYPRIMERGY RX2530M5×2
RX2530M5×2
CPU Intel
Intel Xeon
Xeon Gold
Gold 5222 (3.8GHz/4αΠʥCPU:(3.8GHz/4 cores)GPU:
GPU NVIDIA
NVIDIA Quadro
Quadro P400
Memory:384GiBϟϠϨ 384GiB
Storage 4TB
Storage:
4TBHDD,
HDD,Sony
SonyODAχϧ΢ϔODA drive17

Academic Center for Computing and Media Studies, Kyoto University

Kyoto University Supercomputer Systems (Camphor / Laurel / Cinnamon / Gardenia)
Keiichiro Fukazawa
Kyoto Universityʼs Academic Center for Computing and Media Studies runs four super-computer systems: Operating since May 2023, the Laurel 3, Cinnamon 3, and Gardenia systems were joined in October 2023 by Camphor 3.

- Laurel 3 is a general-purpose supercomputer with Dell PowerEdge C6620 rack servers

with Intel Xeon Platinum 8480+ processors.

computers are intended to provide an environment that can meet the diverse computa-

tional needs of each user. A portion of these supercomputing resources support Japanʼs

HPCI and JHPCN initiatives.

The Academic Center for Computing and Media Studies also has its own supercomputer- Cinnamon 3 is a memory-intensive use Laurel 3 conﬁguration supercomputer with two

joint research system, under which three types of research incentive programs are oﬀered.

- Gardenia is an AI and machine learning system with DELL PowerEdge XE8545 rackage or female researchers regardless of age.

- Camphor 3 is a computational performance-oriented supercomputer with Dell Power-jobs.

terabytes of memory per node.

servers equipped with NVIDIA A100 Tensor Core GPUs.

Edge C6620 rack servers equipped with Intel Xeon CPU Max 9480 processors with HBM2ehigh-bandwidth memory.

The Camphor 3ʼs 1,120 nodes yield a total theoretical performance of 7.63 PFlops, while

the Laurel 3ʼs 370 nodes yield 2.65 PFlops. The Cinnamon 3 has only 16 nodes, but with fourtimes as much memory as the Laurel 3 conﬁguration. Gardenia also has 16 nodes, but eachnode is equipped with four A100 80GB GPUs. The diﬀerent features of these four super-- The ﬁrst program covers all or part of the usage fees for researchers under 40 years of- The second program covers part of the usage fees for research groups with certain large- And the third program supports improvements and reﬁnements to computer programs.

For more details about our supercomputer resources and joint research programs, pleaserefer to the Kyoto University Academic Center for Computing and Media Studies website
below.

https://www.media.kyoto-u.ac.jp/accms̲web/en/

High-speed networking communications InﬁniBand HDR/NDR

18

19

Cybermedia Center, Osaka University

Supercomputer SQUID
Susumu Date

Osaka Universityʼs Cybermedia Center oﬀers the use of supercomputer
system “SQUID” for cloud-linked high-performance computing and data
analytics. This system was ﬁrst made available in May 2021.

総演算性能
16.591 PFLOPS
Theoretical
16.591 PFlops
Computing Speed
汎用CPUノード群

プロセッサ：Intel Xeon Platinum 8368 (Ice Lake / 2.40 GHz 38コア) 2基
1,520 ノード(8.871 PFLOPS)
主記憶容量：256GB
General-purpose CPU nodes
CPU: Intel Xeon Platinum 8368 (Ice Lake / 2.4 GHz 38 cores) x 2
1,520 nodes (8.871 PFlops)
Memory:256Platinum
GB
GPUノード群
プロセッサ：Intel Xeon
8368 (Ice Lake / 2.40 GHz 38コア) 2基

SQUID, an acronym from “Supercomputer for Quest to Unsolved Interdisciplinary Datascience”, is a hybrid cluster system with general-purpose
CPU nodes, GPU nodes, and vector nodes, for a total computing performance of 16.591 PFlops. SQUIDʼs Lustre parallel ﬁle system with 20 PB of
HDD and 1.2 PB of SSD storage can be accessed using DDN's EXAScaler.
The processors and accelerators in each node cluster feature direct liquid
cooling and are designed and built for reliable high-performance. SQUID
can provide high-performance compute nodes with diﬀerent processors,
accelerators, and architectures within a single computing environment,
making SQUID distinctively capable of accommodating each userʼs diverse
computational needs.

ノード構成
Nodes

42 ノード(6.797 PFLOPS)
GPU nodes

主記憶容量：512GB
CPU: Intel Xeon Platinum 8368 (Ice Lake / 2.40 GHz 38 cores) x 2
GPU：NVIDIA HGX A100 8 GPUボード (Delta)
42 nodes (6.797 PFlops)
Memory: 512 GB

ベクトルノード群
36 ノード(0.922 PFLOPS)
Vector nodes

GPU:
NVIDIA
HGX
A100 /8 2.8
GPUGHz
board
プロセッサ
：AMD
EPYCDelta
7402P
(Rome
24コア) 1基

主記憶容量：128GB
CPU: AMD EPYC 7402P (Rome / 2.8 GHz 24 cores) x 1
Vector Engine：NEC SX-Aurora TSUBASA Type20A 8基
36 nodes (0.922 PFlops)
Memory: 128 GB

ストレージ
ノード間接続Storage
Interconnect

DDN EXAScaler (Lustre)

VectorEngine:NEC
SX-Aurora TSUBASA Type20A x 8
HDD：20.0
PB NVMe：1.2
PB

MellanoxInﬁniBand
HDR(200 Gbps)
DDN
EXAScaler(Lustre)HDD：20.0 PB NVMe：1.2 PB

Mellanox InﬁniBand HDR (200 Gbps)

SQUID system conﬁguration

Our faculty and staﬀ are ready to support you in using Osaka University
supercomputer. We hope youʼll give SQUID a try!
For more details, please refer to the Osaka University Cybermedia
Center website below.
http://www.hpc.cmc.osaka-u.ac.jp/en/

20

21Research Institute for Information Technology, Kyushu University

Research Computer System ITO
Kenji Ono

On January 9, 2018, Kyushu University's Research Institute for Information Technology made available the Fujitsu-based “ITO” supercomputer
system. This system is equipped with Intel Skylake Scalable PerformanceCPUs and NVIDIA Tesla P100 GPUs, for a total theoretical computing performance of approximately 10 PFlops, making it one of the most powerful systems in Japan.
The ITO system speciﬁcations were chosen for ﬂexibility, to help achieve
the super smart “Society 5.0” envisioned in the Japanese governmentʼs 5th
Science and Technology Basic Plan, and to provide a research platform for
artiﬁcial intelligence, machine learning, big data, data science, and more.
This is the ﬁrst supercomputer in Japan conﬁgured to connect, via ahigh-speed ﬁle system, the type of large private front-end cloud computing
environment needed for interactive data science and multi-system collaborative work with the type of high-performance back-end computing nodeclusters needed for large simulations and machine learning. In addition to
the standard front-end speciﬁcations, large memory types can be used,and virtual and bare metal servers can be reserved in advance through a
web-based system. ITO also provides a platform for new classes of users
and research projects, implementing a full-ﬂedged collaborative interface
for interacting with public clouds, and providing support for supercomputing that takes advantage of open data found on the Internet. We are also
working on new technologies to allow intelligent power-saving operations.
The ITO system introduces new detailed power monitoring and power-limited job scheduling functions.
Through the use of the ITO system, for various programs provided by ourcenter, providing it broadly to researchers inside and outside the university,and as a computing resource for Japanʼs JHPCN & HPCI initiatives, theKyushu University Research Institute for Information Technology contributes
in developing new academic research and in strengthening the foundations
22

for academic research in Japan.
This system will cease operations at the end of February 2024. The next
system going forward is planned to begin service from July 2024.For more details, please refer to the Kyushu University Research Institute for Information
Technology website below.
https://www.cc.kyushu-u.ac.jp/scp/eng/system/ITO/01̲intro.html

Front-end
Login nodes:

Standard front-end nodes: Large front-end nodes:
nodes

nodes

Interconnect

Subsystem A

Subsystem B

Back-end

Eﬀective
capacity : 24PB

Storage23

Center for Earth Information Science and Technology, Japan Agency for Marine-Earth Science and Technology

Earth Simulator 4
Hitoshi Uehara

The Japan Agency for Marine-Earth Science and Technology (JAMSTEC)
Research Institute for Value-Added Information Generation (VAiG) Center
for Earth Information Science and Technology (CEIST) has updated its“Earth Simulator” and made these supercomputing resources available forJapanʼs HPCI initiative from June 2021.
The upgraded Earth Simulator 4 supercomputer consists of CPU nodes
with AMD EPYC Rome processors on HPE Apollo, GPU nodes with NVIDIA
A100 GPU, Vector Engine nodes with NEC SX-Aurora TSUBASA, high-capacity DataDirect Networks storage, and a high-speed InﬁniBand network.
The Earth Simulator 4ʼs CPU and Vector Engine resources are available forJapanʼs HPCI initiative.
Its 684 Vector Engine nodes provide a total computing power of 14.97
PFlops with a total memory bandwidth of 8.5 PB/s, making this a powerfultool for research that uses vector computing. Its 720 CPU nodes, based on
the widely-used and versatile x86 architecture, provide a total computing
power of 3.3 PFlops and 180 TB of total memory, making it suitable for awide variety of research projects. These nodes can be used for one-oﬀ
batch jobs to meet the particular computational needs of various policy,
industrial, and academic projects.

For more details, please refer to the Earth Simulator website below.
https://www.jamstec.go.jp/es/en/

2 CPUsCPU
8 Vector Engines

2 CPUs8 GPUs

720 nodes

720 nodes

720 nodes

Interconnect network (HDR 200G InﬁniBand)

60PB

The data storage is a Lustre-based shared ﬁle system with 60 PB or 1.3
PB capacities, composed of HDDs or SSDs.
The storage is directly accessible from all nodes and front-end servers,
as are large-scale shared memory servers with 9 TB memory for pre-postprocessing.
The Center for Earth Information Science and Technology provides seminars on how to use the Earth Simulator, as well as robust support for porting and optimizing programs. Please consider using the Earth Simulator.

For large-scale data I/O

1.3PB

For short-time,
high-volume data I/O

Information Technology Research Institute, National Institute of Advanced Industrial Science and Technology

AI-Bridging Cloud Infrastructure
Yusuke Tanimura

To advance Japanese R&D in artiﬁcial intelligence and accelerate AIʼsuse in society, Japanʼs National Institute of Advanced Industrial Scienceand Technology (AIST) has made its AI Bridging Cloud Infrastructure(ABCI) available since August 2018. An upgrade in May 2021 gave the integrated “ABCI 2.0” system new energy-eﬃcient high-performanceGPU-based nodes and enhanced storage.
The upgraded ABCI is now a massively parallel cluster supercomputerwith 1088 “V-type” compute nodes, each consisting of 2 Intel Xeon Skylake
CPUs and 4 NVIDIA V100 GPUs, and 120 “A-type” compute nodes, eachconsisting of 2 Intel Xeon Ice Lake CPUs and 8 NVIDIA A100 GPUs, connected by a high-speed InﬁniBand network. ABCIʼs peak performance is56.6 PFlops at double precision, and 851.5 PFlops at half precision. It has
573.5 TB total memory, and 2.22 PB total NVMe SSD storage. The system
also includes a shared ﬁle system with an eﬀective capacity of 35 PB, and
17 PB of Amazon S3-compatible storage.

To meet the widely varying demands for AI computing resources for R&D
and community uses, ABCI does not draw a distinction between academic andindustrial uses. Anyone aﬃliated with a company, university, or research institute in Japan can apply and use ABCI for a ﬂat rate. ABCI is also available asan HPCI resource, so please consider it for your next project.
For more details, please refer to the ABCI website below.
https://abci.ai/

The system is installed at the AIST Kashiwa facilityʼs AI Data Centerbuilding, an ultra-high density, ultra-low power data center constructedprior to the installation. The AI Data Center employs a “free cooling”
system that produces chilled water using only a cooling tower. Computenodes are cooled with a combination of this chilled water and air from Fan
Coil Units. This results in an average annual Power Usage Eﬀectivenessratio of 1.1.
Since it began operating, ABCI has consistently held high positions on the
TOP500 ranking. In July 2019, it set a world record in the Image Classiﬁcation category on the MLPerf Training v0.6 benchmark for deep learning,taking only 70 seconds training time to recognize ImageNet images on a
ResNet-50 convolutional neural network. In November 2020, it also achieved
the highest speed on the MLPerf HPC v0.7 machine learning benchmark.
26

27

Center for Engineering and Technical Support, The Institute of Statistical Mathematics (ISM)

Supercomputer System for Data Assimilation
Genta Ueno

Data has ballooned in recent years. Obtaining useful insights
requires analysis of massive amounts of data from both real- w o r l d m e a s ur e m e n t s an d c o mp u t e r s imul a t i o n s . Mo r e o v e r,
the int egr ation o f dat a fr om those measur ement s and simulat i o n s , k n o w n a s “d a t a a s s i m i l a t i o n” , t e n d s t o c o m p o u n d t h i s
problem, producing ensemble data and reanalysis data that thenhave to be analyzed.

system has been made available for use as a computing resource
under Japanʼs HPCI initiative.

Working with such massive amounts of data is not easy, largely because most supercomputers rely on distributed memory all o c a t i on an d c anʼ t ﬁ t all t hi s m a s s i v e d a t a in s i d e a s in g l ememory space. Analyzing big data on these distributed memory
systems requires explicitly programming “parallel processing”
software that can separate, compute, and then reintegrate thedata. But, coding parallelization programs is rote and time-consuming work that doesnʼt change the ﬁnal results of the analysis. This work is a hindrance that should be avoided if possible.
Doing so requires a supercomputer equipped with massive
amounts of memory that can be used by any of its CPUs.
In March 2023, the Center for Engineering and Technical Support at the Institute of Statistical Mathematics launched its“Supercomputer System for Data Assimilation” to advance theanalysis of large-scale data without parallel programming. Thisis a distributed shared-memory computer with a large memory
space that can be accessed from any CPU. This system isequipped with t wo HPE Superdome Flex computing nodes andhas a total theoretical computing performance of 154.8 TFlops.
Each node is equipped with 32 28-core CPUs (Intel Xeon Platinum 8280L) with a main memory of 48 TB, and an SSD with 880
TB of usable capacity. Since October 2020, one node of this28

29

HOKUSAI BigWaterfall2

RIKEN Information R&D and Strategy Headquarters

Motoyoshi Kurokawa

In December 2023, the RIKEN Information R&D and Strategy Headquarters began operations of the supercomputer HOKUSAI BigWaterfall2(HBW2). To encourage internal research and development work, RIKEN
has operated supercomputer systems since the 1960s. The new HOKUSAI
BigWaterfall2 consists of a massively parallel computing system with a
total theoretical performance of 2.12 PFlops on 312 nodes equipped with
two Intel Xeon Max (Sapphire Rapids) and HBM2e (128GB), a two-nodelarge-capacity memory processing server with 3TB of memory, and 9.8 PB
of Lustre ﬁle system shared storage. They are connected by a 400Gb/s
InﬁniBand NDR network. The biggest advantage is that the compute nodes
are equipped with HBM2e, which feature a very high memory bandwidth of
3,260Gb/s.
Until now, supercomputers at the RIKEN Information R&D and Strategy

Massively Parallel Computing System (MPC)

Headquarters have been mainly used to drive research work byresearchers and engineers within the institution. But, it is getting diﬃcult
for a single institution to provide and maintain the increasingly varied computing resources needed for ever more diverse and specialized types ofresearch. To address this, Japanʼs collaborative High-Performance Computing Infrastructure (HPCI) initiative provides researchers with access to
Fugaku and other varied supercomputing resources. In 2024, we plan tomake a portion of HOKUSAI BigWaterfall2 availale as an HPCI resource.
This will make it easier for researchers at RIKEN to learn about HPCI
resources and use them as appropriate.
For more details, please refer to the RIKEN Information R&D and Strategy Headquarters website below.
https://i.riken.jp/en/supercom/

Large Memory Compute Server (LMC)

External Connections Network (Ethernet 10Gbps)

Front-End Server (FES)
chassis
nodes

nodes

Total theoretical performance (double precision) : 2.12PFLOPS
Total number of cores: 34,944cores(122cores/node)
Total memory : 39Tib(128GiB/node)

Total theoretical performance(double precision) : 12.2TFLOPS
Total memory : 6Tib(3TiB/node)

All-RIKEN network

High-speed network (InﬁniBand NDR/HDR)Management network (Ethernet 1/10Gbps)

Shared Storage (SFS)DDN ES400NVX2 x2
DDN ES400NVX2 x2
DDN SS9024 enclosures x8 (9.8 PB of HDD)

30

Management Servers (CTS)

System management server x1
Data management server x2
Log management server x1
Job management server x2

31

If you are considering using HPCI computing resources or would like to learn more about Japan s HPCI initiative,
please refer to the HPCI portal site below, or contact our help desk.

HPCI Portal Site  
Help Desk

helpdesk@hpci-oﬃce.jp

About the Research Organization for Information Science andTechnologyʼs Kobe Center
About the Research Organization for Information Science and Technology s Kobe Center
The Research Organization for Information Science and Technology s Kobe Center is responsible
for promoting and supporting the use of Fugaku and other Japanese supercomputers.
Supercomputer simulations across a wide range of ﬁelds are making major contributions toward amore safe and secure society. These include: elucidating the fundamental laws of matter and theevolution of the universe, realizing new sources of energy, analyzing genomic and intracellular
dynamics, ﬁnding physical materials with new properties and capabilities, making highly accurate
predictions of typhoons and tsunamis, eﬃciently designing new drugs, and allowing highly reliable
design and manufacturing research that alleviates the need for physical prototyping and experiments.

HPCI Computing Resource Handbook
January 2024 English version

Research Organization for Information
Science and Technology
Kobe Center

The RIST Kobe Center strives to make its world-class supercomputers available to researchers andengineers in a wide range of ﬁelds in a fair and eﬃcient manner so they can produce many fruitfulresearch results.

  *Please refrain from copying or reprinting thecontents of this booklet without permission.

©

2024 Research Organization for Information Science and Technology