New JCAHPC Supercomputer System
Joint Center for Advanced HPC (JCAHPC)
Center for Computational Sciences, University of Tsukuba
Information Technology Center, The University of Tokyo

Overview
We decided to introduce a new supercomputer system “OFP-II” (tentative, aka. Post-OFP)
with a total peak performance of 79.5 PFLOPS as a successor of Oakforest-PACS. We will
start its operation in January 2025. This system will be installed in the Kashiwa campus,
the University of Tokyo, at the exact same location as the Oakforest-PACS system.
The OFP-II will be offered to researchers in Japan and their international collaborators through various types of programs
operated by HPCI, by MEXT’s Joint Usage/Research Centers, and by each of us under original supercomputer resource
sharing programs. Inheriting the philosophy of the introduction of Oakforest-PACS, OFP-II aims to promote novel
computational science methods leveraging AI, such as AI for HPC/Science, to offer a platform supporting Society 5.0 byintegration of simulation, data analysis, and machine learning, in addition to support users of large-scale applications.

Schedule

2023

2022
345GPUPre-bench

6…

Decide
GPU

11121

2024

23456

7

RFC

RFI

891011RFP

Final Spec.

Draft Spec.

1212025
2…

ContractAward Installation

1The OFP-II system has 1,120 nodes for Accelerator node, each of which
consists of NVIDIA GH200 Grace Hopper Superchip (Arm Neoverse V2core, cache-coherent with GPU), 190 nodes for General-purpose CPU
node, and NVIDIA/Mellanox InfiniBand NDR as a high-performanceinterconnect. This is the first large-scale system with GH200 in Japan.
The system will be installed by Fujitsu, and Supermicro and Fujitsu
PRIMERGY servers are employed as compute nodes. The system employs
the shared files system by All Flash with 10.3 PB provided by Data Direct
Network (DDN). In addition, “Ipomoea-01” is also available as a largescale external filesystem to be coupled to OFP-II.
The peak performance of OFP-II becomes 79.5 PFLOPS and the total
memory BW is more than 5.6 PB/sec. All compute nodes and servers
organizing the filesystem are connected by fat-tree topology based on
InfiniBand-NDR, which provides full-bisection bandwidth. Therefore,
flexible and efficient utilization and operation of compute nodes and filesystems are available.

CPU: NVIDIA Grace(72 core, 2.6 GHz, 117MB L3 Cache)
Mem: 120 GB (LPDDR5X, 512 GB/sec)
GPU: NVIDIA H100
(66.9 TFLOPS, NVLink C2C 450 GB/sec)
Mem: 96 GB (HBM3, 4.022 TB/sec)

InfiniBand NDR200(200 Gbps)(56 core, 1.9GHz, 112.5MB L3 Cache) x2

Mem: 128 GiB (HBM2E, 3.2 TB/sec)

InfiniBand NDR200(200 Gbps)

InfiniBand NDR (400 Gbps), Full-bisection Fat-Tree
1.0 TB/s

Shared Filesystem
Lustre FS

10.3 PBAll Flash

DDN ES400 NVX2 x10

GRACECPULPDDR5X
120 GB

512 GB/s72c, 2.6 GHz

PCIe Gen4x4

NVMe SSD
1.92 TBServer

Login
Login
Login
Login
nodenodenodenode
For CPU node& PrepostIntel Xeon 8480+ x2× 190

Processor

ExternalConnecting
RouterEthernetRDMA

Login node + Prepost
Login
Login
Login
Login
nodenodenodenode
For ACC node
NVIDIA Grace
CPU Superchip

Contact: Toshihiro Hanawa (hanawa@cc.u-tokyo.ac.jp)
Information Technology Center, The University of Tokyo

General-purpose CPU

78.2 PFLOPS

1.29 PFLOPS

1,120190
241.9 TB
23.75 TiB
5.07 PB/sec
608 TB/sec
InfiniBand NDR200 (200 Gbps)
Full-bisection Fat Tree
Lustre FS
DDN ES400NVX21 (4)appx. 23.5 BDDN ES400NVX210 set
10.3 PB (All Flash)
1.0 TB/sec

NVLink C2C
450 GB/s

HopperGPU

HBM396 GB

4.022 TB/s
PCIe Gen5x8

IB NDR HCA
ConnectX-7
IB NDR200(200 Gbps)

CPU: Intel Xeon CPU Max 9480 x 2socket
× 1,120

Accelerator

NVIDIA GH200 Grace Hopper Superchip

Node Type

Fujitsu PRIMERGY Server

CPU+GPU: NVIDIA GH200 Superchip

Node Type
Theoretical Peak
Performance
Number of Nodes
Total Memory Capacity
Total Memory BW
Interconnect
Topology
Shared Filesystem
ServerMDS # of Servers (VM)
# of inodesServer# of Servers
OSS
Capacity
Theoretical BW

Block diagramof Acc. node

Node Specification

CPU node: 1.3 PFLOPS, 608 TB/s

Supermicro…

Operation

Configuration of OFP-II System

ACC node: 78.2 PFLOPS, 5.07 PB/s

Specification of OFP-II System

Common Large-Scale
Filesystem (UTokyo)
Ipomoea-01
Lustre FS

26 PB# of CPUs (Core)
Frequency
CPU Theoretical Peak
Perf.
MemoryCapacity /node
BW /node
Processor# of GPUs# of SMs
Theoretical Peak
Perf.
GPU Memory
CapacityBW
CPU-GPU conn.

SSD NVMe SSD

Supermicro
ARS-111GL-DNHR-LCC
NVIDIA GH200
Grace Hopper Superchip(CPU: NVIDIA Grace)
1 (72)
2.6 GHz

General-PurposeCPU node
FUJITSU Server
PRIMERGY CX2550 M7Intel Xeon CPU Max 9480
（Codename:Sapphire Rapids）
2 (56+56)
1.9 GHz2.99 TFLOPS

6.80 TFLOPSLPDDR5X
120 GB
512 GB/s
NVIDIA Hopper
1132

HBM2E
128 GiB
3.2 TB/s

Accelerator node

66.9 TFLOPSHBM396 GB
4.02 TB/s
NVLink C2C 450 GB/sec
Cache-coherent
1.92 TB, PCIe G4 x4

N/A

N/A