German x Japanese Supercomputing – a success story
Sebastian Pokutta
NHR @ ZIB, Representative
Zuse Institute Berlin (ZIB), Vice President
Technische Universität Berlin, Professor
Tokyo University, JSPS Visiting Fellow

Tokyo, Japan
11th July 2024

⽇独科学技術協⼒ | Wissenschaftlich-Technologische Zusammenarbeit (WTZ)

NHR Alliance
University High Performance Computing in Germany

together with the states of Germanyfedena ral/s
tionw tateide funac dince gss

NHR as Part of Germany’s HPC Infrastructure

Tier 0EuropeanTier 1
Germany

Tier
Tier 22
Regional/ TopicalCenters
NHR / TopicalCenters• Recommendation by Research Council tointroduce Tier-2 National High Performance
Computing (NHR) infrastructure• Competitive applications in 2020
– official start: Jan 1, 2021
– total funding 625M Euro (2021-2030)– currently 9 NHR centers• Key aspects
Tier 3
Local Centers– joint federal/state-funding– transition from regional centers to competenceoriented centers for nationwide use– free access for all researchers from German
universities– strengthen methodological competences throughcoordinated training, continuing education of users– targeted support for young scientists

together with the states of Germany

NHR Alliance - Members§

Rhein-Westfälische Technische Hochschule Aachen§

Zuse Institut Berlin (ZIB)§

Technische Universität Darmstadt§

Technische Universität Dresden§Friedrich-Alexander-Universität (FAU) Erlangen-Nürnberg§

GWDG/Georg-August-Universität Göttingen§

Karlsruher Institut für Technologie§

Johannes Gutenberg Universität Mainz for South-West
Consortium (Goethe-Universität Frankfurt, Rheinland-Pfälzische
Technische Universität Kaiserslautern-Landau, Johannes
Gutenberg Universität Mainz, Universität des Saarlandes)§Universität Paderborn

Executive Board
Christof Schütte (ZIB)
Gerhard Wellein (FAU Erlangen)
Christian Plessl (Uni Paderborn)

NHR Office
Dörte Sternel (Managing Director)

together with the states of Germany

Topic Specialization
Life ScienceSimulation• Centers coordinate to provide
broad and complementary
coverage ofLife Science
DataChemistry– scientific domains– methods– technologies• Goal: provide tailoredhardware, software, support,
training• One coordinating center per
domain/topic– but no exclusive representation/ responsibility– users can still apply at any
center for their projects

Engineering
CFD

Physics

Humanities &SocialSciences

Engineering
MaterialScienceEarthSystems

together with the states of Germany

Methodological Focuses
Parallelization andParallel Programming
ModelsPerformanceEngineering

Research Software
Engineering

Artificial Intelligence(Big Data)

Numerical Methods
and Libraries

Performance andScaling Tools

Artificial Intelligence(Machine Learning)

together with the states of Germany

NHR Resources• Available free of charge to all scientists of German accredited universities
Computing time allocation• On a project-by-project basis according to a joint science-guided procedure• NHR center freely selectable by applicants• Central application portal (JARDS)
Training program• Coordinated training program across centers• Announcement via mailing list NHR announcements and website• Courses on various HPC topics
Consulting• Center-specific focal points

together with the states of Germany

•NHR Alliance• Overview

NHR @ ZIB
Overview

ZIB in a Nutshell

Applied Mathematics and Computer Science

Nanotech
EnergyMedicine
HPC +Artificial
Intelligence

Quantum
Compu?ng
Agent-based
ModelsMobility

Sustainability

Supercomputer LISE
14 Racks

for the compute nodes, with infrastructure for cooling using warm water and air.

1270 Compute Nodes

in total, each with two Intel Xeon Cascade Lake Advanced Processors and 48 computing
cores, resulting in 96 cores per node; of which 1236 are standard nodes with 384 GB of mainmemory, 32 nodes with 768 GB of main memory, and 2 nodes with 1.5 TB of main memory
per node.

2540 CPUsof the type Intel Xeon Platinum 9242 (2.3GHz) with 48 computing cores each.

121,920 Computing Cores

is the total number of physical cores; with hyperthreading, the double number of logicalcores is available for highly parallel applications.

Fat Treeis the topology of the HPC interconnect, an Intel Omni-Path network, enabling low latency
and high bandwidth communication within the entire system.

8 PetaFlop/s

is the theoretical peak performance of the “Lise” computer; a performance of 5.4 PetaFlop/s
was demonstrated in LINPACK.

502 TByteof distributed main memory is available to users across all compute nodes.

8,4 PByteof online storage capacity can be utilized on global parallel file systems (DDN Lustre andGRIDScaler).

System Users in 2023-2024
1,000+ Users and 250+ Projects
0.5%

Scientific Domains (2023-2024)

6.9%Life Sciences

26.3%

Chemistry
Physics

18.5%

Mathematics
Earth System Sciences
Engineering
8.7%

Humanities and SocialSciences

24.1%
15.0%

Current “LISE” architecture
1⃣

2⃣

DAOS
0,5 PB

Lustre
10 PB

GPFS
0,35 PB

ObjectStore

WORK

HOME

100 Gb/s
OPA200 Gb/s
IBGW

400 Gb/s

200 Gb/s
IBIntel
CLX-AP

Nvidia A100 GPU-Partition
42 GPU Nodes
IB HDR 200 Gb/s
Fat Tree 1:1

Intel MAX GPU-Partition
8 GPU Nodes
IB HDR 200 Gb/s
Fat Tree 1:1

3⃣

CPU-Par''on
1270 Cascade Lake AP
Nodes
OPA 100 Gb/s
Fat Tree 1:1

GW

400 Gb/s

200 Gb/s
IB NDR-200

AMD Genoa CPU-Partition
168 CPU Nodes
IB NDR-200 Gb/s
Fat Tree 1:1

Base Services: IdM/LDAP, DNS, SLURM, AccounLng,
User/Project Management, Monitoring

10 Gb/sEthernet

New Einstein Research Unit: Quantum Computing

Perspectives of a quantum digital transformation:near-term quantum computational devices andquantum processors

Research questions from the mission statement:§ What is the precise computational power of such quantum computers?
§ For what specific problems do they oﬀer significant speedups?
§ Can one gain new insights into quantum chemistry and high energy physics?
§ Would an improved solution of combinatorical problems be within reach, with potentially farreaching and economically viable consequences?
§ Is progress in quantum-assisted machine learning possible, again with potentially significantimplications?
§ And technologically speaking, how can we devise new integrated or distributed computational
15
building blocks with quantum optical means?

Some German-Japanese Success Stories

•NHR Alliance• Overview

Graph 500
BFS and SSSPBFS Benchmark

http://graph500.org• At the ISC'21, the Graph500 benchmark of the world's fastest computers for handling graph-based
problems was presented. Using the BFS Benchmark at Scale 38, LISE at ZIB achieved the 7th placeby traversing 5.4 trillion edges per second, running the same code as the 1st placed Fugaku.
• The benchmark performance was achieved as part of a collaboration between ZIB and theKyushu University using software developed by Katsuki Fujisawa and his team and adapted for
LISE by Yuji Shinano (ZIB).

Supercomputer Fugaku
RIKEN Center for Computational Science(R-CCS)

Lise
Zuse Institute Berlin (ZIB)

No.1

No.8

on the Graph500 BFS Ranking of Supercomputers with102955 GE/s on Scale 41
on the 24th Graph500 list published on May 30 2022

on the Graph500 BFS Ranking of Supercomputers with5423.94 GE/s on Scale 38
on the 24th Graph500 list published on May 30 2022

Congratulations from the Graph500 Executive Committee

Congratulations from the Graph500 Executive Committee

is ranked

is ranked

Graph500 SSSP (Single Source Shortest Path) http://graph500.org• Based on this success, a collaboration to extend the code to the more complex SSSP Benchmark
was started where researchers in the Research Campus MODAL in Berlin together with their
Japanese colleges achieved an impressive performance, that still stands. (Note the much longer
construction time of entries 1 and 2)

•NHR AllianceShortest Vector Problem• Overview

Massively Parallel Solver

SVP Challenge

https://www.latticechallenge.org/svp-challengen Lattices and SVP
pAn 𝑛-dimensional lattice is the discrete set$ℒ 𝐁 = % 𝑥! 𝐛! ; 𝑥! ∈ ℤ

MAP-SVP
Massively Parallel Solver for
Shortest Vector Problem!"#

where 𝐁 = 𝐛# , … , 𝐛$ are linearly independentvectors. (This 𝐁 is called a “basis”.)
pThe Shortest Vector Problem (SVP) asks to find ashortest non-zero vector in the lattice:• minimize 𝐯• subject to 𝐯 ∈ ℒ(𝐁) ∖ {𝟎}New records for 104, 111, 121 and 127
dimensions in SVP Challenge using up to103,680 cores

N.Tateiwa, Y.Shinano, S.Nakamura, A.Yoshida, S.Kaji, M.Yasuda, and K.Fujisawa. Massive parallelization for finding shortest lattice vectors basedon ubiquity generator framework. DOI 10.1109/SC41405.2020.00064

•NHR Alliance
Quadratic Assignment Problem• Overview

Solutions to Previously Unsolved InstancesQuadratic Assignment Problem (QAP)
ParaQapNB = ug[QapNB, MPI]: Massively parallel solver for QuadraticAssignment Problem (QAP)
Solved 3 previously unsolved QAP instances from QAPLIB
https://www.miguelanjos.com/projects/2999-the-qaplib-challengenn

HPE SGI 8600 (384 nodes, 13,824 cores)
HLRN-IV System (1270 nodes, 230,000 cores)

min $ 𝑓$& 𝑑!"!#!∈#!

$,&n! feasible solutions
40! = 8.2e47 — toomany combination

Joint work of:
Koichi Fuji
Masakazu Kojima
Yuji Shinano
Hans Mittelmann
Sunyoung Kim

•NHRLarge
Alliance
Scale Chebyshev Filter Diagonalization• Overview

Very large-scale Eigenvalue Computations

Large Scale Chebyshev Filter Diagonalization
Block vectors of size𝒏𝑺 & 𝒏(row-major)

Polynomial
degreeSparse matrix multiple
vector multiplication(spmmv)• Computing 100 inner eigenvalues onmatrices up to 𝑛 = 4×10$

𝑛= 2.1×10%
𝑛𝑜𝑑𝑒𝑛& = 500; 𝑛' = 128

FUSED kernel(×*+,3!"# )/2$ 4!"#For double complex matrix entries [1,2] : 𝐼 𝑛' = (.+(-.×*• Sparse matrix H from topological insulator model• #non-zeros per row: 𝑁$%& = 13 (double complex)

0.4 PFLOP/s

Typical Application:
Topological Insulator

0.5 PFLOP/s

Joint work of:
M. Kreutzer, D. Ernst,
A. R. Bishop, H. Fehske,
G. Hager, K. Nakajima,
and G. Wellein

•NHR Alliance
Improving Communication• Overview

In heterogeneous clusters

Improving Communication
Modelling communication & computation within
Wisteria/BDECOdyssey
Aquarius

Aquarius

Modelling of basic kernels on Odyssey/ Aquarius
On going
joint work between:
Univ. of Tokyo (K. Nakajima, S. Sumimoto)and FAU (A. Afzal, A. Ujeniya, G. Wellein)

Aquarius

Sync

Odyssey

DisCostiC modelling framework (FAU)
No Sync

Odyssey

LoadA64FX< ICLA64FX= ICL systems
A64FX > ICLUnderstand how performancefluctuations
travelbetweenExperiment

Model

Joint MA-Thesis finalized (2024)
Joint JHPCN project•NHR Alliance
Certifying Optimality of Constructions• Overview

Solving very large-scale SDPs

1.0.7

The Dual SDP

Certifying
Proposition 1.7Optimality(dual SDP). Theof
dualConstructions
SDP to Proposition 1.5 iss.t. 8G 2 G0l

maxd2R dX

lk

k+b 0 cd+hX, Dk,G 2 i  bG
kl with k⌘l mod 2(1.0.3)

X<0• Extremely large-scale SDPs to be solved to certify optimality of combinatorial constructions
Theinstances
TangenttoCone•1.0.8
Very large
be solved (small ones are of size 70GB float sdpa file)
It is common practice (called phantom edges [4]) in Flag Algebra theory to consider a•Question:> 3m constraints/variablesof feasible points n for Proposition 1.5 converging to the conjectured optimum 0 . Then•sequenceWillbe solved
using2a modified version of SDPA / SDPARA (on TSUBAME and/or LISE)
Tif b n = bT 0 + o( n1 ), so the variation to the conjectured optimum is small, then we know that
122014 IEEE 28th International
Parallel & +for the derivativein n =· n1Processing+ o(Symposium
0 Distributedn ) we must have for the dual SDP in Proposition 1.7
the additional constraint that
T Semidefinite
Petascale General Solver
hX, fori=0()X =0Programming Problems
These are constraints
that arenot capturedby the complementary condition.
with over Two
MillionConstraints

The question is: Clearly is in the tangent cone at 0 . Will other tangent vectors give additional
Toshio Endo
Yuichiro YasuiHitoshi Sato
constraints?
Willnormal vectors
give additional
constraints? Provided that bT n = bT 0 +Chuo University
Tokyo otherInstitute of Technology
Chuo University
Tokyo Institute of Technology&& JST CREST& JST CREST& JST CREST
k JST CREST
JapanTokyo, mayJapanTokyo, Japano( n1 Tokyo,) withk > 2 wetake derivativesof nTokyo,
upJapanto order k 1 too. How does this relate tofujisawa@indsys.chuo-u.ac.jp
endo@is.titech.ac.jp
yasui@indsys.chuo-u.ac.jp hitoshi.sato@gsic.titech.ac.jp
existing concepts in SDP theory?
Joint work of:
Katsuki Fujisawa

Naoki Matsuzawa

Satoshi Matsuoka

Hayato Waki

University of Tokyo
Tokyo, Japannaoki matsuzawa@ipc.i.u-tokyo.ac.jp

Tokyo Institute of Technology,
JST CREST & NII
Tokyo, Japanmatsu@is.titech.ac.jp

Kyushu University& JST CREST
Fukuoka, Japanwaki@imi.kyushu-u.ac.jp

Abstract—The semidefinite programming (SDP) problem isory, economics, quantum chemistry, sensor network location,

K. Fujisawa, A. Kiem,
S. Pokutta, and C. Spiegel

Thank you
for your attention!
German x Japanese Supercomputing – a success story
Sebastian Pokutta
NHR @ ZIB, Representative
Zuse Institute Berlin (ZIB), Vice President
Technische Universität Berlin, Professor
Tokyo University, JSPS Visiting Fellow

Tokyo, Japan
11th July 2024