重点課題９シンポジウム 2019年

1⽉9⽇

Overview of the Post-Kprocessor
ポスト京システムの概要と開発進捗状況

Mitsuhisa Sato Team Leader of Architecture Development Team
Deputy project leader, FLAGSHIP 2020 project
Deputy Director, RIKEN Center for Computational Science (R-CCS)
Professor (Cooperative Graduate School Program),
University of Tsukuba

FLAGSHIP2020 Project Missions• Building the Japanese national flagshipsupercomputer, post K, and• Developing wide range of HPC applications,running on post K, in order to solve social andscience issues in Japan Overview of post‐K architecture
Node: Manycore architecture••••Armv8‐A + SVE (Scalable Vector Extension)
SIMD Length: 512 bits
# of Cores: 48 + (2/4 for OS) (> 2.7 TF / 48 core)
Co‐design with application developers and highmemory bandwidth utilizing on‐package stacked
memory (HBM2) 1 TB/s B/W• Low power : 15GF/W (dgemm)

Network: TofuD• Chip‐Integrated NIC, 6D mesh/torus Interconnect

2019/01/09

Post‐K processor

Prototype board

 Status and Update• Close to end in “Design and Implementation”.
• The prototype CPU powered‐on anddevelopment is as scheduled• RIKEN announced the Post‐K early accessprogram to begin around Q2/CY2020• We are working on performance evaluationand tuning by simulators and compilers

2

KPIs on post-K development in FLAGSHIP 2020 project
3 KPIs (key performance indicator) were defined for post-K development
 1. Extreme Power-Efficient System 30-40 MW at system level 2. Effective performance of target applications
 It is expected to exceed 100 times higher than the K computerʼs

performance in some applications

 3. Easy-of-use system for wide-range of users

2019/01/09

3

CPU Architecture: A64FX Armv8.2-A (AArch64 only) + SVE (ScalableVector Extension)

 FP64/FP32/FP16(https://developer.arm.com/products/architecture/a-profile/docs)

 SVE 512-bit wide SIMD
 # of Cores: 48 + (2/4 for OS) Co-design with application developers and high “Common” programing model will be torun each MPI process on a NUMA node(CMG) with OpenMP‐MPI hybrid
programming.
 48 threads OpenMP is also supported.

CMG(Core‐Memory‐Group): NUMA node
12+1 core

memory bandwidth utilizing on-package
stacked memory: HBM2(32GiB)

 Leading-edge Si-technology (7nm FinFET),

low power logic design (approx. 15 GF/W(dgemm)), and power-controlling knobs

 PCIe Gen3 16 lanes
 Peak performance> 2.7 TFLOPS (>90% @ dgemm)

Memory B/W 1024GB/s (>80% stream)

Byte per Flops: approx. 0.4

HBM2: 8GiB
2019/01/09

4

ARM v8 Scalable Vector Extension (SVE) SVE is a complementary extension that does not replace NEON, and was developed

specifically for vectorization of HPC scientific workloads.

 The new features and the benefits of SVE comparing to NEON








Scalable vector length (VL)︓Increased parallelism while allowing implementation
choice of VL
VL agnostic (VLA) programming: Supports a programming paradigm of write-once,
run-anywhere scalable vector code
Gather-load & Scatter-store: Enables vectorization of complex data structures with
non-linear access patterns
Per-lane predication: Enables vectorization of complex, nested control code containingside effects and avoidance of loop heads and tails (particularly for VLA)
Predicate-driven loop control and management: Reduces vectorization overhead relative
to scalar code
Vector partitioning and SW managed speculation: Permits vectorization of uncounted
loops with data-dependent exits
Extended integer and floating-point horizontal reductions: Allows vectorization of moretypes of reducible loop-carried dependencies
Scalarized intra-vector sub-loops: Supports vectorization of loops containing complex
loop-carried dependencies
2019/01/09

5

SVE example

Make predicate
mask

SIMDwithmask

 Compact code for SVE as scalar loop
 OpenMP SIMD directive is expected to help the SVE programming
2019/01/09

6

CMG (Core Memory Group) CMG: 13 cores (12+1) and L2 cache (8MiB 16way) andmemory controller for HBM2 (8GiB)

 X-bar connection in a CMG maximize efficiency forthroughput of L2 (>115 GB/s for R, >57 GB/s for W)

 Assistant core is dedicated to run OS demon, I/O, etc
 4 CMGs support cache coherency by ccNUMA with on-chip

directory ( > 115GB/s x 2 for inter-CMGs)

Figures from the slidepresented in Hotchips
30 by Fujitsu

2019/01/09

7

FX64A Core Pipeline
 Superscalar Arch with out-of-order, branch prediction,

inherited from Fujitsu SPARC

 L1D cache: 64 KiB, 4 ways, “Combined Gather” mechanism on L1
 SIMD and predicate operations


2x 512-bit wide SIMD FMA + Predicate Operation + 4x ALU (shared w/ 2x AGEN)

2x 512-bit wide SIMD load or 512-bit wide SIMD store

Address Generation Unit

Figures from the slidepresented in Hotchips
30 by Fujitsu

Integer Exec Unit

Floating Pipeline

2019/01/09

8

Tofu interconnect D

Presented in IEEE Cluster 2018
By Fujitsu

 Direct network, 6-D Mesh/Torus
 28Gbps x 2 lanes x 10 ports (6.8GB/s / link) Network Interface on Chip
 6 TNIs: Increased TNIs (Tofu Network Interface) achieves higher

injection BW & flexible comm. Patterns

 Memory bypassing achieves low latencyRef) K computer: Link BW=5.0GB/s, #TNI=4

2019/01/09

9

Preliminary Performance by “real silicon”
 The prototype CPU has been powered-on and preliminary performance evaluation by the

prototype CPU has been done.

 Improvement by micro architectural enhancements, 512-bit wide SIMD, HBM2 and process

technology

 The results are based on the Fujitsu compiler optimized for our microarchitecture and SVE
 AI apps will be supported by SVE FP16 instructions.

Figuresfrom the slide presented in Hotchips 30 by Fujitsu
2019/01/09

10

Low-power Design & Power Management
 Leading-edge Si-technology (7nm FinFET) Low power logic design (15 GF/W @ dgemm)

 A64FX provides power management function called “Power Knob”
FL pipeline usage: FLA only, EX pipeline usage : EXA only, Frequency reduction …

User program can change “Power Knob” for power optimization“Energy monitor” facility enables chip-level power monitoring and detailed power
analysis of applications

 “Eco-mode” :

FLA only with lower “stand-by” power for ALUs

Reduce the power-consumption for memory intensive apps.

 Retention mode: power state for de-activation of CPU with keeping network

alive



Large reduction of system power-consumption at idle time

2019/01/09

11

KPIs on post-K development in FLAGSHIP 2020 project
3 KPIs (key performance indicator) were defined for post-K development
 1. Extreme Power-Efficient System

Approx. 15 GF/W (dgemm) confirmed by the prototype CPUPower consumption of 30 - 40MW (for system) is expected to be achieved

 2. Effective performance of target applications

It is expected to exceed 100 times higher than the K computerʼs performancein some applications
106 times faster in GENESIS (MD application), 153 times faster inNICAM+LETKF (climate simulation and data assimilation) were estimated 3. Easy-of-use system for wide-range of users

Shared memory system with high-bandwidth on-package memory must makeexisting OpenMP-MPI program ported easily.
No programming effort for accelerators such as GPUs is required.

 Co-design with application developers

2019/01/09

12

Post-K prototype board and rack “Fujitsu Completes Post-K Supercomputer CPU Prototype,Begins Functionality Trials”, HPCwire June 21, 2018

“Fujitsu has now completed the prototype CPU chip that willserve as the core of post-K, commencing functionality fieldtrials.”

Shelf: 48 CPUs (24 CMU)
Rack: 8 shelves = 384 CPUs (8x48)
2 CPU / CMU
2019/01/09

13

Advances from K computer
K computer

Post-Kratio# core848

Si tech. (nm)457Core perf. (GFLOPS)

1656〜

3.5

Chip(node) perf. (TFLOPS)

0.128

2.7〜

21Memory BW (GB/s)

641024B/F (Bytes/FLOP)

0.5

0.4

#node / rack963844

Rack perf. (TFLOPS)

12.3

1036.884#node/system

82,944???

System perf.(PFLOPS)

10.6???

Si TechSVE
CMG&Si Tech
HBM SVE increases core performance Silicon tech. and scalable architecture (CMG)

to increase node performance HBM enables high bandwidth

2019/01/09

14

Global Competitiveness Post-K has good power-performance as a “general-purpose” processor.
 In term with arithmetic performance and memory bandwidth, interconnect

bandwidth, the post-K system is expected to be competitive to other world-class
HPC systems.
Peak Flops

Memorybandwidth

fficiency
in Linpack

Power‐Performance
GFlops/WattInterconnect
Performance
GB/sec840> 85 %

15.040.8

3.0464490

54.4 %

4.912.5 ※3

Niagara／Xeon Skylake※１

1.536

104.5

66.7 %

4.5

6.3 ※3

Summit／GPU Volta
GV100※２

7.885565.2 %

13.8

4.2 ※3

DGX‐1 SaturnV Volta／
GPU Tesla V100※２

7.885558.8 %

15.1

6.3 ※3（doubleprecision）（STREAM triad）

TFlopsGB/secPost‐K／A64fx> 2.7

Oakforest‐PACS／
Xeon Phi KNL※１ one socket performance estimated by open information on two-socket performance of Skylake（Xeon Gold 6148 20C 2.4GHz）※２ Peak performance of one socket connected with NVLINK. Memory bandwidth by one socket GPU.
※３ Network controller is not integrated on chip. Attached Infiniband network of 100Gbps（12.5GB/sec）
For Niagara, one 100Gbps Infiniband for two sockets. For Summit, two 100Gbps Infiniband for 6 sockets.
For DGX-1 SaturnV Volta, four 100Gbps Infiniband for 8 sockets GPU. For all systems, network performance indicated for one socket..

Data taken from the report presented at CSTI, 20182019/01/0915

“PostK” performance evaluation environment
 RIKEN is constructing “PostK” performance evaluation environment for application

programmers to evaluate and estimate the performance of their applications on“PostK” and for performance turning for “postK”.

 The “PostK” performance evaluation environment is available on the servers installedin RIKEN. The environment includes the following tools and servers:

A small-scale FX100 system and “postK” performance estimation tool:
The estimation tool gives the performance estimation of multithreaded programs on “postK” from theprofile data taken on FX100.

“PostK” processor simulator based on GEM-5:“PostK” processor simulator will give a detail performance results including estimated executing time,cache-miss, the number of instruction executed in O3. The user can understand how the compiled code
for SVE is executed on “postK” processor for optimization. (Arm released GEM-5 beta0 of SVE)
FP16 SVE will be available soon.

Compilers for “PostK” processor


Fujitsu Compilers︓Fortran, C, C++. Fully-tuning for “postK” architecture.
Arm Compiler︓LLVM-based compiler to generate code forArmv8-A + SV. C,C++ by Clang, Fortran byFlang SVE emulator on Arm server, developed by Arm for fast SVE code execution.
 Arm Severs (Planned 4Q/2018)2019/01/09

16

Schedule on Development and Porting Support
NOW
CY2017

CY2018CY2019Design and Implementation

Specification

Armv8‐A + SVE

Installation,and Tuning

CY2021
Operation

Overview Detailed hardware info.
Publishing Incrementally

Optimization
Guidebook
RIKEN
Performance
Evaluation
Environment

Manufacturing

CY2020

Performance estimation tool using FX100

RIKEN Simulator

Early Access
Program• CY2018. Q2, Optimization guidebook is incrementally published• CY2020. Q2, Early access program starts• CY2021. Q1/Q2, General operation starts

Note: Fujitsu will reveal features of Post‐K CPU at Hot Chips 2018.
•2019/01/09

Takeo Yoshida, “Fujitsuʼs HPC processor for the Post-K computer,” IEEE Hot
Chips: A Symposium on High Performance Chips, San Jose, August 21, 2018.

17

Post-K CPU New Innovations: Summary
1. Ultra high bandwidth using on-package memory & matching CPU core
 Recent studies show that majority of apps are memory bound, some compute boundbut can use lower precision e.g. FP16
 Comparison w/mainstream CPU: much faster FPU, almost order magnitude fastermemory BW, and ultra high performance accordingly
 Memory controller to sustain massive on package memory (OPM) BW: difficult forcoherent memory CPU, first CPU in the world to support OPM
2. Very Green e.g. extreme power efficiency
 Power optimized design, clock gating & power knob, efficient cooling Power efficiency much better than CPUs, comparable to GPU systems
3. Arm Global Ecosystem & SVE contribution
 Annual processor production: x86 3-400mil, ARM 21bil, (2~3 bil high end)
 Rapid upbringing HPC&IDC Ecosystem (e.g. Cavium, HPE, Sandia, Bristol,…) SVE(Scalable Vector Extension) -> Arm-Fujitsu co-design, future global std.
4. High Performance on Society5.0 apps including AINext gen AI/ML requires massive speedup => high perf chips + HPC massive
scalability across chips
Post-K processor: support for AI/ML acceleration e.g. Int8/FP16+fast memory for
GPU-class convolution, fast interconnect for massive scalingTop performance in AI as well as other Society 5.0 apps
18