Hybrid Accelerator

NEC SX-Aurora TSUBASA
2022年1月20日（木）日本電気株式会社 政岡 靖久 <ymsk-bx@nec.com>© NEC Corporation 2022

AgendaSX-Aurora TSUBASAとは
使い方（VE Offloading）
使い方（Transparent Execution）
Roadmap
For Your Future

SX-Aurora TSUBASAとは
使い方（VE Offloading）
使い方（Transparent Execution）
Roadmap
For Your Future

35

Over
years experiencefor High Sustained
Performance4© NEC Corporation 2022

Features of SX-Aurora TSUBASAHigh Memory Bandwidth（Traditional Super
Computer）

Tower
Type

Vector Engine

Ease of Use
Rack
Type

Downsize!

Downsizing of super computer realized by NEC’s
Technology.
5© NEC Corporation 2022

Vector technology makes it possible to process
multiple and huge data at a time with highmemory bandwidth.

No specialized knowledge is required, AP can be
executed only after compiled.
Use C/C++/Fortran/Python to program.

Flexibility
Customer can choose a system which meets theirneeds. From server type to card specification are
all optional, NEC help customer to maximize thecost performance, to fit all market requirement.

High Memory Bandwidth

Vector Engine◆ Vector technology is packed into a PCI card.
◼ Vector processor (8/10 cores)◼ 1.53TB/s memory bandwidth◼ 48GB memory◼ 2.45/3.07TF performance (double precision)
4.91/6.14TF performance (single precision)◼ A variety of execution modes◼Standard programming with
C/C++/Fortran/Python◼ Power consumption < 300 W
6© NEC Corporation 2022

Flexibility

Lineup of SX-Aurora TSUBASA

Vector Engine supports wide range from desk-side to large-scale Data Centers.
Selling Vector Engine card was started from November, 2020.
Data Center Model
Data Center Model
Huge processing in Data Centers(Water-Cooling)
8VE

Rackmount Model

Rackmount Model

Simulation of manufacturing industry, etc.
Use of AI / big data
8VE

4VE

Edge Model

Edge Model
1VE

Simulation of mid-sized manufacturingindustry, etc. Laboratory desk-side

Vector Engine
Application acceleration
Embedded use
Cloud service engine
7© NEC Corporation 2022

Vector Engine
Application Set Model,
Embedded System,
Cloud service

Vector Engine

Ease of Use

Architecture of SX-Aurora TSUBASA◼ SX-Aurora TSUBASA = VH + VE
◼ Linux + standard language (C/C++/Fortran/Python)◼ Enjoy high performance with easy programming

SX-Aurora TSUBASA
Architecture

Hardware◼ VH(Standard x86 server) + Vector Engine

as-is Linux

Application

Softwarex86 server(VH)

VectorEngine(VE)Interconnect

PCIe◼ as-is Linux◼ C/C++/Fortran/Python◼ Automatic vectorization compiler◼ InfiniBand for MPI✓ VE-VE direct communication support
Easy
programming(standard language)8© NEC Corporation 2022

Automatic
vectorizationcompiler

Enjoy high
Performance!

SX-Aurora TSUBASAとは
使い方（VE Offloading）
使い方（Transparent Execution）
Roadmap
For Your Future

SX-Aurora TSUBASA programming model
VE Offloading
VE Offloading◆ VEをアクセラレータ方式で使うためのフレームワーク

x86
Application

VE
Library

x86
node
10

VectorEngine© NEC Corporation 2022

◆ プログラムの本体はx86（VH）側にあり、その一部をVE側で必要に応じて実行できる。

How to use VE Offloading?
VEライブラリはC/C++、Fortranで記述可能。
VE Offloading◆ x86 Application（プログラム本体）◼ C/C++用のAPIをサポート

x86
Application◆ VE Library（プログラムの一部）
VE
Library

x86
node
11

VectorEngine© NEC Corporation 2022

◼ vi libve.cuint64_t ve_data[64*1024*1024/sizeof(uint64_t)];
int ve_sub(int arg1, ...){for (uint64_t i = 0; i < ...) {ve_data[i] += ...;}
return(0);}◼ /opt/nec/ve/bin/ncc libve.c -shared –fpic -o libve.so（つづき）How to use VE Offloading?
APIs for x86 (VH) application to offload a part of program to VE.
1.Create process on VE

libveo API for C/C++

veo_proc_handle *veo_proc_create( int venode)

libve.so
ve_sub()

User
app

VEOS

threadVEO

2.Load shared library on VE
uint64_t veo_load_shared_library( veo_proc_handle *proc, const char *libname)

3.Create thread on VE
veo_thr_ctx *veo_context_open( veo_proc_handle *proc)

4.Create arguments of func()
veo_args *veo_args_alloc(void)

VEO(VE part)Linux

int veo_args_set_<type>(veo_args *ca, int argnum, <type> val)

5.Call func() asynchronously by nameuint64_t veo_call_async_by_name( veo_thr_ctx *ctx, uint64_t libhdl,
const char *symname, veo_args *args)

x86
node
12

© NEC Corporation 2022

VectorEngine6.Wait result of func()
int veo_call_wait_result( veo_thr_ctx *ctx, uint64_t reqid, uint64_t *retp)

NLCPyPythonスクリプトからNLCの一部機能をご利用いただけます
NumPyスクリプトはモジュール名を変更するだけでVEの演算能力をご活用いただけます◆NLCPyの特長
import nlcpy as np◼NumPyからの移植が容易•

NumPyのAPIのサブセットを提供◼VE向けに最適化•

VE向けに最適化されたライブラリを利用
BLAS、LAPACK、 ASL、SCA等

a = np.random.rand(1000, 1000)b = np.fft.fft(a)c = a + b
d = np.linalg.solve(a, c)
VH-VE間を自動でデータ転送

•

•
13

ベクトル化された演算機能を多数提供
四則演算、数学関数、乱数、ソート、FFT等
VE上で実行する関数を非同期に呼び出し、
VH-VE間オフロードのオーバヘッドを削減© NEC Corporation 2022

VE上で演算ScriptLibraryx86
ノード

VectorEngineNEC Numeric Library Collection (NLC)
NLCは、広範な分野の数値シミュレーションプログラムの作成を強力に支援する数学ライブラリのコレクションであり、Vector Engineに対応しています
ASL統合インタフェース

BLAS / CBLAS

フーリエ変換、乱数

ベクトル、⾏列の基本演算

FFTWインタフェース

LAPACKFFTW (version 3.x)のAPI でASL 統合インタフェースのフーリエ変換を利⽤するためのインタフェースライブラリ

連⽴⼀次⽅程式、固有値解析

ASL

ScaLAPACK
連⽴⼀次⽅程式、固有値解析(分散メモリ並列⽤)

基本⾏列演算、連⽴1 次⽅程式、固有値・固有ベクトル、
フーリエ変換、スプライン関数、近似・補間、数値微分、
数値積分、⽅程式の根、極値問題・最適化、微分⽅程式とその応⽤、特殊関数、ソート・順位付け、確率分布、基礎
統計量、推定と検定、分散分析・実験計画、ノンパラメト
リック検定、多変量解析、時系列分析、近似・回帰分析

14

SBLAS
スパースBLASSCA

HeteroSolver

ステンシル計算

線形連⽴⽅程式のスパース直接法ソルバ© NEC Corporation 2022

Stencil calculation Code Accelerator Interface of NLCPy◆ SCA Interface◼ Boosts performance of a wide variety of stencil calculations.

◆ Example◼ User can easily define arbitrary stencil shapes by specifying relative locations.
Laplace Equation
𝜕𝜕
𝐴+ 2𝐴 = 02𝜕𝑥𝜕𝑦import nlcpy as vp
a = vp.random.rand(100).reshape(10,10)b = vp.zeros_like(a)din, dout = vp.sca.create_descriptor((a, b))
desc_in = 0.25 * ( din[0,-1] + din[-1,0] + din[1,0] + din[0,1] )desc_out = dout[0,0]kern = vp.sca.create_kernel(desc_in, desc_o=desc_out)
for i in range(maxitr):a[...] = kern.execute()

discretization

finite-difference
1𝑏𝑖,𝑗 = 𝑎𝑖,𝑗−1 + 𝑎𝑖−1,𝑗 + 𝑎𝑖+1,𝑗 + 𝑎𝑖,𝑗+1
4

j+2
0.25

j+1
0.25 0.0

j-1
j-2
i-2 i-1

15© NEC Corporation 2022

0.25

j
0.25

i

i+1 i+2

Benchmark Result (2-D XY-axial stencils)◆ NLCPy on Vector Engine shows the highest performance.

6x6ya

16

© NEC Corporation 2022

Benchmark Result (3-D XYZ-axial stencils)◆ NLCPy on Vector Engine shows the highest performance.

6x6y6za17© NEC Corporation 2022

Benchmark Conditions◆ Stencil Shapes and Time Steps◼ 2-D: XY-axial, size 6.
◼ 3-D: XYZ-axial, size 6.
◼ For 100 time steps.

◆ Target Libraries◼ Numba:• A100 PCI-E 40GB
• Xeon Gold 6126 x2 (Skylake 2.60GHz, 48 cores)◼ pystencils:• A100 PCI-E 40GB
• Xeon Gold 6126 x2 (Skylake 2.60GHz, 48 cores)◼ CuPy on A100 PCI-E 40GB
◼ NLCPy on VE Type 20B (8 cores)

6x6ya◆ Data Size
◼ 2-D: (64, 64)(128, 128)
(256, 256)
(512, 512)
(1024, 1024)◼ 3-D: (64, 64, 64) (128, 128, 128) (256, 256, 256) (512, 512, 512) (1024, 1024, 1024)
For each case, single precision was used.
18© NEC Corporation 2022

6x6y6zaNLCPy : Just-In-Time Compilation for VE
NLCPy provides Just-In-Time(JIT) compilation functionality.
By using this, you can customize your C/C++/Fortran kernel directly on your Python program.

◆ Example of JIT Compilation
import nlcpy
c_source = r"""void ve_add(double *px, double *py, double *pz, int n) {for (int i = 0; i < n; i++) pz[i] = px[i] + py[i];}"""Define C source

lib = nlcpy.jit.CustomVELibrary(code=c_source)

Build & Load

ve_add = lib.get_function('ve_add',
args_type=('uint64_t', 'uint64_t', 'uint64_t', 'int32_t'),
ret_type='void',)

Find a symbol of a function

x = nlcpy.random.rand(10)y = nlcpy.random.rand(10)z = nlcpy.empty(10)ve_add(x.ve_adr, y.ve_adr, z.ve_adr, z.size)
19© NEC Corporation 2022

Execution

SX-Aurora TSUBASAとは
使い方（VE Offloading）
使い方（Transparent Execution）
Roadmap
For Your Future

SX-Aurora TSUBASA programming modelTransparent Execution
Transparent Execution◆ プログラムの実体はVE側に全てある◆ 特別なプログラミングは必要なし

VEOS

VE
Application◼ with glibc, C/C++, Fortran◆ アクセラレータであることを意識せずに使用可能
VE Driver◼ Hybrid Accelerator

Linux
x86
node
21

• OS機能が必要な時はx86（VH）側に処理が自動的にオフロードされる

VectorEngine© NEC Corporation 2022

– システムコール契機• x86（VH）側からのOS制御も可能– 実行開始/停止/終了、シグナル送信、 gdbによるデバッグ、コンテキストスイッチ、、他

Transparent Execution./a.out

VH(x86)

executeVE

resultvi sample.c
Program on VH（x86)$ gcc sample.c$ ./a.out
Hello World !
$Compile using ncc$ ncc sample.c
ncc: opt(1135): sample.c, line 10: Outer loop conditionallyCompiler message
executes inner loop.
ncc: vec( 101): sample.c, line 13: Vectorized loop.
Execute on VE$ ./a.out
Result shown on VH
Hello World !
$22© NEC Corporation 2022

Trusted and Chose by World Famous HPC Centers
Weather
ClimateTohoku university : Academic

23© NEC Corporation 2022

: Weather /ClimateNIFS : Fusion Science
National Institute for Fusion Science

JAMSTEC : Earth Science

Osaka university：AcademicⒸJAMSTEC

Cybermedia Center

Performance of large-scale computer system
JAMSTEC Earth Simulator is ranked in TOP10 in the latest HPCG ranking.
High Byte/Flops and high performance single core => High execution efficiency
RankⒸJAMSTECJAMSTEC

NIFS
24

National Institute for Fusion© NEC Corporation 2022
Science

Cores

HPCG
[TFlop/s]

Rpeak[TFlop/s]

Execution
efficiency

7,630,848
2,414,592
706,304

16,004.50
2,925.75
1,905.44

537,212.00
200,794.88
89,794.48

2.98%
1.46%
2.12%

1,572,480

1,795.67

125,712.00

1.43%

Selene555,520
1,622.51
JUWELS Booster
Atos
449,280
1,275.36
ModuleDammam-7
HPE
672,520881.40HPC5Dell EMC
669,760
860.32
Wisteria/BDEC-01
Fujitsu
368,640
817.58Earth Simulator SX-AuroraNEC
43,776747.80TSUBASA
TOKI-SORA
Fujitsu
276,480
614.22
Trinity
Cray/HPE
979,072546.12
Plasma Simulator
NEC
34,560529.16
Marconi-100 https://www.top500.org/lists/hpcg/2021/06/
IBM
347,776498.43

79,215.00

2.05%

70,980.00

1.80%

55,423.56
51,720.76
25,952.26

1.59%
1.66%
3.15%

13,447.99

5.56%

19,464.20
41,461.15
10,510.66
29,354.00

3.16%
1.32%
5.03%
1.70%

System

HPCG

HPL

1
2
3

1
2
5

FugakuSummitPerlmutter

4

3

Sierra

5

6

6

8

7
8
9

11
9
13

10

39

11
12
13
14

25
16
54
14

VendorFujitsu
IBM
HPE
IBM /
NVIDIA /Mellanox
NvidiaSX-Aurora TSUBASA programming model
The 3 models are available in Aurora architecture.
VE Offloading

Transparent Execution

VH Callx86
Application

VE
Application

VE
Library

x86
node
25

VectorEngine© NEC Corporation 2022

x86
node

VE
Applicationx86
Library

VectorEnginex86
node

VectorEngineVH Call
Simple APIs for VE application to call library function on VH.
libvhcall API

1.Load shared library on VH
vhcall_handle vhcall_install(const character *filename)libuser.so
0xaa func()

2.Find address of func() on VH
int64_t vhcall_find(vhcall_handle libhdl, const char *symname)

VEOS
User
application

3 Create arguments of func()
vhcall_args *vhcall_args_alloc()
int vhcall_args_set_<type>(vhcall_args *args, int argnum, <type> val)

4.Call func() synchronously

Linux OSint vhcall_invoke_with_args(int64_t symid, vhcall_args *args, uint64_t *retptr)

x86
node
26

© NEC Corporation 2022

VectorEngineSX-Aurora TSUBASAとは
使い方（VE Offloading）
使い方（Transparent Execution）
Roadmap
For Your FutureVector Engine 3.0
VE30
10C/3.07TF
1.53TB/s
8C/2.45TF
1.35TB/s

8C/2.45TF
1.22TB/s

VE10
E

VE1
0

28

© NEC Corporation 2022

VE20

2+TB/s memory bandwidth■Targeting the largest memory bandwidth■Inheriting and improving VE/VH architecture■Higher Flops per processor■Improved memory subsystem including cache■Accelerating short vector, and scalar operations■Maintaining high power efficiency■Virtual machine support

Roadmap

Memory bandwidth / VE

VE50

VE40
VE30

2+TB/s memory bandwidth

VE20
VE20

VE10E
VE10
2019

29

2++TB/s memory bandwidth© NEC Corporation 2022

8C/2.45TF
1.35TB/s memory bandwidth

10C/3.07TF
1.5TB/s memory bandwidth

8C/2.45TF
1.22TB/s memory bandwidth

2020

2021

2022

2023

2024

Find more information on our website

Aurora Web Forum

SX-Aurora TSUBASA Website

http://www.hpc.nec

http://www.nec.com/en/global/solutions/hpc/sx/index.ht
ml➢ Latest updates➢ Manual, documents➢ Bulletin board➢ Hardware and software overview➢ Supported applications

info@hpc.jp.nec.com
30© NEC Corporation 2022

Tohoku university : Academic

JAMSTEC : Earth Science

Osaka university：AcademicⒸJAMSTEC

Cybermedia Center

SX-Aurora TSUBASA
トライアルプログラム概要◆ 目的◼ ご購入前評価を目的としたマシン環境の提供

評価環境（一例）

SX-Aurora TSUBASA A300-2◆ ご利用条件◼ 対象：SX-Aurora TSUBASAをご利用予定のお客様• 法人、大学、高等専門学校、他に所属されているお客様◼ 期間/回数：30日/１回限り

CPU：Intel Xeon Gold 6126 (12core, 2.6GHz) x1
MEM：96GiB
Disk：1TB HDD
VE：TypeC (8core, 1.4GHz, 24GB MEM) x1◼ 費用： 無料（オンプレ貸出時の返送費用はお客様ご負担となります)◼ 形態： クラウド環境、ご要望に応じてオンプレ貸出◼ サポート：専用Webサイトでのマニュアル、チュートリアル情報、FAQのご提供
専用フォームからのお問い合わせ対応◼ その他• 貸出期間中あるいは期間終了後に提供された質問、意見や提案、あるいは移植したOSSコードについて、他のお客様への公開を依頼させて頂く場合があります。

• お客様からのフィードバックは、専用Webサイトなどで公開させて頂きます。
• ご要望多数の場合は弊社で審査の上、貸出可否を判断させていただく可能性がございますのでご了承ください。

info@hpc.jp.nec.com
31© NEC Corporation 2022

SX-Aurora TSUBASAとは
使い方（VE Offloading）
使い方（Transparent Execution）
Roadmap
For Your FutureBackground of multi-architecture system -towards Heterogeneous ComputingArchitecture is selected according to characteristics of each of applications.
One of trends in HPC system is hybrid, composed of a variety types ofprocessors.
Scientificcalculation•••

Weather forecast•
Aerodynamic analysis •Collision analysis•

x86

General34© NEC Corporation 2022

Statisticalprocessing

Recommendation
Demand prediction
Fraud detection

VectorEngineImage
recognition•••

Self-driving
Checking goods
Cancer diagnosis

GPGPU

Real-timetransaction•••

Financial transaction
Face recognition
Industrial robot

ASIC/
FPGA

Combinatorial
optimization•••

Financial portfolio
Shift schedule
Delivery planning

Quantum
AnnealingSpecialAurora is ready for your future◆ Aurora HPCG performance efficiency is better than the other machines.
◼ The Fraction of Peak is over 5%. This will be one of the advantages of Aurora system in yourfuture, especially when you want to build your LARGE computing system.

◆ Aurora can easily collaborate with the other machines, such as GPGPU, etc.
◼ Through interconnect, right now Infiniband• Hybrid VE-x86/GPGPU MPI is available right now.

◼ In VH (Linux/x86)• Not only VEs, but GPGPU, etc. can be installed in the same VH.
– Hybrid VE-x86 MPI through PCIe is available right now.
– Hybrid VE-GPGPU MPI through PCIe will be available in 2022.
– NLCPy/Aurora with GPGPU will be available in 2022.
35© NEC Corporation 2022

OrchesPy
OrchesPy is a NumPy-like library for VE-GPU heterogeneous systems.
◆ Feature◼ User will be able to choose execution devices just by specifying Python decorators.

◆ Schedule◼ April 2022 (Preview)36© NEC Corporation 2022

Tohoku university : Academic

JAMSTEC : Earth Science

Osaka university：AcademicⒸJAMSTEC

Cybermedia Center

SX-Aurora TSUBASA
トライアルプログラム概要◆ 目的◼ ご購入前評価を目的としたマシン環境の提供

評価環境（一例）

SX-Aurora TSUBASA A300-2◆ ご利用条件◼ 対象：SX-Aurora TSUBASAをご利用予定のお客様• 法人、大学、高等専門学校、他に所属されているお客様◼ 期間/回数：30日/１回限り

CPU：Intel Xeon Gold 6126 (12core, 2.6GHz) x1
MEM：96GiB
Disk：1TB HDD
VE：TypeC (8core, 1.4GHz, 24GB MEM) x1◼ 費用： 無料（オンプレ貸出時の返送費用はお客様ご負担となります)◼ 形態： クラウド環境、ご要望に応じてオンプレ貸出◼ サポート：専用Webサイトでのマニュアル、チュートリアル情報、FAQのご提供
専用フォームからのお問い合わせ対応◼ その他• 貸出期間中あるいは期間終了後に提供された質問、意見や提案、あるいは移植したOSSコードについて、他のお客様への公開を依頼させて頂く場合があります。

• お客様からのフィードバックは、専用Webサイトなどで公開させて頂きます。
• ご要望多数の場合は弊社で審査の上、貸出可否を判断させていただく可能性がございますのでご了承ください。

info@hpc.jp.nec.com
37© NEC Corporation 2022

NEC LLVM-IR Vectorizer
https://www.hpc.nec/documents/llvm-vec/en/index.html

Provide binary plugin-module and runtime-library to add automatic
vectorization feature for VE into your compilers
Source fileCompiler
Front-end(clang / flang / etc.)
LLVM(Modules to create LLVM-IR)
LLVM-IR

LLVM-IR Vectorizer

VE Vectorizer

VE code generator

Assembler source fileincluding vectorized loops
38© NEC Corporation 2022

LLVM-IR Vectorizer⚫ 1st release based on LLVM 11.0 was May, 2021
◼ Bug fix version was released at Aug, 2021 & Oct, 2021

⚫ Includes vectorizer and code generator for VE.
⚫ Inputs LLVM-IR from memory or an IR-file and outputs anassembler source code for VE.

⚫ Applies automatic vectorization to LLVM-IR.
⚫ Runtime library including vector mathematical functions(sin,cos etc.) and vector matrix-multiply functions is bundled.
⚫ Source patch for clang to enable LLVM-IR Vectorizer is
included. It helps to change your compiler’s front-end

You can build your own compiler having vectorization
feature for VE !!

VEOS and NLCPy◆ VEOS概要設計◼ https://www.hpc.nec/documents/veos/jp/VEOS_high_level_design.pdf◆ VEOS◼ https://github.com/veos-sxarr-NEC◆ NLCPy◼ https://www.hpc.nec/documents/nlcpy/ja/index.html

39© NEC Corporation 2022