JLUG2018

Topics from ISC18, LAD & Lustre DeveloperSummit 2018
Shinji Sumimoto, Ph.D.
Next Generation Technical Computing Unit

FUJITSU LIMITED

Oct. 25th, 2018

0

Copyright 2018 FUJITSU LIMITEDOutline of This Talk
Lustre History
SSD Based First Layer File System for the Next GenerationSuper-computer A64FX: High Performance Arm CPU
 SSD Based First Layer File System for the Next Generation Super-computer Current Status of Lustre Based File System Development

Contribution of DL-SNAP to Lustre Community

1

Copyright 2018 FUJITSU LIMITEDLustre History
 1999: Lustre file system 研究プロジェクト開始 by Peter J. Braam 2001: Cluster File Systems社設立
 2007/9: Sun社がCluster File Systems社を買収
 2009/4： Oracle社がSun社を買収、コミュニティに不安が走る

 2010/4: Oracle社がサポートを自社ハードに制限、コミュニティ激震
 欧州EOFS、米国国研ベースOpenSFS、World WideユーザベースHPCFSの３つのコミュニティに分散 2010/9: Whamcloud社設立
 Oracle社からWhamcloud社に技術者が集結

 2010/12: Oracle社がLustre開発凍結
 2011/4: LUG2011で３つのコミュニティがOpen SFS+EOFSとして再始動

 2012/7: Intel社がWhamcloud社を買収
 2018/6: DDN社がIntelのLustre事業を買収、新生Whamcloud誕生
 2019: Luster 20 years anniversary
2

Copyright
FUJITSULIMITEDLIMITED
Copyright
20112018
FUJITSULUG2009: Lustre 10th Anniversary

3

Copyright 2018 FUJITSU LIMITEDLUG 2011: Single Community

4

Copyright 2018 FUJITSU LIMITED2018/6: DDN社がIntelのLustre事業を買収

5

Copyright 2018 FUJITSU LIMITEDISC18: Lustre BoF

6

Copyright 2018 FUJITSU LIMITEDWhamcloud&Lustreへの期待Whamcloudベンダー中立の立場でこれからもコミュニティへの貢献

Lustre真のクラスタファイルシステムとして正当に発展富士通引き続きLustreを支持しコミュニティ発展に協力

7

Copyright 2018 FUJITSU LIMITEDA64FX: High Performance Arm CPU
 From presentation slides of Hotchips 30th andCluster 2018
 Inheriting Fujitsu HPC CPU technologies withcommodity standard ISA

8Copyright 2018 FUJITSU LIMITEDA64FX Chip Overview<A64FX> Architecture Features
CMG specification• Armv8.2-A (AArch64 only)
13 cores
L2$ 8MiBMem 8GiB, 256GB/s• SVE 512-bit wide SIMD
• 48 computing cores + 4 assistant cores**All the cores are identical• HBM2• TofuD

32GiB
6D Mesh/Torus
28Gbps x 2 lanes x 10 ports• PCIe Gen3 16 lanes

Tofu
28Gbps 2 lanes 10 ports

Tofu
controller

I/O
PCIe Gen3 16 lanes

PCIecontroller

HBM2

HBM2
Netwrok
on
Chip

HBM2

 7nm FinFET• 8,786M transistors• 594 package signal pins

 Peak Performance (Efficiency)• >2.7TFLOPS (>90%@DGEMM)• Memory B/W 1024GB/s (>80%@Stream Triad)
9HBM2

A64FX

SPARC64 XIfx(Post-K)(PRIMEHPC FX100)

ISA (Base)Armv8.2-A

SPARC-V9

ISA (Extension)

SVE

HPC-ACE2

Process Node

7nm

20nm

Peak Performance>2.7TFLOPS

1.1TFLOPS

SIMD

512-bit

256-bit# of Cores

48+432+2

MemoryHBM2HMC

Memory Peak B/W

1024GB/s

240GB/s x2 (in/out)
Copyright 2018 FUJITSU LIMITEDA64FX Memory System
Extremely high bandwidth• Out-of-order Processing in cores, caches and memory controllers• Maximizing the capability of each layer’s bandwidth
CMG

12x Computing Cores + 1x Assistant Core

CorePerformance>2.7TFLOPS
L1 Cache>11.0TB/s (BF= 4)

CoreCoreCore

512-bit wide SIMD
2x FMAs
>230
GB/s

>115
GB/s

L1D 64KiB, 4way

L2 Cache>3.6TB/s (BF = 1.3)

>115
GB/s

>57
GB/s

L2 Cache 8MiB, 16way
Memory1024GB/s (BF =~0.37)

256
GB/s

HBM28GiBHBM28GiBHBM28GiB
HBM2 8GiB
10Copyright 2018 FUJITSU LIMITEDA64FX Core Features Optimizing SVE architecture for wide range of applications with Arm
including AI area by FP16 INT16/INT8 Dot Product Developing A64FX core micro-architecture to increase application
performance
A64FX(Post-K)

SPARC64 XIfx
(PRIMEHPC FX100)

SPAR64 VIIIfx(K computer)

ISA

Armv8.2-A + SVE

SPARC-V9 + HPC-ACE2

SPARC-V9 + HPC-ACE

SIMD Width

512-bit

256-bit128-bit

Four-operand FMA

9 Enhanced

99

Gather/Scatter

9 Enhanced

9Predicated Operations

9 Enhanced

99Math. Acceleration

9 Further enhanced

9 Enhanced

9Compress

9 Enhanced

9First Fault Load

9 New

FP169 New

INT16/ INT8 Dot Product

9 New

HW Barrier* / Sector Cache*

9 Further enhanced

9 Enhanced

9* Utilizing AArch64 implementation-defined system registers
11Copyright 2018 FUJITSU LIMITEDA64FX Chip Level Application Performance Boosting application performance up by micro-architectural enhancements,
512-bit wide SIMD, HBM2 and semi-conductor process technologies• > 2.5x faster in HPC/AI benchmarks than that of SPARC64 XIfx
tuned by Fujitsu compiler for A64FX micro-architecture and SVE
A64FX Kernel Benchmark Performance (Preliminary results)
8HPCThroughputNormalized toSPARC64 XIfx

(DGEMM / Stream)

9.4x

AI

Application
Kernel6Memory B/W

512-bit SIMD

Combined
Gather

L1 $ B/W

L2$ B/W

INT8
dot product422.5TF

830

3.4x

3.0x

2.8x

2.5x

GB/s

0
DGEMM

Stream Triad

Fluid dynamics

Atomosphere

Seismic wave
propagation

Convolution FP32

Convolution
Low Precision(Estimated)

Baseline: SPARC64 XIfx
12Copyright 2018 FUJITSU LIMITEDA64FX TofuD Overview
 Halved Off-chip Channels
Tofu

Tofu2K.comp

FX100

6.25

25.78

28.05

# of signal lanes per link

842Link bandwidth (GB/s)

5.0

12.5

6.8

# of TNIs per node
Injection bandwidth pernode (GB/s)

44

62050

40.8

 Power and Cost Reduction

 Increased Communication Resources

Data rate (Gbps)

 TNIs from 2 to 4
 Tofu Barrier Resources

 Reduced Communication Latency
 Simplified Multi-Lane PCS

 Increased Communication Reliability

TofuD

 Dynamic Packet Slicing: Split and Duplicate

c c c c c c c cc c c c c c c cc

CMG
HMC HMC HMC HMC

TNI2
TNI3

NOCc c c cc c c cc c c cc
CMG

To f u 2Tofu2HBM2
13c c c cc c c cc c c cc
CMG
HBM2

A64FX
PCIe
TNI0
TNI1
TNI2
TNI3
TNI4
TNI52 lanes × 10 ports

TNI1cc c c cc c c cc c c c

CMGcc c c cc c c cc c c c

Tofu Network Router

PCIe
TNI0

HBM2

CMG

4 lanes × 10 ports

CMGcc c c c c c c cc c c c c c c c

HBM2

SPARC64 XIfx

Tofu Network Router

HMC HMC HMC HMC

To f u D
TofuD

Copyright 2018 FUJITSU LIMITEDTofuD: 6D Mesh/Torus Network
 Six coordinates: (X, Y, Z)×(A, B, C)
 X, Y and Z: sizes are depends on the system size
 A, B and C: sizes are fixed to 2, 3, and 2 respectively

 Tofu stands for “torus fusion”
X×Y×Z×2×3×2

Z
BX

YA
14

C
Copyright 2018 FUJITSU LIMITEDTofuD: Packaging – CPU Memory Unit
 Two CPUs connected with C-axis
 X×Y×Z×A×B×C = 1×1×1×1×1×2

 Two or three active optical cable cages on board
 Each cable is shared by two CPUsCPU
AOC (X)

AOC

AOC (Y)AOCAOC (Z)
CPU15Copyright 2018 FUJITSU LIMITEDTofuD: Packaging – Rack Structure RackRack 8 shelves
 192 CMUs or 384 CPUs

 Shelf
 24 CMUs or 48 CPUs
 X×Y×Z×A×B×C = 1×1×4×2×3×2

 Top or bottom half of rack
 4 shelves
 X×Y×Z×A×B×C = 2×2×4×2×3×2

Shelves
16Copyright 2018 FUJITSU LIMITEDTofuD: Put Latencies & Throughput& Injection Rate
 TofuD: Evaluated by hardware emulators using the production RTL codes
 Simulation model: System-level included multiple nodes

Tofu

Tofu2
TofuD

Tofu
Tofu2TofuD

Communication settings

Latency

Descriptor on main memory

1.15 µs

Direct Descriptor

0.91 µs

Cache injection OFF

0.87 µs

Cache injection ON

0.71 µs

To/From far CMGs

0.54 µs

To/From near CMGs

0.49 µs

Put throughput
4.76 GB/s (95%)
11.46 GB/s (92%)
6.35 GB/s (93%)

Injection rate
15.0 GB/s (77%)
45.8 GB/s (92%)
38.1 GB/s (93%)

17

Copyright 2018 FUJITSU LIMITEDNext Generation File System Design
Next Generation File System Structure and Design

Next-Gen 1st Layer File System Overview

18Copyright 2018 FUJITSU LIMITEDK computer: Pre-Staging-In/Post-Staging-Out Method
 Pros: Stable Application Performance for Jobs

 Cons: Requiring three times amount of storage which a job needs
 Pre-defining file name of stage-in/out processing lacks of usability
 Data-intensive application affects system usage to down because of waiting prestaging-in/out processing
Computing Node
Application
LinuxStage-in/out

Loopback

Job Control
NodeGlobal File System
19Local File Systemusing FEFS

LoginNodeUsers
Copyright 2018 FUJITSU LIMITEDNext-Gen File System Requirement and Issues
Requirements
10 times higher access performance
100 times larger file system capacity

Lower power and footprintIssuesHow to realize 10 times faster and 100 times larger fileaccess at a time?

20Copyright 2018 FUJITSU LIMITEDNext-Gen. File System Design
 K computer File System Design
 How should we realize High Speed and Redundancy together?
 Introduced Integrated Two Layered File System.

 Next-Gen. File System/Storage Design
 Another trade off targets: Power, Capacity, Footprint•Difficult to realize single Exabyte and 10TB/s class file system
in limited power consumption and footprint.

 Additional Third layer Storage for Capacity is needed:
ComputeNodes
Nodes
ComputeComputeNodes
Compute
Nodes

Thousands of Users
LoginServerHigh Speed for Application
LustreBased
Based
LustreLustreBased
Ext[34]Based
Based
Ext[34]Ext[34]
Based
Application
ObjectSpecific
Based
ObjectBased
ObjectBased
Existing FSSpecific
Application
Specific
Application
Application
Specific
Object Based

SharedUsability

Lustre BasedTransparent Data Access

Job Scheduler
21

The Next Integrated Layered File System Architecture
for Post-peta scale System (Feasibility Study 2012-2013)

OtherOrganization

OtherSystems

High Capacity & Redundancy& Interoperability
HSM, Other Shared FS,Grid or Cloud Based/dataCopyright 2018 FUJITSU LIMITEDNext Gen. File System Design
 Introducing three level hierarchical storage.
 1st level storage: Accelerating application file I/O performance (Local File System) 2nd level storage: Sharing data using Lustre based file system (Global File System) 3rd level storage: Archive Storage (Archive System) Accessing 1st level storage as file cache of global file system and local storage File cache on computing node is also used as well as 1st level storage
Computing NodeApplication
LinuxSSD Based
1 Level StoragestJobControlNode

Global File System
Lustre based file system on 2nd Level Storage

Login
NodeUsers

Archive Storage for 3rd Level Storage
22Copyright 2018 FUJITSU LIMITEDApplication’s Access Pattern and SSD Cache Effects
 Comparison of Effective Pattern for SSD based storage
Distributed
Files

Single Shared
Files (1)Single Shared
Files (2)Rereading Case：◎
Non Rereading ：×

I/O
Master

Processes

File
Reading

Processes

File
Writing

File Read：
Effects

Rereading Case：◎
Non Rereading ：×

Rereading Case：◎
Non Rereading ：×

File Write：
Effects

Rewriting Case：◎
Non Rewriting ：○Rewriting Case：◎
Non Rewriting ：○23Rereading Case：◎
Non Rereading ：×
Rewriting Case：◎
Non Rewriting ：○Copyright 2018 FUJITSU LIMITEDData Sharing in a Job on SSD
 Write-Read in a process and among processes are effective to use SSD
 For Persistent Files: File cache of global file system should be shared among processes
 For Temporary Files: Two types of temporary file systems are effective to use SSD
 Temporary Local System (in a process) Temporary Shared File System (among processes)
In a process
(1)

write

(2)

write

readProcess

Process(1) write(2) writeFile(1) write(1) read
Filewrite(2) read

File

Process

read

Among processes(2) write

Process(1) read(2) read

File24

Process

readProcess

Process(1) write(2) write(1) write

FileFile

Process

Process(2) read

Process(1) read(2) write (1) read

FileFile

Process

Process(2) read

Copyright 2018 FUJITSU LIMITEDNext Gen. Layered File System Requirements
 Application views:
 Local File System: Application Oriented File Accesses(Higher Meta&Data I/O)
 Global File System: Transparent File Access Archive System: In-direct Access or Transparent File Access(HSM) Transparent File Access to the Global File System
 Local File System Capacity is not enough as much as locating whole data of
Global File System
 File Cache on node memory and Local File System enables to accelerateapplication performance
MetaPerf.

DataBWs

Capacity

Scalability

Data Sharing
in a Job

Data Sharing
among Jobs

Local File System

◎

◎×◎

◎

×Global File System

○○○○×

◎

Archive System××◎×××25

Copyright 2018 FUJITSU LIMITEDNext-Gen 1st Layer File System Overview
 Goal: Maximizing application file I/O performance Features:
 Easy access to User Data: File Cache of Global File System
 Higher Data Access Performance: Temporary Local FS (in a process) Higher Data Sharing Performance: Temporary Shared FS (among processes)

 Now developing LLIO(Lightweight Layered IO-Accelerator) Prototype
Job Ａ
I/O w/Assistant
Cores

Node

Job B
NodeApp.

Node

NodeApp.

App.

Node

App.

ScalableApp.

ComputeCores
1st Level

2nd Level

file1

Local

file2

file3

SSD

CacheCache

file3

file4
26

TemporarySharedCacheLocal File
Systems

Global FileSystem(Lustre Based)
Copyright 2018 FUJITSU LIMITEDLLIO Prototype Implementation
 Two types of Computing Nodes
 Burst Buffer Computing Node(BBCN)•Burst Buffer System Function with SSD Device

 Computing Node(CN)•Burst Buffer Clients: File Access Request to BBCN as burst buffer server
Background data flushing
On demand data staging

…
CN

CN

CN

CN

BBCN

arm／x86

SSD

…
CN

CN

CN

CN

BBCN

IO/meta Requests

Computing Node Cluster
27

interconnect

IO/meta Requests

SSD

2nd Layer File System
Copyright 2018 FUJITSU LIMITEDFile Access Sequences using LLIO (Cache Mode)CN

2nd Layer
File SystemBBCN

Meta Reqs: Pass through to 2nd Layermeta server

open(file)

I/O serverwrite(fd, buf, sz)

App/gfs

LLIO

LLIO

2nd Layer FSClient

2nd Layer
FS Serverwrite(fd, buf, sz)
flush

ＬFSSSD
28

Background Flushing

HDD
Copyright 2018 FUJITSU LIMITEDLLIO Prototype I/O Performance
Read Performance

I/O Bandwidth

I/O Bandwidth

Write Performance

Device# of IOR Streams

Device# of IOR Streams

Evaluated on IA servers using Intel P3608

 Higher I/O performance than those of NFS, Lustre

 Utilizing maximum physical I/O device performance by LLIO29Copyright 2018 FUJITSU LIMITEDContribution of DL-SNAP to Lustre Community Our Strategy for contributing DL-SNAP: We are ready to contribute our current DL-SNAP code for Lustre 2.6
 We have made the LU-ticket for the DL-SNAP (LU-11512, Oct. 2018)30

Copyright 2018 FUJITSU LIMITEDWhat is DL-SNAP?
 DL-SNAP is designed for user and directory level file backups.

 Users can create a snapshot of a directory using lfs command withsnapshot option and create option like a directory copy.
 The user creates multiple snapshot of the directory and manage
the snapshots including merge of the snapshots.
 DL-SNAP also supports quota to limit storage usage of users.

31

Copyright 2018 FUJITSU LIMITEDDL-SNAP Use-case 1
 Avoiding file deletion or corruption by file operation

Needs data
three daybefore#!/bin/bash

User A
File1
File2(Mod)
File3

GEN=7
NEW_SNAP=`/bin/date '+%Y%m%d'`
OLD_SNAP=`/bin/date -d "${GEN} day ago"+%Y%m%d`

Creating
Snaphot

File 3
DeletedMaking snapshot day by day using cronSnapshot Data-1 Day-2 Day-3 Day
File1
File1
File1
File2(Mod)
File2
File2
File3(Del)
File3…

lfs snapshot --create /home/usrA@$NEW_SNAP
lfs snapshot --delete /home/usrA@$OLD_SNAPRestoring
File3

User A
File1
File2(Mod)File3-1 Day
File1
File2(Mod)Restoringby File Copy

32

Snapshot Data
-2 DayFile1
File2
File3(Del)-3 DayFile1
File2
File3…

Copyright 2018 FUJITSU LIMITEDDL-SNAP Use-case 2
 Maintaining large database with partially different data Updating database by an application using DL-SNAPApplication
Daily Incoming Data

Input 1
Input 2
Input 3
Input 4：Analyzing differenceof databases

User A

Outputs
Large Database

Snapshot
Dec.1
Output33

Snapshots
Dec. 3
OutputDec. 10Output…

Copyright 2018 FUJITSU LIMITED34

Copyright 2018 FUJITSU LIMITED