インテルHPC最新技術アップデート
Technical Solution Specialist
Fumiyasu Ishibashi

パラダイムチェンジを牽引 - HPC & AI融合

気象パターン検出

創薬

高エネルギー物理学

HPC のワークフローに組み込まれる AI

HPC のシミュレーションを高速化する AI

HPC のシミュレーション
にとって代わる AI

第 2 世代インテル® Xeon®
スケーラブル・プロセッサー
インテル® Xeon®

Platinum 9200
プロセッサーインテル® Xeon®

インテル® Xeon®

Platinum 8200

Gold 6200

プロセッサープロセッサーインテル® Xeon®

インテル® Xeon®

インテル® Xeon®

Gold 5200

Silver 4200

Bronze 3200

プロセッサープロセッサープロセッサー

新たなクラスの高度なパフォーマンス

価値

トップクラスのワークロード・パフォーマンス

画期的なメモリー・
イノベーション

人工知能
ハードウェア支援型
アクセラレーション
セキュリティー

俊敏性と使用率の向上

http://www.intel.co.jp/xeonscalable/インテル® Xeon® プロセッサーさらなる発展

唯一、コンバージェンス向けに最適化されたデータセンター CPU
インテル® アドバンスト・ベクトル・エクステンション 512
インテル® ディープラーニング・ブースト (インテル® DL ブースト)
インテル® Optane™ DC パーシステント・メモリー

2019年

2020年

2021年

Cooper lake

Cascade lake

14NM
次世代インテル® DL ブースト (BFLOAT16)

14NM
AI の新たな加速化 (VNNI)
メモリーストレージの新しい階層

ice lake
10NM
現在サンプルを出荷開始

業界最先端のパフォーマンス

Sapphire rapids
次世代テクノロジー

Memory and Storage Hierarchy Gaps
The Capacity Gap

MEMORYCAPACITY GAPSTORAGEIntel® 3D NAND

Big and Affordable memory
Byte addressable
High performance storage
High reliability and security
Two operational modes

More to be Gained by being on memory Bus
Hardware latency

100

Software latency

75

50

25

10µs

~100ns to ~340ns

0

See Aopendix J
Performance results are based on testing as of July 24, 2018 set forth in the configurations and may not reflect all publicly available security updates. See configuration disclosure for details. No product can be absolutely secure.
For more complete information about performance and benchmark results, visit www.intel.com/benchmarks.

Lower is better

PerformanceIntel® SSD DC
P4610

up
to

Measurement notes:- For SSDs Random 4KB
Accesses over entire SSD,
Read latency measured
per 4kB access- For Intel® Optane™ DC
Persistent Memory Random
256B accesses Over entire
module 256B random
accesses w/ read latency
measured per 64B access

LOWER
latency

Read idle latency
Ranges from 180ns to 340ns(vs. DRAM ~70ns)

Higher is better

See Appendix K
Performance results are based on testing as of February 22, 2019 set forth in the Configurations and may not reflect all publicly available security updates. See configuration disclosure for details. No product can be absolutely secure.
For more complete information about performance and benchmark results, visit www.intel.com/benchmarks.

Performance Details•

Intel® Optane™ DC persistent memoryis programmable for different power
limits for power/performance
optimization• 12W – 18W, in 0.25 watt granularity - forexample: 12.25W, 14.75W, 18W• Higher power settings give best performance•

Performance varies based on traffic
pattern• Contiguous 4 cacheline (256B) granularity vs.
single random cacheline (64B) granularity• Read vs. writes

Two operational modes
Load/Store

Load/Store

L1I

L1I

L1D

L2 CacheL2 Cache

L3 CACHE
DDR4
Cache
Hit

L1D

DDR4
Cache
Miss

HW Managed

DATA PLACEMENT

VolatilePERSISTENCYSW Managed
Persistent

L3 CACHE

Persistency: Early Engagement with Industry
PERSISTENT MEMORY PROGRAMING MODELPERSISTENT MEMORY DESIGN KIT (PMDK)

developed through SNIAManagement library

Application
Standard
Raw Device
AccessApplication
Standard
File API

Application
Load/
Store

“DAX”

Generic nvdimm driverMmu
mappings

KERNEL SPACE

Pmem-aware
File system

USER SPACEManagement ui

available on http://pmem.io

A Strong Memory & Storage Future

INTEL®
3D NAND SSD

APACHE PASSBARLOW PASS

3RD GEN DCPERSISTENT MEMORY

4TH GEN DCPERSISTENT MEMORYIntel® SSD DC P4800X(COLDSTREAM)

ALDER STREAM

NEXT GENERATIONNEXT GENERATIONINTEL® SSD DC
P46XX/P45XX

CLIFFDALE-R/ARBORDALE+ (96-L, 144-L)NEXT GENERATIONNEXT GENERATION

For each processor above, Intel® Optane™ DC Persistent Memory will be supported on select SKUsND
BARLOW PASS OVERVIEW – 2 GENERATIONBARLOW PASSIntel® Optane™ DC persistent memory

Barlow Pass will be productized with aBlue colored Full DIMM Heat spreader for
better Identification in the data center.

15w

3200 MT/S

TDP

DDR4

Up to

CEDAR ISLAND

WHITLEY

4 SOCKET PLATFORM

2 SOCKET PLATFORM

8CH 3200 MT/S 4TB
Memory

IMC
IMC

DDR4

Ice LAKE

PROCESSORBPS Per Socket

6CH 2933 MT/S

DDR4

3TB

1

MemoryIMC
IMC

LegendBPS

15% BW

Improvement vs AEP

1. For Cedar Island, 2933 2DPC & 3200 1DPC is POR. Stretch target is 3200 2DPC

IMC

DDR4

COOPER LAKEPROCESSORBPS Per Socket

IMC

INTEL OPTANE DC DESIGN IN ADVOCACY

storage

Cloud & VMs

database

AI / analytics

hpc

COMMS

Write Buffers

VM and Container
Density

DB Memory Buffer Pool

Real Time
Analytics

Larger Memory Nodes

Content Delivery
Network (CDN)

Vaulting
PMoF

Extended VM
Memory Capacity

Caching Layers

Application Density

Meta-Data

Large Memory Java

Hi-Perf Direct Attached
Storage

Compute-side Storage
and Cache

In-Memory DB
Mixed Logging andmemoryUser-Density
Fast Restart

Machine Learning
Analytics
In-Memory Analytics
Off-Heap Memory

Fast Checkpoints
Memory Converged
Infrastructure

In-Memory DB:
Analytics/AI,
Mobility Mgmt

Fast Restart

Resilient VNFs

Scratch & IO Nodes

Distributed Memory:
Cloud Native

Direct-attached DB
Storage
Dedicated Logging

VNF – Virtual Network Function
PMoF - Persistent Memory over FabricDistributed Asynchronous Object Storage
Benefits
3rd Party
Applications
Rich Data
ModelsStorage
Platform▪ Built natively over new userspace
PMEM/NVMe software stack

Workflow

POSIX I/O

HDF5

ApacheArrow

MPI-IO

Open Source Apache 2.0 License▪ Fine-grained, low-latency &True zero-copy I/Os
HDDIntel® QLC 3D Nand SSD▪ Rich storage semantics▪ High throughput/IOPS @arbitraryalignment/size

DAOS Storage Engine

Storage Media…

▪ Scalable communications▪ Software-managed redundancy▪ Rely on COTS hardware

DAOS Architecture• Low-latency, high-message-rate
communications• Collective operations & in-storage
computing• High-latency communications• P2P operations• No HW acceleration

DAOS Storage Engine

Conventional Storage Systems

Metadata, low-latency I/Os &indexing/query

Data & Metadata
BlockInterface

Linux Kernel I/OIntel® 3D-XPoint Storage

HDDIntel® 3D-NAND Storage

MemoryInterface

PMDKIntel® 3D-XPoint Storage

Bulk data
NVMe
Interface

SPDK

3D-NAND/XPoint Storage

HDD

Deployment Options
DAOS system can be deployed in two different ways:

Pooled Storage Model :The DAOS servers can run on dedicated storage nodes inseparate racks. This is a traditional pool model wherestorage is uniformly accessed by all compute nodes. In orderto minimize the number of I/O racks and to optimize floor
space, this approach usually requires high density storage
servers.

Disaggregated Storage Model :In the disaggregated model, the storage nodes areintegrated into compute racks and can be either dedicatedor shared (e.g. in a hyper-converged infrastructure) nodes.
The DAOS servers are thus massively distributed andstorage access is non-uniform and must take locality into
account.

DAOS Data Model• Non-POSIX rich storage API as
the new foundation•

Scalable storage model suitablefor both structured &unstructured data

Storage Pool• key-value stores, multi-dimensional
arrays, columnar databases, …
• Accelerate data analytic/AI
frameworks••

Non-blocking data & metadata
operations
Extendable throughmicroservice architecture

Container

ObjectRecord[]Data Model Library
Array

KV Store

DAOS Storage Engine

Multi-level KV Store

Open Source Apache 2.0 LicenseDataset Management
DAOS Container

DAOS Containerroot• Aggregate related datasets into
manageable and coherent entities
Distributed consistency & automatedrecovery
Full Versioning
Simplified data management

•
•
•
•

Snapshot•

Cross-tier Migration•

Indexing

dir
data
data
data
data
file

dir

dir

data
data
file

data
data
data
file

Encapsulated POSIX Namespace

group

data
data
data
data
data
data
data
data
data
datasetdatasetdatasetDAOS Container
Value
Value
Value
Value

data
data
data
data
file

File-per-process

value

key

value

key

value

key value

key value

Key-value store

HDF5 « File »

key
key
key
key

data
data
data
data
file

key

group
group

data
data
data
data
file

DAOS Container

DAOS Container

group

data
data
data
data
file

DAOS Container
Value
Value
Value
Value

Columnar Database

Root

Shot

Shot

midpoint

data
data
data
data
data
data
data
data
Trace data
Trace @trace

SEGY

Advanced Storage API
Application
key1

@
val1

DAOS

key3
@

@

val3

key3

@

val3

key2
@

val2

@
key1

@

Application

root

@
key2

@

NVMe SSD
val2

val1
val2 con’dFast data retrieval• Avoid file serialization and offset management• Keys can be of any size/type• Keys can be ordered with range query support
Scalable Insert & Fetch• Allow concurrent access/update• Unconstrained by POSIX serialization• Non-blocking• Distributed transactions keep KV store always
consistent
Data indexing• Enable in-storage computing• Query & custom index• Data provenance

POSIX I/O Support• DAOS File System (libdfs)

Single process address space

Encapsulated POSIX namespace
Application/framework can link
directly with libdfs

•
•
•
•
•

Interception Libraryior/mdtest backend provided
MPI-IO driver leveraging collective open
TensorFlow, …

DAOS File System (libdfs)
DAOS library (libdaos)• FUSE Daemon (dfuse)••

Transparent access to DAOS
Involves system calls

End-to-end
user space
No system calls• I/O interception library
•

dfuse

Application / Framework

OS bypass for read/write operations

RPC

RDMA

DAOS Storage Engine

Intel® QLC 3D Nand SSD

MPI-IO Driver for DAOSThe DAOS MPI-IO driver is implemented within the I/O library in MPICH(ROMIO).
• Added as an ADIO driver
Logical File012
• Portable to Open-MPI, Intel MPI, etc.
• https://github.com/daos-stack/mpich
1 byte• daos_adio branch• PR to mpich master in review
DAOS Array DKeys
0• 1 MPI File = 1 DAOS Array Object
Chunk Size = 3

3

4

5

1

6

7

2

8

9

3

Akey = e•0•f
Array Type
Records

0

1

2

3

4

5

6

7

8

Record = 1 cell

Application works seamlessly by just specifying the use of the driver by appending “daos:” to the path.

9

DAOS in IO-500 2019-11#1 in 10 Node Challenge#2 in Ranked List

DAOS: Primary Storage on Aurora
Aurora DAOS configuration▪ Capacity: 230PB▪ Bandwidth: >25TB/s"Combined in Aurora, the Intel compute system, Cray Slingshot network, and the Intel DAOS storage open new possibilities foraccelerating the scientific research needed to solve critical human challenges such as cancer and disease. DAOS enables thecreation of new storage data models tailored specifically to applications like the Cancer Distributed Learning Environment(CANDLE) which provide a powerful platform to advance a wide array of scientific challenges using deep learning.”– Rick Stevens, Associate Laboratory Director for Computing, Environment and Life Sciences"The Argonne Leadership Computing Facility is excited to be the first major production deployment of the DAOS storage
system as part of Aurora, an US exascale system coming in 2021. As designed, it will provide us unprecedented levels of
metadata operation rates and extremely high bandwidth for I/O intensive workloads."– Susan Coghlan, ALCF-X Project Director/Exascale Computing Systems Deputy Director

DAOS Community Roadmap
4Q19

1Q20
1.0

2Q20

3Q20
1.2

Application Interface:- POSIX I/O with conditional
update support

Application Interface:- MPI-IO Driver- HDF5 DAOS Connector- Basic POSIX I/O support- Spark

1Q21

2Q21

1.4

DAOS:
- End-to-end data integrity- Per-container ACL- Improved control plane- Lustre/UNS integration- Replication & self-healing- Erasure code (preview)- Conditional updates

DAOS:- NVMe & DCPMM support- Per-pool ACL- UNS in DAOS via dfuse- Replication & self-healing(preview)

4Q20

3Q21

4Q21

2.0

1Q22
2.2

DAOS:
- Erasure code
- Telemetry & per-job statistics- Distributed transactions

2Q22

3Q22
2.4

DAOS:
- Catastrophic recovery tools

Application Interface:- POSIX I/O with distributed
transaction support- HDF5 data mover- Container parking/serialization

DAOS:- Online server addition- Advanced control plane- Multi OFI provider support

DAOS:- Progressive layout- Placement optimizations- Checksum scrubbing

Application Interface:- POSIX data mover- Async HDF5 operations over DAOSNOTE: All information provided in this roadmap is subject to change without notice.

Resources

Source Code on GitHub
Community mailing list on Groups.io
daos@daos.groups.io
Supporthttps://jira.hpdd.intel.com

Admin Guide

DAOS Solution Brief

Legal Disclaimer
All information provided here is subject to change without notice.
The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from publishedspecifications. Current characterized errata are available on request.
Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation.
Performance varies depending on system configuration. No computer system can be absolutely secure. Check with your system manufacturer orretailer or learn more at intel.com.
Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as
SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of thosefactors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated
purchases, including the performance of that product when combined with other products. For more complete information visit
www.intel.com/benchmarks.

Cost reduction scenarios described are intended as examples of how a given Intel-based product, in the specified circumstances and configurations, may
affect future costs and provide cost savings. Circumstances will vary. Intel does not guarantee any costs or cost reduction.
Intel does not control or audit third-party data. You should review this content, consult other sources, and confirm whether referenced data are
accurate.
Results have been estimated or simulated using internal Intel analysis or architecture simulation or modeling, and provided to you for
informational purposes. Any differences in your system hardware, software or configuration may affect your actual performance.
© Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may beclaimed as the property of others.