Hierarchical matrix(H-matrix) for CNN acceleration- Hierarchical matrix is an efficient data-sparse representations ofcertain densely populated matrices.
- CNN(Convolutional Neural Network)• Back ground- Hierarchical matrix(H-matrix) is a an
approximated form represent × correlationsof objects, which usually requires a × huge
dense matrix.
dense matrix

Hierarchical matrix

The H-matrix approximation of dense matrix.
The red blocks are dense matrices. The green block
are low-rank matrices with rank .

- Significant savings in memory when compressed⟹

log- Computational complexity()⟹ (log)such as matrix-matrix multiplication,
LU factorization, Inversion…

Preliminary Results – Compression rate of matrices
Deep Learning (CNN)

SDPARA
Compression rate(%)

25.00

3.567

14.63

13.96

12.39

10.86

11.85

11.88
10.52

7.72
5.17

5.00

6.00

3.000
12.81

4.05

2.500

2.80

2.25

1.64

1.33

2.396
1.834

2.000
1.500
1.000

0.841

1632

2673

4150

Size of matrix(n × n)
6171
8856
12337
bem1d

16758

22275

29056

sdparaCompressive rate = (uncompressed size) / (compressed size)

4.717

5.00

4.338

4.00
3.00

1.610

2.00

0.500

1.00

0.000

0.00

6.946

7.00

3.500

15.00
10.00

8.00

Size(MB)

15.80

4.000
17.06

Size(MB)

20.00

Size of matrix

Size of matrix

20.23

0.00

Matrix A

Matrix B

non-compressed

compressed(m, n, k) = (1764, 1350, 178)

Matrix A

non-compressed

Matrix B

compressed(m, n, k) = (324, 298, 1908)→ Matrix A successfully compressed! → Matrix B successfully compressed!

We can compress the matrix in some applications.
- bem1d: 1-Dimention Boundary element method- sdpara: A parallel implementation of the inter-point
method for Semi-Define Programming(SDP)In CNN system application, Sgemm(Single precision
floating General Matrix Multiplication) C = AB + C
accounts for large part of calculation (around 70%).

Power optimization using Deep Q-Network・ Background
Power optimization by frequency control in existing research

Performance counter
Temperature
Frequency,…

P = f (x1, x2 ,...)
Texe = g(x1, x2 ,...)

Deep Q-Network (DQN)

Use Deep Learning for analysis.

CounterPowerFrequency
Temperatureetc.

↓Deep reinforcement learningCalculate action value function Q from neural network
Used for game playing AI, robot car, AlphaGO.

Frequency↑Detailed analysis is necessary
Low versatility
・ Objective
Implement the computer
control system using Deep Q-Network.

Kento Teranishi

Frequency
controlUsing ML to Approximate Sciences - Fluid Dynamics
Example (slide courtesy of Bill Dally @ NVIDIA)“… Implementation led to a speed-up of one to three orders of magnitude
compared to the state-of-the-art position-based fluid solver and runs inreal-time for systems with up to 2 million particles”“Data-driven Fluid Simulations using Regression Forests” http://people.inf.ethz.ch/ladickyl/fluid_sigasia15.pdf
16/08/08

SWoPP2016930

The current status of AI & Big Data in JapanWe need the triage of algorithms/infrastructure/data but we lack the
infrastructure dedicated to AI & Big Data (c.f. Google)
Machine Learning
Algorithms

Use of Massive Scale Data now
WastedPetabytes of Drive
Recording VideoFA&Robots

AI&Data Processing
Infrastructures

Web access andmerchandice

Data

IoT Communication,
location & other data

The current status of AI & Big Data in JapanWe need the triage of algorithms/infrastructure/data but we lack theinfrastructure dedicated to AI & Big Data (c.f. Google)
Acceleration & Scaling of DL& other ML Algorithms & SW

Application-based Solution providersof ML (e.g. Pharma, Semiconductors)
Custom ML/DL Software

Machine Learning
Algorithms

Investigating the Application of DL“Chainer” OSS DL Framework
Many applications in manufacturingweb, pharma, etc.

Analysis of automotive cameras
Performance analysis & improvement of DL

Use of Massive Scale Data now
WastedPetabytes of Drive
Recording Video
FA&Robots

AI&Data Processing
Infrastructures

Web access andmerchandice

Data

IoT Communication,
location & other data

The current status of AI & Big Data in JapanWe need the triage of algorithms/infrastructure/data but we lack theinfrastructure dedicated to AI & Big Data (c.f. Google)
深層学習処理の⾼度化・⾼速化を模索

Application-based Solution providers
of ML (e.g. Pharma, Semiconductors)
Custom ML/DL Software

Machine Learning
Algorithms

Investigating the Application of DL“Chainer” OSS DL Framework
Many applications in manufacturingweb, pharma, etc.

Massive Rise in Computing
Requirements

Analysis⾞載カメラ映像解析
of automotive cameras
深層学習高性能化高速化
Performance
analysis & improvement of DLに関する基礎研究

Use of Massive Scale Data now
WastedPetabytes of Drive
Recording Video
FA&Robots

Insufficient to Counter the Giants(Google, Microsoft, Baidu etc.)
in their own game
AI&Data

Massive “Big” Data
Infrastructures in Training

Web access andmerchandice

Data

IoT Communication,location & other data

The “Chicken or Egg Problem” ofAI-HPC Infrastructures• “On Premise” machines in clients => “Can’t invest in big in AI
machines unless we forecast good ROI. We don’t have theexperience in running on big machines.”• Public Clouds other than the giants => “Can’t invest big in AI
machines unless we forecast good ROI. We are cutthroat.”• Large scale supercomputer centers => “Can’t invest big in AImachines unless we forecast good ROI. Can’t sacrifice our existingclients and our machines are full”• Thus the giants dominate, AI technologies, big data, and people staybehind the corporate firewalls…

But Commercial Companies esp. the “AIGiants”are Leading AI R&D, are they not?
• Yes, but that is because their shot-term goals could harvest thelow hanging fruits in DNN rejuvenated AI• But AI/BD research is just beginning--- if we leave it to theinterests of commercial companies, we cannot tackle difficult
problems with no proven ROI• Very unhealthy for research• This is different from more maturefields, such as pharmaceuticals oraerospace, where there is balanced
investments and innovations in bothacademia/government and the industry

TSUBAME-KFC/DL: TSUBAME3 Prototype [ICPADS2014]
Oil Immersive Cooling＋ Hot Water Cooling + High Density Packaging + FineGrained Power Monitoring and Control, upgrade to /DL Oct. 2015
High Temperature Cooling
Oil Loop 35~45℃⇒ Water Loop 25~35℃(c.f. TSUBAME2: 7~17℃)
Single Rack High Density OilImmersion
168 NVIDIA K80 GPUs + Xeon
413+TFlops (DFP)
1.5PFlops (SFP)~60KW/rack

Cooling Tower：
Water 25~35℃⇒ To Ambient Air

2013年11月/2014年6月
Word #1 Green500
Container Facility
20 feet container (16m2)
Fully Unmanned Operation

ABCI Prototype: AIST AI Cloud (AAIC)March 2017 (System Vendor: NEC)
• 400x NVIDIA Tesla P100s and Infiniband EDR accelerate various AI workloads
including ML (Machine Learning) and DL (Deep Learning).
• Advanced data analytics leveraged by 4PiB shared Big Data Storage and Apache
Spark w/ its ecosystem.
SINET5InternetConnection

Firewall
10-100GbE• FortiGate 3815D x2• FortiAnalyzer 1000E x2

UTM Firewall
40-100Gbps class
10GbE

Service and Management Network

GbE or 10GbE

GbE or 10GbE

AI Computation System

Computation Nodes (w/GPU) x50• Intel Xeon E5 v4 x2• NVIDIA Tesla P100 (NVLink) x8• 256GiB Memory, 480GB SSD

Computation Nodes (w/o GPU) x68• Intel Xeon E5 v4 x2• 256GiB Memory, 480GB SSD

Computation Network

400 Pascal GPUs
30TB Memory
56TB SSDInteractive Nodesx2
Mgmt & Service
Nodes x16

IB EDR (100Gbps)

Mellanox CS7520 Director Switch• EDR (100Gbps) x216

Large Capacity Storage System
DDN SFA14K• File server (w/10GbEx2,IB EDRx4) x4• 8TB 7.2Krpm NL-SAS
HDD x730• GRIDScaler (GPFS)

Bi-direction 200Gbps
Full bi-section bandwidth>4PiB effective
RW100GB/s
IB EDR (100Gbps)

TSUBAME3.0

2017 Q2 TSUBAME3.0 Leading Machine Towards Exa & Big Data

1.“Everybody’s Supercomputer” - High Performance (12~24 DP Petaflops, 125~325TB/s Mem,55~185Tbit/s NW), innovative high cost/performance packaging & design, in mere 180m2…
2.“Extreme Green” – ~10GFlops/W power-efficient architecture, system-wide power control,advanced cooling, future energy reservoir load leveling & energy recovery
3.“Big Data Convergence” – Extreme high BW &capacity, deep memory
hierarchy, extreme I/O acceleration, Big Data SW Stack2013
for machine learning, graph processing, …
TSUBAME2.5
4.“Cloud SC” – dynamic deployment, container-based
node co-location & dynamic configuration, resource
elasticity, assimilation of public clouds…
5.“Transparency” - full monitoring &user visibility of machine& job state,accountability
via reproducibility
2006 TSUBAME1.080 Teraflops, #1 Asia #7 World“Everybody’s Supercomputer”

upgrade5.7PF DFP
/17.1PF SFP
20% powerreduction2016 TSUBAME3.0~20PF(DFP) 4~5PB/s Mem BW
10GFlops/W power efficiency
Big Data & Cloud Convergence2010 TSUBAME2.0
2.4 Petaflops #4 World“Greenest Production SC”

Large Scale Simulation2013 TSUBAME-KFC Big Data Analytics
38#1 Green 500
IndustrialApps
2011 ACM Gordon Bell PrizeComparison of Machine Learning / AI Capabilities
of TSUBAME3+2.5 and K-Computer

X7~10>>TSUBAME2.5(2013)+TSUBAME3.0(2017)(effectively more
due to optimized
DL SW Stack onGPUs)

Deep Learning / AI Capabilities
FP16+FP32 up to ~100 Petaflops+ up to 100PB online storage

K Computer (2011)
Deep Learning
FP32 11.4 Petaflops

BG/Q Sequoia (2011)
22 Petaflops SFP/DFPMETI AIST-AIRC ABCI
as the worlds first large-scale OPEN AI Infrastructure• ABCI: AI Bridging Cloud Infrastructure• Top-Level SC compute & data capability (130~200 AI-Petaflops)• Open Public & Dedicated infrastructure for Al & Big Data Algorithms,Software and Applications• Platform to accelerate joint academic-industry R&D for AI in Japan••••130~200 AI-Petaflops< 3MW Power< 1.1 Avg. PUE
Operational 2017Q3~Q4Univ. Tokyo Kashiwa Campus40

ABCI – 2017Q4~ 2018Q1• Extreme computing power– w/ 130〜200 AI-PFlops for AI, ML, DL– x1 million speedup over high-end PC: 1 Day training for
3000-Year DNN training job– TSUBAME-KFC (1.4 AI-Pflops) x 90 users (T2 avg)

• Big Data and HPC converged modern design– For advanced data analytics (Big Data) and scientificsimulation (HPC), etc.
– Leverage Tokyo Tech’s “TSUBAME3” design, butdifferences/enhancements being AI/BD centric• Ultra high bandwidth and low latency in memory,network, and storage– For accelerating various AI/BD workloads– Data-centric architecture, optimizes data movement• Big Data/AI and HPC SW Stack Convergence– Incl. results from JST-CREST EBD– Wide contributions
community desirable.

fromthePC

Cluster41

Cloud Infrastructure• Ultra-dense IDC design from ground-up

Reference Image– Custom inexpensive lightweight “warehouse”
building w/ substantial earthquake tolerance• Cloud ecosystem– Wide-ranging Big
software stacks

DataandHPCstandard• Extreme green– Ambient warm liquid cooling, large Li-ion battery
storage, and high-efficiency power supplies, etc.

引用元: 共同通信デジタル

Reference Image• Advanced cloud-based operation– Incl. dynamic deployment, container-based
virtualized provisioning, multitenant partitioning,
and automatic failure recovery, etc.
– Joining HPC and Cloud Software stack for real

引用元: NEC導⼊事例
42

National Institute forラボ長（産総研研究職 or 東工大
松岡
Advanced Industrial
教員/クロスアポ）
Science and Technology
副ラボ長（産総研研究職）(AIST)
Matsuoka will be副ラボ長（産総研事務職）
appointed 15% toGSIC
ラボ研究主幹（産総研研究職）
AIST AI-OIL
Resources and Acceleration ofstarting summer
ラボ構成員
Ministry of Economics
AI / Big Data, systems research
Tsubame 3.0/2.5
Joing Organization@Odaiba
Trade and Industry (METI)Big Data /AI
AIST-TokyoTech
resources
AIST Artificial

Intelligence
ResearchCenter (AIRC)
Application Area
Natural Langauge
Processing
Robotics
Security

Joint
Research on
AI / Big Dataandapplications

AI/Big Data Open
Innovation
Laboratory (OIL)

Industrial
Collaboration in data,applications
Industry

Basic Researchin Big Data / AI
algorithms andmethodologies“Smart AI TechnologyResearchOrganization”
Other Big Data / AI
research organizations
and proposals

Software Ecosystem for HPC in AI

Different SW Ecosystem between HPC and AI/BD/Cloud

How to achieve convergence—for real, for rapid tech transfer
Existing Clouds
BD/AI User Applications

Machine Learnig
MLlib/
Mahout/Chainer

Graph Processing
GraphX/
Giraph/ScaleGraphSQL/Non-SQL
Hive/Pig

Java・Scala・Python + IDL
MapReduce Framework
Spark/Hadoop
RDB

PostgresQL

CloudDB/NoSQL

Hbase/Cassandra/MondoDB

Distributed Filesysem
HDFS & Object Store

Coordination Service
ZooKeeper

VM(KVM), Container(Docker), Cloud Services(OpenStack)Linux OSEthernet
TOR Swtiches
HighLatency/Low
Capacity NW

Application Layer• Cloud Jobs often Interactive w/resource control REST APIs
• HPC Jobs are Batch-Oriented, resource control by MPI
System Software Layer• Cloud employs High Productivity Languages but
performance neglected, focus on data analytics anddynamic frequent changes• HPC employs High Performance Languages but requires
Ninja Programmers, low productivity. Kernels & compilers
well tuned & result shared by many programs, less rewrite• Cloud focused on databases and data manipulation workflow• HPC focused on compute kernels, even for data processing.
Jobs scales to thousands of jobs, thus debugging andperformance tuning• Cloud requires purpose-specific computing/data environment
as well as their mutual isolation & security• HPC requires environment for fast & lean use of resources,
but on modern machines require considerable system
software support

x86 CPU

HPC User CodeNumerical Libraries
LAPACK, FFTW

• Cloud HW based on Web Server “commodity” x86 servers,
distributed storage on nodes assuming REST API access• HPC HW aggressively adopts new technologies such a sGPUs, focused on ultimate performance at higher cost,shared storage to support legacy apps

Various DSLs

WorkflowSystems

Fortran・C・C++ ＋ IDLMPI・OpenMP/ACC・CUDA/OpenCL
Parallel Debuggers and Profilers
Parallel Filesystem
Lustre, GPFS,

OS Layer

Hardware Layer

Local NodeStorage

Existing Supercomputers

Batch Job Schedulers
PBS Pro, Slurm, UGE

Linux OSInfiniBand/OPA
High Capacity
Low Latency NWHigh Performance
SAN＋Burst Buffers

Various convergence research efforts underway but no realistic converged SW
Stack yet => achieving HPC – AI/BD/Cloud convergence key ABCI goal

X86 +Acceleratorse.g. GPUs,FPGAs

We are implementing the US AI&BD strategies already…in Japan, at AIRC w/ABCI• Strategy 5: Develop shared public datasets andenvironments for AI training and testing. The
depth, quality, and accuracy of training datasetsand resources significantly affect AI performance.
Researchers need to develop high quality
datasets and environments and enable
responsible access to high-quality datasets as wellas to testing and training resources.
• Strategy 6: Measure and evaluate AI technologies
through standards and benchmarks. Essential toadvancements in AI are standards, benchmarks,testbeds, and community engagement that guideand evaluate progress in AI. Additional research isneeded to develop a broad spectrum ofevaluative techniques.

Co-Design of BD/ML/AI with HPC using BD/ML/AI- for survival of HPC
Accelerating
Conventional HPC Apps

Optimizing System
Software and Ops

Acceleration and Scaling of
BD/ML/AI via HPC andTechnologies andInfrastructures

Big Data AIOriented
Supercomput
ers

Future Big Data・AI
Supercomputer Design

ABCI: World’s first andlargest open 100 Peta AIFlops AI Supercomputer,
Fall 2017, for co-design

Mutual and SemiAutomated CoAcceleration ofHPC and BD/ML/AI
Acceleration
Scaling, andControl of HPC via
BD/ML/AI andfuture SC designs

Big Data andML/AI Apps
andMethodologies

Large Scale Graphs

Image and Video

Robots / Drones

What is worse: Moore’s Law will end in the 2020’s•Much of underlying IT performance growth due to Moore’s law• “LSI: x2 transistors in 1~1.5 years”• Causing qualitative “leaps” in IT and societal innovations• The main reason we have supercomputers and Google...

•But this is slowing down & ending, by mid 2020s…!!!
• End of Lithography shrinks
The curse of constant• End of Dennard scaling
Gordon Moore
transistor power shall• End of Fab Economics
soon be upon us•How do we sustain “performance growth” beyond the “end ofMoore”?
• Not just one-time speed bumps• Will affect all aspects of IT, including BD/AI/ML/IoT, not just HPC• End of IT as we know it

20 year Eras towards of End of Moore’s Law
20-year•
Moore-Dennard
Single Core
ILP-Vector
Killer-Micro Era

1980s~2004
Dennard scaling,
perf+ = single
3-5nm andthread+ = transistor
beyond 2025& freq+ = power+Constant• 2004~2015 feature
Transistor Power
20 yearscaling, perf+ =Post-Dennard
Many-Core Eratransistor+ =core#+, constantpower• 2015~2025 allabove gets harder
20-year• 2025~ post-Moore,
Next-Gen
constant
Post-Moore era
feature&power =Need to realize the next 20-year era of supercomputing flat performance

The “curse of constant transistor power”- Ignorance of this is like ignoring global warming • Systems people have been telling the algorithm people that
“FLOPS will be free, bandwidth is important, so devise
algorithms under that assumption”• This will certainly be true until exascale in 2020…
• But when Moore’s Law ends in 2025-2030, constant transistor
power (esp. for logic) = FLOPS will no longer be free!
• So algorithms that simply increase arithmetic intensity will nolonger scale beyond that point• Like countering global warming – need disruptive change incomputing – in HW-SW-Alg-Apps etc. for the next 20 year era

Performance growth via data-centric computing:“From FLOPS to BYTES”• Identify the new parameter(s) for scaling over time
• Because data-related parameters (e.g. capacity and bandwidth) will still
likely continue to grow towards 2040s• Can grow transistor# for compute, but CANNOT use them AT THE SAMETIME(Dark Silicon) => multiple computing units specialized to type of data
• Continued capacity growth: 3D stacking (esp. direct silicon layering) andlow power NVM (e.g. ReRAM)• Continued BW growth: Data movement energy will be capped constant bydense 3D design and advanced optics from silicon photonics technologies• Almost back to the old “vector” days(?), but no free lunch – latency still
problem, locality still important, need general algorithmic acceleration
thru data capacity and bandwidth, not FLOPSMany Core Era

Post Moore Era

Flops-Centric Algorithms and Apps

Bytes-Centric Algorithms and Apps

Flops-Centric System Software

Bytes-Centric System Software

Hardware/Software System APIs
Flops-Centric Massively Parallel Architecture

Hardware/Software System APIs
Data-Centric Heterogeneous Architecture

Homogeneous General Purpose Nodes
Compute+ Localized Data
Compute
Nodes

Nodes

Gen CPUGen CPUDataData
ComputeNodes

ComputeNodes~2025
M-P Extinction
EventHeterogeneous CPUs + Holistic DataMassive BW
3-D Package
Non-Volatile
Memory汎用CPUGen CPU

Data

Data

Loosely Coupled with Electronic Interconnect

Transistor Lithography Scaling(CMOS Logic Circuits, DRAM/SRAM)

Reconfigurable
Dataflow
DNN&Neuromorphic

Low Precision
Error-Prone

OpticalComputing
Quantum
ComputingUltra Tightly Coupled w/Aggressive
3-D+Photonic Switching Interconnected

Novel Devices + CMOS (Dark Silicon)(Nanophotonics, Non-Volatile Devices etc.)

Multi-Phyics
SImulation

Manufacturing

Massive Medical
Imaging

Fusion/Plasma

EMF Analysis

Auto Tuning

Post-Moore is NOT aMore-Moore device
as a panacea

Post-Moore Computational
Science Libraries

Couplers
BW Reducing Alg.
Data Assimilation

Post-Moore Data Scienceand AI Libraries

Device & arch. advances Post-Moore Programming Model
improving data-related
Data-oriented
Schedulingparameters over time

Low Rank
Approximation

Parallel Space-and-Time
Algorithms

Out-of-core Alg
High B/F Algorithms

Uncertainty
Quantification
Latency
HidingPost-Moore High Bandwidth Hierarchical Memory
Model
Data-Movement
Hierarchical Data
Fault
Runtime
Abstractions
Tolerance

Post-Moore
Performamce
ModelsHigh-Level
Synthesis
Compilers

Machine Learning
based acceleration

AcceleratorSpecific
Compilers

ProgrammableLogic“Rebooting Computing”
Accelerator
Data & Custom Compute Centric Platform“Binaries”
in terms of devices,
Silicon Photonics WDM Interconnect
Photonic Switching
Brain-inspired Computing
architectures, software.New memory Devices
Photonic Interposes
Quantum Computing
PC-RAM
Low Precision &Neural Networks/
Photonic Compute Devices
Algorithms, andProbablistic
ReRAM
Neromorphic/
Optical Packet Switching
Computing
Izing - Annealing
applications necessary STT-MRAM
Low-Reliability computing
Low-Reliablility Communication
Near threshold computing=> Co-Design even
3D architecture
Building Block “gluable” architectureNext gen VIAs & silicon Inductive ＴＣＩ
Data Memoization
more importantCustomizable logic
fabrication
Tnugsten VIAs and 3D siliconc.f. Exascale
Communication
MemoryComputation

Post-Moore
Performance
Parameters

53

Post Moore Era Supercomputing Workshop @ SC16
• https://sites.google.com/site/2016pmes/• Jeff Vetter (ORNL), Satoshi Matsuoka (Tokyo Tech) et. al.