Extreme Big Data andDeep Learning Algorithm Platform
Predicting Statistics of ASGD
Collaborative work with DENSO CORPORATION and DENSO IT LABORATORY, INC

In large-scale Asynchronous
Stochastic Gradient Descent(ASGD), mini-batch size andgradient staleness tend to be large
and unpredictable, which increase
the error of trained DNN

Objective functionE
Mini-batch size
Staleness=0-ηΣi ∇ Ei

W(t)W(t+1)

Two asynchronous
updates within one
gradient computation

W(t+3)

W(t+1)
W(t+2)

Staleness=2-ηΣi ∇ Ei
DNN parameters space

We propose a empirical performance model for an ASGD deeplearning system SPRINT which considers probability distribution ofmini-batch size and staleness

These Researches are Supported by JST CREST Grant Numbers JPMJCR1303 and JPMJCR1687

Evaluating Apps on FPGAs

Nowadays, FPGA can rival CPU/GPU performance and energy
efficiency, but also known for its hardness for programming.
We compared three high-level programming approaches for FPGAs● 30-core many-core system (reps. for programmability)● LegUp High-Level Synthesis (reps for multiple custom accelerators)● Intel OpenCL for FPGA (reps for Deep-pipeline designs)
We evaluated using Rodinia Benchmark Suite on Stratix V FPGA.
We improved memory hierarchy for many-core and multi-accelerator
designs through cache multi-banking.

● Intel OpenCL for FPGA shown highest average performance● LegUp can remain competitive for good performance andspatial/temporal locality, even without improvement.
● Many-core system offers good programmability, but often does not
perform well compared to other approaches
Reference: Yosuke Oyama, Akihiro Nomura, Ikuro Sato, Hiroki Nishimura, Yukimasa Tamatsu, and Satoshi Matsuoka, "Predicting StatisticsReference: "Evaluating High-Level Design Strategies on FPGAs for High-Performance Computing", A. Podobas, H.R. Zohouri, N.

of Asynchronous SGD Parameters for a Large-Scale Distributed Deep Learning System on GPU Supercomputers", IEEE BigData 2016

Maruyama, S. Matsuoka, IEEE FPL 2017

We have devised new Sparse General Matrix-Matrix Multiplication
algorithm on GPU, which achieves further speedups and reduces
memory usage so that various matrix data can be applied byutilizing GPU’ s on-chip shared memory and appropriate assigningof GPU resources.
(1) Count #intermediate products
Two Phases Algorithm : 1st phase(2) Divide the rows into groups bycounts the number of non-zero#intermediate products
elements of output matrix, and 2nd(3) Count #non-zero elements
phase calculates the output matrix(4) Set row pointers ofoutput matrix→ Reduce memory usage
Grouping rows (1, 2, 6)(5) Memory allocation of output matrix→ Better utilization of GPU resources(6) Divide the rows into groups by
Two ways threads assignments#non-zero elements→ Improve the load-balance(7) Compute the output matrix
Hash table on fast shared memory
a. Calculate values and column
indices on hash table→ Accelerate counting part (3) andb. Shrink the hash tablecalculation part (7)c. Store to the memory with sorting

Multi-GPU batch-queue systems usually experience large numberof idle GPUs due to the scattered idle-GPU problem (Fig.1). We
addressed this problem by allowing jobs to utilize remote GPUs andmigrating execution on a remote GPU back to a local GPU as soon
as one becomes available. This method enables the systems toserve more GPU jobs concurrently while minimizing execution time
penalty caused by remote GPU communication.

Fast SpGEMM on GPU

Double Precision Performance
CUSP

cuSPARSE

BHSPARSE

nsparse35CUSP

252500202000

15MByteGFLOPS

Fig.1: Job 1 and Job 2 cannot run concurrently
as Job 2 wants two unoccupied GPUs on thesame node.

F ig .2 : T h e ar c h i tec tur e o f mrC U D A , o u r
middleware for handling remote GPU migration
on top of rCUDA.

Memory Usage

30

10Increasing GPU Occupancy

cuSPARSE

BHSPARSE

nsparse

10.9% reduction onaverage fromcuSPARSE

1500
1000
50000Protein

FEM/Spheres

webbaseProtein

FEM/Spheres

singleCUSPcuSPARSE

BHSPARSEGFLOPS

double[1] Dalton et al., “CUSP: Generic parallel algorithms for sparse

nsparse65webbasematrix and graph computations ver.0.5.1”

Speedup is up
to x4.1[2] NVIDIA, “Nvidia cuda sparse matrix library (cuSPARSE)”

4[3] Liu et al., “An Efficient GPU General Sparse Matrix-Matrix

3

Multiplication for Irregular Data” , IPDPS2014

2

Code available at:https://github.com/EBD-CREST/nsparse

10Economicsmakespanis reducedby ~30%5Circuit

Epidemiology

Fig.3: GPU occupancy patterns when using
FCFS (top) and our method (bottom).

Fig.4: Distribution of jobs’ lifetime (waiting +execution time) decrease when using ourmethod compared with FCFS.

webbaseReference: Yusuke Nagasaka, Akira Nukada, Satoshi Matsuoka, "High-performance and Memory-saving Sparse General
Matrix-Matrix Multiplication for NVIDIA Pascal GPU", ICPP 2017.

Reference: P.Markthub, A.Nomura, and S.Matsuoka, “Serving More GPU Jobs, with Low Penalty, using Remote GPU
Execution and Migration,” IEEE Cluster 2016.

http://www.gsic.titech.ac.jp/sc17