Extreme Big Data andDeep Learning Algorithm Platform
Predicting Statistics of ASGD
Collaborative work with DENSO CORPORATION and DENSO IT LABORATORY, INC

In large-scale Asynchronous
Stochastic Gradient Descent(ASGD), mini-batch size andgradient staleness tend to be large
and unpredictable, which increase
the error of trained DNN

Objective functionE
Mini-batch size

Staleness=0-ηΣi ∇ Ei

W(t)W(t+1)

Two asynchronous
updates within one
gradient computation

W(t+3)

W(t+1)

W(t+2)

Staleness=2-ηΣi ∇ Ei
DNN parameters space

We propose a empirical performance model for an ASGD deeplearning system SPRINT which considers probability distribution ofmini-batch size and staleness

These Researches are Supported by JST CREST Grant Numbers JPMJCR1303 and JPMJCR1687

Evaluating Apps on FPGAs

Nowadays, FPGA can rival CPU/GPU performance and energy
efficiency, but also known for its hardness for programming.
We compared three high-level programming approaches for FPGAs● 30-core many-core system (reps. for programmability)● LegUp High-Level Synthesis (reps for multiple custom accelerators)● Intel OpenCL for FPGA (reps for Deep-pipeline designs)
We evaluated using Rodinia Benchmark Suite on Stratix V FPGA.
We improved memory hierarchy for many-core and multi-accelerator
designs through cache multi-banking.

● Intel OpenCL for FPGA shown highest average performance● LegUp can remain competitive for good performance andspatial/temporal locality, even without improvement.
● Many-core system offers good programmability, but often does not
perform well compared to other approaches

Fast SpGEMM on GPU

We have devised new Sparse General Matrix-Matrix Multiplication
algorithm on GPU, which achieves further speedups and reducesmemory usage so that various matrix data can be applied byutilizing GPU’ s on-chip shared memory and appropriate assigningof GPU resources.
(1) Count #intermediate products
Two Phases Algorithm : 1st phase(2) Divide the rows into groups bycounts the number of non-zero#intermediate products
elements of output matrix, and 2nd(3) Count #non-zero elements
phase calculates the output matrix(4) Set row pointers ofoutput matrix→ Reduce memory usage
Grouping rows (1, 2, 6)(5) Memory allocation of output matrix→ Better utilization of GPU resources(6) Divide the rows into groups byTwo ways threads assignments#non-zero elements→ Improve the load-balance(7) Compute the output matrix
Hash table on fast shared memory
a. Calculate values and column
indices on hash table→ Accelerate counting part (3) andb. Shrink the hash tablecalculation part (7)c. Store to the memory with sorting
CUSP

cuSPARSE

BHSPARSE

nsparse35

25250020

2000

15MByteGFLOPSCUSP

105

cuSPARSE

BHSPARSE

65GFLOPS

4BHSPARSE

Protein

FEM/Spheres

webbaseProtein

FEM/Spheres0Circuit

Epidemiology

vfs
page-cache
NVM Device(up to 64GB)

Data transfer methods

webbaseReference: Yusuke Nagasaka, Akira Nukada, Satoshi Matsuoka, "High-performance and Memory-saving Sparse General
Matrix-Matrix Multiplication for NVIDIA Pascal GPU", ICPP 2017.

um Unified Memory + IO
dg DRAGON

doubleCode available at:https://github.com/EBD-CREST/nsparse
Economics

dragon-ext

webbaseMultiplication for Irregular Data” , IPDPS2014

1

nvidia-uvm(up to 12GB)[3] Liu et al., “An Efficient GPU General Sparse Matrix-Matrix

2libcudart

hr cudaHostRegister + mmap

[2] NVIDIA, “Nvidia cuda sparse matrix library (cuSPARSE)”

3libdragon

1000

matrix and graph computations ver.0.5.1”

Speedup is upto x4.1

Operation

df cudaMemcpy + IO

[1] Dalton et al., “CUSP: Generic parallel algorithms for sparse

nsparseGPU Application

10.9% reduction on
average fromcuSPARSE

singlecuSPARSE

Architecture

1500

0

CUSP

Problem sizes grow larger than GPU and even host memory capacity.
We proposed DRAGON, a framework that seamlessly enables allclasses of GPU applications to directly access terabytes of data on
storage with user-oblivious addressing. DRAGON leverages GPU
hardware page-faulting mechanism and two-level prefetching withdirect page-cache access to achieve this functionality while bestowing
up to 2.3x speedup compared with using Unified Memory + POSIX IO.

Performance Evaluation

nsparse5000

Collaborative work with Oak Ridge National Laboratory

GPUMemory Usage

30

Unlimited GPU Memory with DRAGON

2.3x

Double Precision Performance

Maruyama, S. Matsuoka, IEEE FPL 2017

Libraryof Asynchronous SGD Parameters for a Large-Scale Distributed Deep Learning System on GPU Supercomputers", IEEE BigData 2016

Reference: "Evaluating High-Level Design Strategies on FPGAs for High-Performance Computing", A. Podobas, H.R. Zohouri, N.

Driver

Reference: Yosuke Oyama, Akihiro Nomura, Ikuro Sato, Hiroki Nishimura, Yukimasa Tamatsu, and Satoshi Matsuoka, "Predicting Statistics

Out-of-core was faster thanthe extrapolated baseline (um)

DRAGON is the only
existing out-of-core
solution with useroblivious addressing

Presentation at SC18 Technical Program:
11:00-11:30, Wednesday, Room C141/143/149

Reference: Pak Markthub, Mehmet E. Belviranli, Seyong Lee, Jeffrey S. Vetter, and Satoshi Matsuoka,“DRAGON: Breaking GPU Memory Capacity Limits with Direct NVM Access” , SC18

http://www.gsic.titech.ac.jp/sc18