Research towards
Future Supercomputer for Everybody
Exploring Next-Gen Architecture with Complex Memory Hierarchy
To achieve higher performance and larger capacity on recent and future architectures, we need to explore next-gen memory hierarchy,
including heterogeneous devices. Also placement of many-cores and memory devices can be reconsidered; 3D stacking of cores andmemory chips may go mainstream in HPC/Big-data area.

Understanding Memory Performance

Under the above-mentioned assumptions, understanding memory
performance will be more challenging for data and address trafficrouted among many-cores, which may often conform 2D mesh.
To analyze its effects, we have conducted preliminary measurementsof memory latency from every core of Xeon Phi. We see 9% differencein the current architecture; the effect will be expanded in futurearchitecture where chip cores are also stacked.
3 nm UV fabrication

High Capacity Flash NVM>1 TBytes Capacity
Medium Bandwidth 2.5D DRAM
>64GBytes Capacity~3TB/s Bandwidth
NVM / FlashNVM / FlashNVM / Flash33 32 25 24

170.9 171.0 168.1 168.1

35 34 27 26 19 18 13 125410763298161.2 161.2 163.1 163.2

171.3 171.4 168.2 168.3 166.5 166.5 162.9 162.9 161.5 161.5 163.8 163.7

Low Latency 3D SRAM
>8GBytes Capacity>10TB/s Bandwidth

2.5D DRAM

3D SRAM

2.5D DRAM

3D SRAM

2.5D DRAM

3D SRAM

2.5D DRAM

Dataflow + Scalar Processor

29 28 21 20 15

Photonic Network
VCSEL-based Multi-Port High Injection
1Tbps x 12 = 12Tbps

168.9 168.9 166.8 166.8 163.6

Memory Performance Estimation Tool

Towards high performance node architecture in future, it is important tounderstand application performance, not only synthetic benchmark.
For this purpose, we are developing an estimation tool of memory
performance of future architecture, whose features are:
Light weight: it takes history of the entire application execution as input,
which can be too heavy for cycle-wise simulators.
To support complex structure: it considers placement of many coresand topology including chip-lets and memory chips.

162.0 162.0

31 30 23 22 17 16 11 10

169.9 169.9 167.8 167.8 164.7 164.7 162.9 162.9

63 62 59 58 53 52 47 46 41 40

174.1 174.1 171.8 171.8 168.8 168.9 165.9 165.8 165.1 165.1

65 64 61 60 55 54 49 48 43 42 37 36

VCSEL Optical
Launch Pad

VCSEL Optical
Launch Pad

175.7 175.6 172.8 172.8 170.5 170.6 167.3 167.3 166.2 166.2 167.7 167.8

SRAM

67 66

Neuromorphic Accelerator

176.7 176.6

TSV Interposer

TSV Interposer

57 56 51 50 45 44 39 38

171.6 171.6 168.4 168.7 167.5 167.5 168.8 168.8

max: 176.7 (ns)delta: 15.5 (ns)min: 161.2 (ns)Organic Substrate

An Example of Future Architecture with 3D Stacking Technologyand Heterogeneous Memory

Memory access latency from every coreof Xeon Phi 7285. As the target ofaccess, one of MCDRAM modules,
placed close to core 4, 5 is used.

Acknowledgments: This research is supported by the New Energy and Industrial Technology Development Organization (NEDO), Japan.
Contact: Toshio Endo (endo@is.ttiech.ac.jp)

Deep Learning Technology Dealing with Deeper Memory HierarchyHybrid data and model parallelism

Deep Compression

Second order optimization methods
require the communication of theHessian matrix. The present method
reduces the communication significantly through Kronecker factorization and the use of reduce-scatter andallgather collectives to swtich between data parallel and model parallel execution. The optimal implementation of allreduce operations has thesame communication pattern, so thepresent method simply inserts thecomputation of the Hessian.

Quantized Weight Matrix

Pre-Trained NetworkPruning & Splicing
Network2-1-2-2-2-2

0

2-1-2-2-2-2

02-1

Encoded Weight Matrix

Re-Trainindices (3/4 bit int)

3122

31

scalar indicesParameter Fine-Tuning

Adversarial Training + Quantization

shape info

202-2-2

Train Network

12-1

Deep Compression focuseson shrinking model size
whilst retaining accuracy oftrained models. In this work,
pruning and splicing forfine-tuning and adversarial
training as a regularizer resulting in better accuracy
and up to 91x compression
rate without compromisingon any other performanceloss.

codebook metadata

DEFLATE Encoding

Web-based Interactive Access to Supercomputer Nodes
Supercomputers are traditionally designed to execute large andnon-interactive jobs with a batch scheduler. This style does notmatch the users who want to interact with compute nodes: debugging, visualization, and education of novice users in the classroom.
In TSUBAME3.0 supercomputer, we introduced two new features
to satisfy such demands:
Interactive use only nodes: We spare four nodes as dedicatednodes for shared interactive use. Users can run interactive jobs
without waiting for job execution, even if the compute nodes arefilled with batch jobs. Performance might not be optimal as thenodes are shared with multiple jobs, but still acceptable for such interactive usage.
Web-based access to Jupyter Lab on compute nodes: Novice
users can use command-line shell and Python console running on
high-performance TSUBAME nodes without any complicatedknowledge of Linux, such as SSH key-pair authentication.

User
Compute Node3. Direct Access toJupyter on Compute Nodevia Web Portal
Web-based User Portal

https://www.gsic.titech.ac.jp/sc20