Research towards
Future Supercomputer for Everybody
Exploring Next-Gen Architecturewith Complex Memory Hierarchy

Web-based Interactive Accessto Supercomputer Nodes

Supported by the New Energy and Industrial Technology Development Organization (NEDO), Japan.

To achieve higher performance and larger memory capacity on
future architectures, we need to explore next-gen memory
hierarchy. It is important to understand application performance,
not only synthetic benchmark, on variety of memory hierarchy.
For this purpose, we are developing an estimation tool ofmemory performance of future architecture, whose features are:
Light weight: it takes history of the entire application execution as input, which can be too heavy for cycle-wise simulators.
To support complex structure: it considers placement ofmany cores and topology including chip-lets and memory chips.

GUI tool to describe the architecture to be estimated.
The user describes thememory hierarchy as a graphthat consists of memory objects, caches and CPU cores.

Supercomputers
are traditionally designed to executelarge and non-intercode-server
active jobs with a(VS code)batch scheduler.
This style does notmatch the users
who want to interact
with computenodes: debugging, visualization, and education of novice users inthe classroom.
In TSUBAME3.0 supercomputer, we introduced two new features
to satisfy such demands:
Interactive use only nodes: We spare four nodes as dedicated nodes for shared interactive use. Users can run interactive
jobs without waiting for job execution, even if the compute nodes
are filled with batch jobs. Performance might not be optimal as
the nodes are shared with multiple jobs, but still acceptable forsuch interactive usage.
Web-based access to Applications on compute nodes:
Novice users can use command-line shell and Python console
running on high-performance TSUBAME nodes without any complicated knowledge of Linux, such as SSH key-pair authentication. Currently, Jupyter Lab and code-server (web-based clone ofVS code) are supported, and web-based VNC server is planned.
(Planned)

User

Compute Node

3. Direct Access toJupyter on Compute Nodevia Web Portal

Web-based User Portal

Power Management Framework JST CREST Project on
Scalable Deep Learning
for Petascale Supercomputers

This work was supported by JST CREST Grant Number JPMJCR19F5, Japan.

Power consumption is expected to bea first class design constraint fordeveloping petascale supercomputers.
To make effective use of limited powerbudget, hardware over-provisioning isproposed. We are creating apower-aware resource manager.

Scheduler overview:

The Case for Strong Scaling in Deep Learning:
Training Large 3D CNNs with Hybrid Parallelism

2007515050

Idle nodes may
degrade systemthroughput

2500150300

450600Execution time (s)

750

100
50

9000

wasted(idle) power

only-low
many-low
middlemany-highonly-high250100job power

low-only
many-low
middlemany-highhigh-only

300

1.00.90.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0

low-only
many-low
middlemany-highhigh-only

Number of wai�ng jobs

low-only
many-low
middlemany-highhigh-only

125

Idle node

low-only
many-low
middlemany-highhigh-only

Running nodePower consump�on normalized
to TDP of CPUs x 400 nodes

We present a newcapability for spatiallypartitioning the trainingof 3D convolutional
neural networks. We
study the impact of
Job Scheduler for Power-Constrained HPC Systems data size and present
Objective: Providing a common resource management tool for comprehensive performance and scaling results of two different
huge 3D CNNs, the CosmoFlow network and 3D U-Net,
hardware-overprovisioned HPC systems
includingtraining
asinglemodelwithup
to2048
V100GPUs.
- Maximize throughput of a system within a given power budget
Furthermore, we demonstrate an order-of- magnitude- Provide extendibility and flexibility (plug-in based interface)
improvement in the prediction quality of the CosmoFlow network.
Plug-in based interfaces and functionalities:
A Systematic Performance Analysis ofPower Analyzer plug-in estimates job characteristics
Second-order Optimization using K-FAC
Power Allocator plug-in controls power-cap dynamically
Node Power Manager plug-in provides a way to manage theWe perform second order
power consumption within compute nodes
optimization on deep neuralPower monitor provides continuous monitoring of power
networks using Kronecker factored
approximate curvature (K-FAC).
consumption of every compute node
Various performance optimizations
Experiment: Measurement of power consumption on HA8000 HPC System (965nodes)
Power utilization
Runtime power consumption
were performed including the use of
fp16 gradients and fp21 Fishermatrices, hierarchical ring
AllReduce communication, and theuse of stale Fisher matrices. We achieved 2 minutes trainingof ResNet-50 / ImageNet-1K with 2048 GPUs by Distributed
K-FAC.
960 nodes
400 nodes 540 nodes 680 nodes 820 nodes(140 nodes (280 nodes (420nodes(560 nodes(no overoveroveroverprovisioned) overprovisioned) provisioned) provisioned) provisioned)

https://www.gsic.titech.ac.jp/sc21