Osaka University, Japan
SC22 BOOTH 1613

About Us：Cybermedia Center, Osaka University
As a resource provider of knowledge and technology derived from advanced researches conducted in Osaka
University, the Cybermedia Center (CMC) offers support in the areas of large-scale computation, information communication, multimedia content and education. The center also works closely with educational and research organizations within Osaka University, as well as with industries and institutes outside the University. By sharing its
resources and encouraging local communities to use its facilities for public lectures and other events, CMC has
helped to create a more internationally-oriented IT society for the region.
Repair and Maintenance of the Information Network, ahigh-speed, stable and reliable campus-wide network environment,
as well as wireless access networks, as information infrastructure forsupporting the educational, research, and social contribution activities
of Osaka University.

Location Map

Kyoto
TokyoOsaka★

Location

Academic Cloud improves the integration of computing resources
scattered across the university. The objectives of the system are tooptimize administrative operations, enhance security, and reduce
costs.
IT Core Annex is a two-story steel-frame data center housinglarge-scale computers. The perimeter wall is designed with gently
curving surface and light-permeable metal panels, to harmonize withthe surrounding environment.
Open-type cooling towerCap

Exhaust vent celling

University-Wide Services
Large-Scale Computer System, we provide a high-performancecomputing environment, consisting of OCTOPUS and SQUID, to boththe academic and industrial communities. Part of the overall computer
system is provided, as a computational resource, to the national
High-Performance Computing Infrastructure(HPCI).
Information Media Education Multimedia Language Education,
we have implemented a consistent curriculum, from the basics ofcomputer utilization to advanced subject matter, while the Computer
Assisted Language Learning System supports foreign language
learning and cross-cultural understanding in accordance with eachindividual’s language-proficiency level.

Outerwall

Air passage
Hot aisle
Inner wall
Cold aisle
Server Rack

Second
floor

First
floor
Air-conditioning chamber

Cybermedia Commons is an active learning space for students,
exploiting a wide variety of the Cybermedia Center’s information
technology, to support student’s active learning and research
activities.

Cybermedia Commons

Digital Library provides academic information databases and remote
access to electronic journals. It is equipped with multimedia terminals
and public network jacks with an authentication system.

Visit us at www.cmc.osaka-u.ac.jp

chilled-water
air-conditioner

Centrifugal chiller

Cooling mechanism in IT Core Annex

Osaka University, Japan
SC22 BOOTH 1613

Large-scale Computing Systems at the Cybermedia Center
Overview of High-Performance Computing Environment at the CMC
USER

Supercomputer
Network

Login & Front End
NodeInternet

Cloud

Large-scale computing systems ( OCTOPUS and SQUID ) and data aggregate
infrastructure ( ONION ) are deployed on CMC-Supercomputer network, a.k.a
CMC-SCinet, a low-latency and wide-bandwidth network. This architectural design
allows users to access to large-scale storage systems, perform large-scale
high-performance computation and analysis on our large-scale computingsystems.

Large-scale Computing System

OCTOPUSSQUID

OCTOPUS is short for Osaka university Cybermedia cenTer
Over-Petascale Universal Supercomputer. OCTOPUS is a cluster
system being operated since December 2017. This system iscomposed of General purpose CPU nodes, GPU nodes, Large-scale
shared-memory nodes, and Xeon Phi nodes, total 319 nodes.
These nodes and large-scale storage EXAScaler ( Lustre 3.1PB ) are interconnected on InfiniBand EDR ( 100 Gbps ) and forma cluster.

SQUID is short for Supercomputer for Quest to UnsolvedInterdisciplinary Datascience. SQUID is a new cluster systembeing operated since May 2021. This system is composed ofGeneral purpose CPU nodes, GPU nodes, and Vector nodes,total 1,598 nodes. These nodes and large-scale storage EXAScaler ( Lustre 21.2 PB ) are interconnected on InfiniBand HDR( 200 Gbps ) and form a cluster.
Table 2 Data Sheet of SQUID

Table 1 Data Sheet of OCTOPUSGPU

Large–scale
shared-memory

Xeon PhiIntel Xeon Skylake( 2.6 GHz, 12 cores ) x 2

Intel Xeon Skylake( 2.0 GHz, 16 cores ) x 8Intel Xeon Phi KNL( 1.3 GHz, 64 cores )General purpose
CPUType of nodes
CPUType of nodes
CPU

RHEL 7.3

OS

General purpose
CPUGPUVector

Intel Xeon Icelake ( 2.4 GHz, 38 cores ) x 2

AMD EPYC Rome
( 2.8 GHz, 24 cores )CentOS 8.4

OS# of nodes (total)

236

37244

# of nodes (total)

1,520

4236

# of cores (total)

5,664

888

256

2,816

# of cores (total)

115,520

3,192

864

# of memory (total)

457128# of memory (total)

389 TB

22 TB

5 TB

Peak performance

471.2 TFLOPS

16.4 TFLOPS

858.3 TFLOPS

117.1 TFLOPS

Peak performance

8.871 PFLOPS

6.797 PFLOPS

0.922 PFLOPSNVIDIA
A100 x 336

NEC SX-Aurora TSUBASA
Type20A x 288

NVIDIA Tesla
P100 x 148

Accelerator

Accelerator

Data Aggregation Infrastructure

ONION
Primary storage

Primary storage

Externalstorage

Cloud

S3
S3

S3SFTP
S3DAV

S3Cameras /Sensors

Cameras /Sensors

S3Lustre

EXAScaler

objectS3For archives

S3SFTP
S3For high-speed
analysis

SCP/SFTPSQUID users
HTTPS
For data manipulation

HTTPS

HTTPS

WWWPCBrowserMobilefileONION stands for Osaka university Next-generation Infrastructure for Open researchand open innovatioN. ONION is a new data aggregation infrastructure that is linked to
SQUID. ONION consists of ONION-object ( AWS S3 compatible object storage ) ,ONION-file ( storage service using Nextcloud ) , and EXAScaler ( a parallel file system
based on Lustre ) .
ONION makes it easy for users to data between your PC and large-scale computing
system. In addition, ONION can be used in a variety of ways, such as immediate sharingof calculation results with those who do not have a SQUID or OCTOPUS account andmanipulating data from a smartphone. Of course, it can also be used to store and shareresearch data in the laboratory.
Table 3 EXAScaler ( on SQUID )Table 4 ONION-object

Effective capacity (HDD)

20 PB

Effective capacity (NVMe)

1.2 PBMax number of inodes

Approx. 8.8 Billion

Max expected effective throughput (HDD)

Over 160 GB/s

Max expected effective throughput (NVMe)

Write : Over 160 GB/s Read : Over 180 GB/s

Contact : sc22@ais.cmc.osaka-u.ac.jp

Effective capacity

Data protection method

950 TiB* We plan toexpanse sequentially
Erasure Coding(Data chunk:4 +Parity chunk:2)

http://www.hpc.cmc.osaka-u.ac.jp/en/

Osaka University, Japan
SC22 BOOTH 1613

AI assisted job scheduler / Profile guided vector optimization
AI assisted job scheduler: Cloud Burst Optimization with Deep Q Network
BackgroundResults

・Cloud bursting becomes attractive for HPC systems to prevent anincrease of job waiting time under high load.
・However, it is still difficult to control the tradeoff between job waitingtime and cloud cost.

・Proposed architecture can control tradeoff between waiting time andcloud costs
Waiting time

Proposal

0.35

3.0Architecture
・Job scheduler provides state of job queue and on-premise and cloud
resources to DQN when scheduling a job
・DQN returns action that shows the scheduler should assign
on-premise or cloud resources to the job or skip scheduling・Job scheduler schedules the job based on the action
・Job scheduler provides reward to DQN for evaluating the action
based on a waiting time and a cloud cost
Job queue

Job scheduler

Backlog part

Observation part

DQN AgentState Reward( m w, m c ) = (0 . 5 , 0 . 5 )( m w, m c ) = (0 . 4 , 0 . 6 )( m w, m c ) = (0 . 3 , 0 . 7 )( m w, m c ) = (0 . 2 , 0 . 8 )

0.30

2.5

0.25

2.0costwaiting time・Job scheduler with DQN (Deep Q Network) that can optimize thetradeoff of cloud bursting

1.5

0.15

1.0

0.10

0.5
0.0

0.20

0.05
0.00dqndqn

Future Work・Evaluation and implementation of the proposed method into theSLURM scheduler
DQN Agent

SLURM

Input (State):・Job queue
・On-premise resource
・Cloud resource

state

SchedulingthreadOn-premise resource

inferenceaction
Nodes

Node SelectPlugin with
DQN I/F

Available
Cloud resource

Cost
0.40

3.5

Time

Allocated

ActionrewardOutput:Q-values of actions

Model
updateTraining(async)

Profile guided vector optimization for SX-Aurora TSUBASAResults

・Realization of vector optimization by users without HW knowledge
Proposal・Automatic source-to-source translation tool by Profile Guided Vector
Optimization (PGVO) for SX-Aurora TSUBASAArchitecture
・PGVO uses source codes, compiler analysis results and execution
profile data as inputs
・PGVO outputs translated source codes
Source code
do i = 1..100a[ i]=a[ i]+1

Compiler analysis result

Exec. profile data

Loop unvectorized
Dependency unknow

Vector op ratio
Ave. vector length

・Significant performance improvement by PGVO compared toautomatic vectorization compiler with 3/8 workloads* of NPB(* Human-optimized codes that assumes tool behavior are evaluated)

Normalized Performance (VE=1)

Background・NPB 3.4.2 (omp version)
・VE: SX-Aurora 20B
1VE(8core) + ncc/nfort□ EP/SP/BT: PGO achieves great improvement□ CG/MG: Compiler already achieves good performance. No room for PGVO.

□ FT/IS/LU: Some room for optimization, but current PGVO achieves little or no improvement.

Future Work

optimized source code

Profile guided vector
optimizer

Inline expansion
Loop collapse
Outer-loop vectorization…

・Implement as a tool and confirm the feasibility
・Evaluate with more workloads
・Support more vectorization technologies

These works were carried out in Joint Research Laboratory for Integrated Infrastructure of High Performance Computing and Data Analysis
https://www.nri.cmc.osaka-u.ac.jp/

Contact : sc22@ais.cmc.osaka-u.ac.jp

https://www.cmc.osaka-u.ac.jp/

Osaka University, Japan
SC22 BOOTH 1613

Provenance Recording System for Research Data Management
Background

Requirements for Provenance Recording System・Growing importance of research data management (RDM)
・To ensure reproducibility (transparency): Preserving data thatprovide evidence of research results
・To improve reusability: Promoting the global sharing of knowledge
and increasing research efficiency
・Provenance, which identifies the input data and the process used toobtain data, should be secured for reproducibility and reusability

・Automatically record the provenance and the metadata (date/usercreated, etc.) of a file generated in a HPC system
・Support a typical HPC environment: workload manager (Slurm),
MPI, etc.
・Minimalize impacts on performance and user’s operations
・Secure the records not to be falsified
・Provide interfaces to verify that a file has not been fabricated/falsified
Computing Center

・HPC systems generate data through simulations and experiments,
but there is no established method to manage the provenance ofthe data

OperatePublishProvenanceCollect・A system that implements RDM on HPC systems is needed

VerifyResearcherdb.dat

convertindb.dat

raw.datFile

Gather data fromequipment

Experimentin a HPC system

Author a paper

Publish the paper
and the data

Discussion
on the data

Review

Reuse

out
simulation

result.csv

args: -k 1

date: 2022-09-09
user: u1
digest: 0xabcd...

Process

Collaborator

Example of a provenance

Reviewer

AnotherResearcher

Use cases

Prototype
1 Tracer captures system call invocations (exec(), open(), write(), etc.) ofProvenance Recording System

Database

4Aggregator

24View / Verify

ReseacherUse

a user program. BPF, a low overhead observability scheme in Linuxkernel is used for the capture. Tracer also captures metadata (datecreated, SHA-256, etc.). No modifications in the user program andoperations are required.

3View / Verify

RecordTracer1

Collaborator,Reviewer, etc.

2 Aggregator builds a provenance of files from the history of the systemcall invocation: a file read/written by a process is an input/output of theprocess in the provenance. Parallel processes by MPI are aggregated.

3 The provenance and the metadata are stored in Apache Atlas (anopen-source data catalog).

4 Find and verify the provenance and the metadata of a file (shownbelow).

HPC System

Show the provenance of a file

Show the metadata of a file

This work was carried out in Joint Research Laboratory for Integrated Infrastructure of High Performance Computing and Data Analysis
https://www.nri.cmc.osaka-u.ac.jp/

Contact : sc22@ais.cmc.osaka-u.ac.jp

https://www.cmc.osaka-u.ac.jp/

Osaka University, Japan
SC22 BOOTH 1613

ns-3-based Interconnect Simulator for Network Simulation with Job Scheduling
Background : Aim of Interconnect Design in Supercomputing Systems is Changing
A variety of jobs are performed on today's supercomputing systems. The number of compute nodes requested by such jobs is diverse and thenmuch inter-node communication take place.
⇨ Interconnects of supercomputing systems should be designed using simulators to examine the performance in communication.

Traditional Supercomputing Systems

Next Supercomputing System

Expected Workload

Computation-intensive MPI (Message
Passing Interface) jobs.

Expected Workload

Communication-intensive jobs using
distributed processing frameworks.

Aim of Interconnect
DesignFocus on the cost to increase the numberof compute nodes.

Aim of Interconnect
DesignFocus on the performance to accelerateinter-node communication.

Method of Interconnect
Design・Select from stable and mature technologies
such as Fat-tree and ECMP.
・Parameters are determined by calculations
and other simple estimates.

Method of Interconnect
Design・Select from stable and mature technologies
and/or state-of-the-art technologies such
as DragonFly and adaptive routing.
・Parameters are determined by simulations
to examine interconnect performance.

Problem: The Effects of Job Scheduling Are Missed by Existing Network Simulators
When simulating interconnects in a supercomputing system, the simulation result is incorrect in the case of using only existing network simulators.
The reason is existing network simulators cannot reproduce job placement by job schedulers.
・Adversarial traffic patterns** cause misunderstanding of thenetwork performance. (**Traffic patterns that degrade networkperformance)

・Traffic patterns* are changed by job placement.
(*A set of communications within a certain time period)

Bandwidth consumption increases
when communicating between
distant nodes⇨ Without simulating job placement
accurately, bandwidth consumption
is not simulated accurately.

job1

Adversarial traffic patterns cause bottlenecks,
resulting in simulation results worse.
⇨ Job placement also affects simulation
results because of increasing adversarial trafficpatterns.

job1

job2

job2

Example of link capacity consumption depending on job placement

Adversarial traffic patterns in DragonFly topology [1][1] P. Fuentes, E. Vallejo, C. Camarero, R. Beivide and M. Valero, "Throughput Unfairness in Dragonfly Networks under Realistic Traffic Patterns," 2015 IEEE International Conference on Cluster Computing, 2015, pp. 801-808, doi: 10.1109/CLUSTER.2015.136.

Proposal : ns-3-based Interconnect Simulator for Interconnect Design (In-Progress)
To achieve network simulation with job scheduling, we decided to implement a job scheduling function as a module for ns-3.

・Assets: Interconnect research results are implemented in ns-3, enabling simulations using state-of-the-art technologies.
・Expandability: ns-3 is modularized, making it easy to expand the job scheduling functionalities.
・Packet-level simulation: accurate simulation of network latency should reduce performance estimation errors.

Parameter file

・Topology
・Routing algorithm
・TCP congestion control algorithm
・Job scheduling algorithm
・Number of request nodes
・Contents of communication(prepared communication patternor pcap file) and computation
time (communication interval)

Job scripts

User Input

Step 0

Setup topology as input files

Step 1

According to the job scheduling algorithm,
the job scheduler dynamically places traffic
generators using the ns3 Schedule function

Step 2Present simulation results including job runtime andbottleneck links/switches

By repeating Step 0 ~ 2 with different inputs,
interconnect design based on quantitative comparisons will be achieved.

Contact : sc22@ais.cmc.osaka-u.ac.jp

https://www.cmc.osaka-u.ac.jp/

Osaka University, Japan
SC22 BOOTH 1613

Toward a Practical Cloud Bursting Operation on SQUID
BackgroundJobThe on-premise supercomputing systems in CMC are sometimes faced with a surge ofthe computing demand. This situation causes a longer wait time from when a user
submits a job until when the job starts. In order to alleviate the peak, we have built
SQUID as our new on-premise supercomputing system with the idea of offloading theworkloads on an on-premise supercomputing system to cloud computing resources.
This idea is referred to as cloud bursting.

JobJob

Job

Job

CloudJob

Job

Job

Offloading
the workload

Fig. 1 Overview of cloud bursting.

Environment
The cloud bursting environment on SQUID allows the users to execute their jobs without their being aware of Azure. The following two ideas
achieve transparency in terms of usage.
• Cloud bursting queue which dynamically allocates their jobs to SQUID or Azure.
• CPU computing nodes on SQUID and Azure access Lustre on SQUID with NFS.
However, the cloud bursting environment discourages the users from executing their jobs on Azure. This reason is they have to be aware of Azurein terms of performance and monetary cost.
SQUID

CPU computing nodes

Node
Node

Node
Node

Cloud bursting queue

Microsoft Azure

the Internet
NFS

NFS

LustreCPU computing nodes

IPsec VPN

Scheduler

Node
Node
Node

Node

Job allocation

Cloud-dedicated queue

Job allocation

Fig. 2 The cloud bursting environment on SQUID.

Operation view
SQUID

Microsoft Azure
NFSCPUIPsec VPN

LustreInputdata

Step 1Step 2Local storage output
dataStep 3

NodeFig. 3 The staging functionality.

• The two queues on SQUID
1. Cloud bursting queue : Transparency in terms of usage, low cost for the users
2. Cloud-dedicated queue : Immediate provision of computing resources, high costfor the users.
To improve transparency in terms of performance and cost, we propose threeoperations on SQUID which combine the staging functionality and the two queues.
Policy 1. Staging functionality unable + Cloud bursting queue
Policy 2. Staging functionality enable + Cloud-dedicated queue
Policy 3. Improved staging functionality enable + Cloud bursting queue (unfinishedimplementation)

Step 1The users manually copy input data to a local storage
before executing their jobs. (stage-in)

Step 2

The jobs access the input/output data on the local storage.

Policy 1

easy

low

low

difficulthighhigh

Step 3The users manually copy the output data to Luster after
executing the jobs. (stage-out)

Policy 2
Policy 3

easy

high

low

Transparency

Usage

Performance

CostTab. 1 The three operation policy.

Performance profileFig. 4 The I/O time with the staging
functionality unable and enable (BT-IO).

Fig. 5 The I/O time and the stage-out timewith the staging functionality enable (BT-IO).

Contact : sc22@ais.cmc.osaka-u.ac.jp

Fig. 6 The cost with the staging
functionality unable and enable (BT-IO).

https://www.cmc.osaka-u.ac.jp/