About Us：Cybermedia Center, Osaka University
Cybermedia Center, Osaka University, Japan

About Us：Cybermedia Center, Osaka University

As a resource provider of knowledge and technology derived from advanced researches conducted in Osaka University,
the Cybermedia Center (CMC) offers support in the areas of large-scale computation, information communication,multimedia content and education. The center also works closely with educational and research organizations within
Osaka University, as well as with industries and institutes outside the University. By sharing its resources and encouraging local communities to use its facilities for public lectures and other events, CMC has helped to create a moreinternationally-oriented IT society for the region.

Research Divisions
Informedia Education Research Division is involved in constructing an
advanced information education environment, providing information andinformation ethics education, and conducting research and education
activities for faculty development of information education staff.
Multimedia Language Education Research Division seeks to create an
ideal environment for language education by developing an innovative, userfriendly learning management system for all language teachers/learners, andself-learning software for foreign-language learners. It also supports theoperation and maintenance of several computer-assisted language laboratories, and provides students with opportunities to optimize their learning of
foreign languages.
Large-Scale Computational Science Research Division is involved inassisting in the operation of the supercomputer system, disseminating technologies for visualizing computational results, providing education on
advanced technologies for using the supercomputer system, and conducting
education and research activities for computational science and other relatedcourses.

Vector supercomputer

Applied Information Systems Research Division conducts education andresearch into system architecture and operating technology involving
large-scale data, to assist in the operation of our super computer and cloudsystems, and to support users. It also performs research and education in thevisualization of large-scale data and the architecture of cyber-physical systems.
University-wide Information and Communications Infrastructure
Services Promotion Division is involved in promoting and managing thesmooth execution and enhancement of university-wide support services which
the Cybermedia Center is implementing, such as the maintenance, operation,and user-support of information communication systems installed foreducation, research, and clerical work.

Vector supercomputer

Computer Assisted Science Research Division supports efficient computer
applications and education (relevant also to supercomputers) aimed at identifying and solving scientific problems.It also conducts education and research
activities in mathematical and computational modeling of scientific problems.

Kyoto

Cybercommunity Research Division is involved in the design of digital
libraries, cyber communities, and social networks, building information modeling (BIM), development of risk management systems for urban areas, andevaluation of urban infrastructure, while providing computer-aided design
and graphic science education.

Tokyo

Advanced Network Environment Research Division supports the operation and utilization of the Osaka Daigaku Information Network System(ODINS), which introduces novel networking technologies such as high-speednetworks, and mobile networking environments, with lower energy consumption. It also conducts educational activities on networking technologies,
security issues, information ethics, etc., for university students and staff. In
addition, it conducts state-of-the-art research on network-related topics.

CMC Osaka U
Location

Visit us at www.cmc.osaka-u.ac.jp/

New Supercomputer System SX-ACE at the Cybermedia Center
Cybermedia Center, Osaka University, Japan

New Supercomputer System SX-ACE at the Cybermedia Center

System Overview
The SX-ACE, newly introduced by the Cybermedia Center (CMC) isa “clusterized” vector-typed supercomputer, composed of 3 clusters,
each of which is composed of 512 nodes. Each node is equippedwith 4-core multi-core CPU and a 64 GB main memory. These 512
nodes are interconnected on a dedicated and specialized network
switch, called IXS (Internode Crossbar Switch) and forms a cluster.
Note that IXS interconnects 512 nodes with a single lane of 2-layer
fat-tree structure and as a result exhibits 4 GB/s for each direction ofinput and output between nodes. In the CMC, 2 Peta-byte storageis managed on NEC Scalable Technology File System (ScateFS), NECdeveloped fast distributed and parallel file system, so that it can beaccessed from the large-scale computing systems including theSX-ACE at the CMC

Node Performance
As a single SX-ACE node has a multi-core vector-typed processor composedof 4 cores, each of which exhibits 64 GFlops vector performance, and a 64
GB main memory, the vector performance per node becomes 256 GFlops.
On the other hand, the maximum transfer between processor and mainmemory is performed with 256 GB/s. This fact indicates that a single SX-ACE
node achieves high memory-bandwidth performance of 1 Byte/Flops, taking
higher CPU performance into consideration.
Internode communication enables 4GB/s x 2 (bi-directional) high-bandwidth
data communication with a specialized internode communication control
unit named RCU connected to IXS. Also, communication between node anddisk storage enables 10 Gbps data communication with I/O control unit
called IOP.

IXS

Disk
4 GB/s x2（bidirectional）

10 GbpsIOP

IOPCPUcoreRCUIOPRCURCUIOPRCUCPUCPUCPUCPU
core CPU
core CPU
core CPU
IOP
core CPU
core CPU
core CPU
core CPUCPU
core
core CPU
core CPU
core CPU
256GB/score
core
core
256 GB/s core
CPUCPU
256 GB/s CPU
core
core
core
256 GB/s
X-bar switch
X-bar switchi
X-bar switchi
X-bar switchi

RCUCPUcore

256 GB/s

X-bar switchi
Main memory
Main memory
Main memory
Main memory
Main memory

System Performance
SX-ACE
Per-node

1 cluster

3 cluster# of CPU

1512

1536

# of core

42048

6144

Performance

276 GFLOPS

141 TFLOPS

423 TFLOPS

Vector performance

256 GFLOPS

131 TFLOPS

393 TFLOPSMain memory

64 GB

32 TB

96 TB

Storage

2 PB

The SX-ACE which the CMC has introduced is composed of 3 clusters (1536 nodes). Theoretical peak performance of the SX-ACE at
the CMC is derived as the left table indicates.
Importantly, note that performance is the sum of vector-typed processor and scalar processor on SX-ACE. SX-ACE has a 4-core
multi-core vector-typed processor and a single scalar processor.

IT Core Annex
IT Core Annex is a new datacenter that aims to aggregate andaccommodate computer systems and supercomputer systems oncampus for energy-efficient administration and management. It wasdesigned and built based on the careful consideration on air flow
and circulation for efficient cooling. The SX-ACE at the CMC is set
up at the IT Core Annex.
A remarkable feature of IT Core Annex has two indirect evaporative
air cooling systems to improve PUE for more energy-efficient drivingof the newly introduced supercomputer system SX-ACE at the CMC.

Contact : system@cmc.osaka-u.ac.jp
http://www.hpc.cmc.osaka-u.ac.jp/

An MPI Framework Enhanced for SDN Architecture Cluster
Cybermedia Center, Osaka University, JapanAn MPI framework Enhanced for SDN Architecture Cluster

Message Passing Interface
Message Passing Interface (MPI) has been a de facto standard programming model in parallel
computing for around two decades. MPI offers two types of communications:
1. One-to-one communication: used for low-level description of communication pattern.
2. Collective communication: used for high-level and human-friendly description.

Software-Defined Networking
1. Centralized controller

Network Controller
2. Programmable

3. Dynamically controllable

Software-Defined Networking
Software-Defined Networking (SDN) is a concept of network architecture that decouples conventional networking function into a programmable control plane (network controller) and a data
plane (physical network).

Problem
However, collective communication of MPI often suffers from performance degradation. One ofthe main causes is that most of the MPI implementations are designed on the assumption that:
1. MPI programs are not able to acquire network topology information.
2. Network resources are statically allocated.

Physical Network

Conventional parallel computing’s
communication method

Research Goal
Integrate the dynamic controlling ability of Software-Defined Networking into MPI in order tooptimize collective communication by overturning the assumption that network is a static resource.
In the future, we’d like to implement a new MPI library which cuts down communication latency
and traffic amount by programmatically controlling the underlying network.

Fast MPI_Bcast Design Enhanced with SDN

Efficient Bandwidth Utilization of Clusters

MPI_Bcast in Conventional Methods

Link Contentions in Conventional Clusters

MPI_Bcast uses combination of multiple
one-to-one communciations.
Intra- and Inter-node communications
does not overlap with each others well.

321

In conventional computerclusters without full bisection bandwidthinterconnect, link congestion
could happen due to thedeviation of packet flow.

Link Contention
SwitchComputing Node

Node 1SDN MPI_Bcast Method

Node 2

Node 3

Node 4

SDN_MPI_Bcast uses SDN-enabled switches as packet copier.

Cooperation of MPI Library and SDN Controller
Stage 1 – Broadcast Data
Duplication
Duplication

Controller

Execution time ofSDN_MPI_Bcast

Duplication

1

Stage 2 – Assure Reliability
MPI processors send data to “next”
processor using one-to-one communication.
SDN Controller calculates “next”
processor considering network topology and physical location of processes.
Intra-node communications occurduring Stage 2.

Controller[1] Khureltulga Dashdavaa, Susumu Date, Hiroaki Yamanaka, Eiji Kawai, Yasuhiro Watashiba,Kohei Ichikawa, Hirotake Abe and Shinji Shimojo. Architecture of a High-speed MPI_Bcast
Leveraging Software-Defined Network. In UCHPC2013: The 6th Workshop on UnConventional
High Performance Computing 2013, Archen, Germany, Aug. 2013.

We propose a new computer cluster architecture
consisting of a customized
MPI library, SDN controller
and LLDP daemon. Thissystem dynamically reconfigures the underlying network
depending on the MPI communication request so thatlink contention will nothappen.
An experiment conducted
on a computer cluster withfat-tree interconnect
revealed that our method
increased the performanceof MPI_Allreduce for 36 %at maximum.

(3) Network
preparation
requestSDN
Controlelr(1) Topologydiscovery(4) Routecalculation
SDN-enabled Switch(5) Routeinstallation

Computing Node(2) MPI_Allreduce
invoked[2] Keichi Takahashi, Dashdavaa Khureltulga, Yasuhiro Watashiba, Yoshiyuki Kido, Susumu
Date, Shinji Shimojo, "Performance Evaluation of SDN-enhanced MPI_Allreduce on a Cluster
System with Fat-tree Interconnect", The International Conference on High Performance
Computing and Simulations (HPCS2014), Bologna, Italy, Jul. 2014.

Contact : Khureltulga Dashdavaa huchka@ais.cmc.osaka-u.ac.jp
Keichi Takahashi takahashi.keichi@ais.cmc.osaka-u.ac.jp

Toward efficient and flexible resources provisioning
on SDN-enhaned Job Management System Framework
Cybermedia Center, Osaka University, Japan

Nowadays, usage patterns of users' computations on high-performance cluster system have been diversified for large-scale simulations and analyses in the various sciencefields. Since an HPC cluster system needs to accommodate multiple jobs concurrently, efficient and flexible resource management is essential for providing high performancecomputing capabilities for multiple users. However, most Job Management Systems (JMSs) available today, which are deployed on HPC cluster system for computational
workload distribution and balancing purposes, determine resource allocation to each jobs only based on computational resources such as CPU and memory. In thisresearch, we realize a novel JMS framework for handling various kinds of resources such as network resources and virtualized computational resources.

Proposed SDN-enhanced Job Management System Framework
We have been studying and developing a novel network-aware JMS integrated Software-Defined Networking (SDN) concept, which can dynamically control an entire
network in a centralized manner, into a traditional JMS [1]. The mechanisms to manage the network resources are designed and implemented as Network Management
module (NMM) leveraging OpenFlow, which is an implementation of the SDN concept. The SDN-enhanced JMS can allocate both computational and network resources
to each job according to the resource usage on a cluster system and Resource Assignment Policy defined by administrator.
Moreover, we have also been developing a mechanism for deploying job's processes to virtual machines (VMs) on computing nodes, and guaranteeing available bandwidth on communication paths allocated to a job by using QoS functions of Open vSwitches (OVSs) connected with VMs.

SDN-enhanced JMS Framework

NMM configures Flow Entries on OpenFlow switches
for assigning communication paths to the job.
OpenFlow network

Network Management Module(NMM)
Administrator can create resource assignment
policy for computational and network
programmablly through a ruby script.

ResourceAssignment
PolicyDatabase

Resource assignment
policy class module

Administrator#!/bin/csh#$ -q QUEUE#$ -pe orte 4#$ -l netprio=bandwidth#$ -l qos=1000#$ -l vm=my-vm

0.6/1.0NetworkControlJobscript

0.5/1.0

1.0/1.0

0.9/1.0

OpenFlow
Controller(Trema)

Brain

1.0/1.0

0.8/1.0

Flowentry

Job

Job

Brain decides computational andnetwork resource to allocate to
the job according to resource
assignment policy.

Job

Job

Job

Job

Job

Job

Job

Job

Job

Job

Job

Job

Open vSwitch
VMJobJob

Queueqmaster
JMS(OGS/GE)… run program …

“wrapper” configures parameters
for QoS on Open vSwitch, andboots up the VM with information
to execute the job.

shepherd

VM
imagesUserwrapper

User can request resource assignment policy,
bandwidth and VM image in a job script.

execd“execd” calls “wrapper”
instead of “shepherd”.

Computing node

Evaluation
To assess the efficiency of explicitly allocating communication paths to each jobs
by the SDN-enhanced JMS, we conducted an experiment to compare the average
execution time of jobs on a fat-tree cluster system. Note that processes of jobs
were allocated to real computational resources and a single computing nodeaccommodated only one process. In the experiment, we submitted multiple jobs,
each of which ran NAS Parallel Benchmarks with class B and generated 4 processes. As a result, our proposed SDN-enhanced JMS succeeded to reduce theaverage execution time of jobs.

SDN-enhanced JMS

1008060

SW2SW1OGS/GE

120Time (sec)

Toward efficient and flexible resources provisioning on SDN-enhaned Job Management System Framework

Motivation and Objectives

40SW3SW4SW5SW620

02-tier fat-tree cluster system:
The number of computing nodes is 28 and every link is connected on a Gigabit Ethernet LAN.

BTCGEPFTISLUMGSP

Average execution time of NAS Parallel Benchmarks

Acknowledgments
This research was supported in part by the collaborative research of the National Institute of Information and Communications Technology (NICT) and Osaka University (Research on High Functional Network Platform Technology forLarge-scale Distributed Computing).

[1] Y. Watashiba, Y. Kido, S. Date, H. Abe, K. Ichikawa, H. Yamanaka, E. Kawai, H. Takemura, "Prototyping and Evaluation of a Network-aware Job Management System on a Cluster System Leveraging
OpenFlow", The 19th IEEE International Conference On Networks (ICON 2013), Dec. 2013.

Contact : Yasuhiro Watashiba, E-mail : watashiba-y@cmc.osaka-u.ac.jp
Web : http://hpc-sdn.ime.cmc.osaka-u.ac.jp/sdnjms/