Osaka University, Japan
SC16 BOOTH 1437

About Us：Cybermedia Center, Osaka University
As a resource provider of knowledge and technology derived from advanced researches conducted in Osaka
University, the Cybermedia Center (CMC) offers support in the areas of large-scale computation, information
communication, multimedia content and education. The center also works closely with educational andresearch organizations within Osaka University, as well as with industries and institutes outside the University.
By sharing its resources and encouraging local communities to use its facilities for public lectures and otherevents, CMC has helped to create a more internationally-oriented IT society for the region.
R e p a i r a n d M a i n t e n a n c e o f t h e I n f o r m a t i o n N e t w o r k, ahigh-speed, stable and reliable campus-wide network environment,
as well as wireless access networks, as information infrastructure forsuppor ting the educational, research, and social contribution
activities of Osaka University.

Location Map

Kyoto
Tokyo

Osaka

★

Visualization Services, we maintain two types of high-resolutionstereo visualization systems, as primary visualization facilities. The
systems can be used for scientific visualization, information
visualization, visual analytics, and other research activities.

Location

University-Wide Services
Large-Scale Computer System, we provide a high-performancec o m p u t i n g e n v i r o n m e n t , c o n s i s t i n g o f t h e N E C S X- A C E
supercomputer and PC clusters, to both the academic and industrial
communities. Part of the overall computer system is provided, as ac o m p ut at i o na l res o ur c e, to t h e nat i o na l H i gh - Per fo r ma n c e
Computing Infrastructure(HPCI).
Information Media Education Multimedia Language Education,
we have implemented a consistent curriculum, from the basics ofcomputer utilization to advanced subject matter, while the Computer
Assisted Language Learning System supports foreign language
learning and cross-cultural understanding in accordance with eachindividual’s language-proficiency level.
Cybermedia Commons is an active learning space for students,
exploiting a wide variety of the Cybermedia Center’s information
technology, to suppor t student’s active learning and research
activities.

Visualization Services

Academic Cloud improves the integration of computing resources
scattered across the university. The objectives of the system are to
optimize administrative operations, enhance security, and reduce
costs.
IT Core Annex is a two-story steel-frame data center housinglarge-scale computers. The perimeter wall is designed with gently
curving surface and light-permeable metal panels, to harmonize withthe surrounding environment.

Cybermedia Commons

Digital Library provides academic information databases andremote access to electronic journals. It is equipped with multimedia
terminals and public network jacks with an authentication system.

Air-conditioning mechanism in IT Core Annex

Contact : system@cmc.osaka-u.ac.jp Visit us at www.cmc.osaka-u.ac.jp

Osaka University, Japan
SC16 BOOTH 1437

Large-scale Computing and Visualization Systems at the Cybermedia Center
Overview of high-performance computing environment
at the CMC
L arge - s c ale c omputing systems (SX-ACE, VCC, and HCC), and large - s c alevisualizati on systems are depl oyed on CM C - Sup erc omputer net work, a .k.a
CMC-SCinet, a low-latency and wide-bandwidth network. This architectural design
allows users to ac c ess to large -scale storage systems, per form large -scale
high-performance computation and analysis on our large-scale computing systems,
and then visualize its computation and analysis results without loosing any important
information on our large-scale visualization system.

Large-scale Computing System
The large-scale computing systems at the CMC are classified into (1) Vector-typed Supercomputer and (2) Scalar-typed Supercomputer.

SX-ACE

VCC (PC Cluster for large-scale visualization)
Type: Vector
OS: Super UX

Type: Scalar
OS: Linux# of nodes: 1536
# of cores: 6144
Total memory: 96 TB
Peak performance: 423 TFlops

SX-ACE is a “clusterized” vector-t yped
supercomputer, composed of 3 cluster,each of which is composed of 512 nodes.
Each node equips 4-core multi-core CPU
and a 6 4 GB main memor y. These 512
nodes are interconnected on a dedicatedand specialized network switch, called IXS(Internode Crossbar Switch) and forms acluster. Note that IXS interconnects 512
nodes with a single lane of 2-layer fat-tree
structure and as a result exhibits 4 GB/s foreach direction of input and output between
nodes.
Library
MathKeisan(BLAS, LAPACK, etc)
ASL, ASLSTAT, ASLQUAD# of nodes: 65# of cores: 1300
Total memory: 4.160 TB
Peak performance: 26.0 TFlops
Accelerator: NVIDIA Tesla K20×59

PC cluster for large -scale visualization(VCC) is a cluster system composed of 65
nodes. Each node has 2 Intel Xeon
E5-2670v2 processors and a 64 GB main
memory. These 65 nodes are interconnected on InfiniBand FDR and forms a cluster.
Also, this system has introduced ExpEther, asystem hardware virtualization technology.
Each node can be connected with extension
I/O nodes with which GPU resource, and
SSD on 20Gbps ExpEther network. A major
characteristic is that this cluster system isreconfigured based on user’s usage andpurpose by changing the combination ofnode and extension I/O node.

MPI/SX

Library

HPF/SXIntel MKL(BLAS, LAPACK, etc)

XMPIntel MPI, Open MPI

HCC (General-Purpose PC Cluster)

Type: Scalar (VM)
OS: Linux# of nodes: 575# of cores: 1150
Total memory: 2.6 TB
Peak performance: 16.6 TFlops

IPC-C (Image Processing PC Cluster on Campus)

Type: Scalar
OS: Windows/Linux# of nodes: 7# of cores: 84
Total memory: 448 GB
Peak performance: 1.68 TFlops
Accelerator: NVIDIA Quadro K5000×7

IPC-U (Image Processing PC Cluster on Umekita)

Type: Scalar
OS: Windows/Linux# of nodes: 6# of cores: 72
Total memory: 384 GB
Peak performance: 1.44 TFlops
Accelerator: NVIDIA Quadro K5000×6

Application
GROMACS

Gaussian09

Marc / Mentat

Nastran

GROMACS
for GPU

IDL

Dytran

AVS/Express(DEV/PCE/MPE)

LAMMPS

NEC Remote
Debugger

PatranOpenFOAM

NEC Ftrace
viewerAdams

Large-scale Visualization System
The large-scale visualization systems at the CMC are set up on Campus and on CMC’s Umekita Office. Large-scale and interactive visualization
processing becomes possible through the dedicated use of PC cluster for large-scale visualization (VCC) on these systems.

24-screen Flat Stereo
Visualization System

The visualization system is composedof 24 5 0 - inc h Full HD (1920 x10 8 0)
stereo projection module (Barco
OLS-521), Image-Processing PC cluster(IPC-C) driving visualization processing
on 24 screens. A notable feature of thisvisualization system is that it enables
approximately 50 million high-definitions te r e o d i s p l ay w i t h h o r izo nt a l 15 0
degree view angle.

15-screen Cylindrical Stereo
Visualization System

This visualization system is composedof 15 46-inch WXGA (1366x768) LCD,an d Image - Proc es sing PC Cluster(IPC-U) driving visualization processing
on 15 screens. A notable characteristicof this visualization system is that it
enables approximately 16-million-pixel
very high-definition stereo display.

Contact : system@cmc.osaka-u.ac.jp Visit us at www.cmc.osaka-u.ac.jp

Osaka University, Japan
SC16 BOOTH 1437

SDN-enhanced MPI: Towards Dynamic and Application-aware Interconnect Architecture

1. Software-Defined Networking (SDN)
Software-Defined Networking (SDN) is a new concept of networkarchitecture that decouples conventional networking function into a
programmable control plane (responsible for deciding how to control
the packets) and a data plane (responsible for the actual packet
delivery). Currently, OpenFlow is the most common implementationof SDN, which enables to dynamically control the for warding
functionality of network from a centralized controller.

3. SDN-enhanced MPI Communication Primitives
Controller
Duplication

Network Controller

2. Centralized Control
over the Network
1. Programmable

3. Dynamic Control

Network Device

Network Device

SDN
Controller(3) Network
preparation request(1) Topology discovery(4) Route calculation(5) Route installation

Network Device(2) MPI_Allreduce invoked

Network Device

SDN_MPI_Bcast is an SDN-enhanced version of MPI _ Bcast,which is the broadcasting funct i o n i n M PI . SD N _ M PI _ B c a s t
offloads packet duplication operations during the broadcast onto
SDN switches. As a result,
SDN_MPI_Bcast has successfully decreased the number of communications and communication
latency of MPI_Bcast.

B. SDN_MPI_Allreduce [2]SDN _ MPI _ Allreduce is an
SDN-enhanced version of MPI_
Allreduce. Since it requires multiple simultaneous communication
between nodes, congestion may
happen on an oversubscribedinterconnect. We employ areal-time traffic load balancing
method to solve this problem.

4. Coordination Mechanism of Communication
and Computation

2. Aim of SDN-enhanced MPI
We have been developing SDN-enhanced MPI based on the ideathat a mechanism that configures and controls the network of acluster system depending on the requirement of each application isessential. The key concept of SDN-enhanced MPI is to utilize theunderlying network of a computer cluster to its maximum capacity
by leveraging the flexible network controllability of SDN.

A. SDN_MPI_Bcast [1]We propose an integrated framework [3] to combine SDN-MPI
components that we have developed in our previous works. In thisframework, MPI packets are tagged with MPI-layer information which
are used by the SDN switches to determine how to control thepackets.

SDN Controller(3) Packet Control based on tags

(1) Pre-install rules

Dynamic Interconnect Reconfiguration

ed

dM

gg

ge

SDN Switch

ta
MP

g
Ta

Un

Communication
Scheduling

PI

IP
ac
ke
t

MPI Packet

MPI Application
Communication Pattern

Tagging Module
MPI Library
MPI Application(2) Packets aretagged with application level info

Interconnect Topology and Usage

Publications[1] K. Dashdavaa et al., Architecture of a High-Speed MPI_Bcast Leveraging
Software-Defined Network, in: UCHPC2013 6th Work. Unconv. High Perform.
Comput., 2014: pp. 885–894.
[2] K. Takahashi et al., Performance Evaluation of SDN-enhanced MPI_Allreduceon a Cluster System with Fat-tree Interconnect, in: Proc. Int. Conf. High
Perform. Comput. Simul. - HPCS 2014, 2014: pp. 784–792.
[3] K. Takahashi et al., Concept and Design of SDN-Enhanced MPI Framework,in: Proc. Fourth Eur. Work. Softw. Defin. Networks - EWSDN 2015, 2015: pp.
109–110.

Takahashi : takahashi.keichi@ais.cmc.osaka-u.ac.jp
Contact Keichi
Khureltulga Dashdavaa : huchka@ais.cmc.osaka-u.ac.jp

MPI Packet

MPI Library
MPI

Socket API

MPI Packet

NetworkStack

To NICMPI Packet

System Call
User MPI
Application

Tagging
ModuleUser Space

Kernel Space

Visit us at www.cmc.osaka-u.ac.jp

Osaka University, Japan
SC16 BOOTH 1437

A Proposal of Access Control Mechanism Towards IoT Era
1. Introduction: Network of IoT Era

3. Previous work: FlowSieve

In the IoT (Internet of Things) era, there is a large number of IoT
devices distributed widely throughout world.Many groups of users
share such devices.In the network that connects such IoT devices,
access control becomes an important problem for assuring
security.

2. Access control mechanism
To date, various security technologies have been proposed andimplemented. However, these technologies have targeted only computational resources. We propose an access control mechanismthat targets network resources as access-controlled resources.
We have adopted RBAC (Role Based Access Control) and SDN todevelop the mechanism. The mechanism works as shown below,
and provides user-dedicated infrastructure.

1. SDN controller authenticates every user connectedto the network.
SDN Controller
PolicyUser A’s Rolein network resources

IoT Device
PolicyUser A’s Rolein IoT device

computational resource
User A

PolicyUser A’s Role incomputational resource

FlowSieve is a preliminary implementation of the access control
mechanism.FlowSieve is implemented as an OpenFlow controller,and controls access to network resources.
FlowSieve works as follows.
1. FlowSieve works as IEEE 802.1X authentication server, andauthenticates every user connected to the network.
2. When a device sends a packet to an OpenFlow switch, the switch
handles the packet based on the rules installed in it.
3. If the switch cannot find any rule for handling the packet, theswitch sends the packet to FlowSieve.
4. FlowSieve determine how the packet should be treated by thecontroller, and installs rules for handling the packet to the switch.

roles:- name A
allowed_roles:-B- name B
allowed_roles:-A-C- name C
allowed_roles:-B

User_A

Sliced network #1
Srv_B

Srv_C

users:- name: User_A
role: A

Sliced network #2- name: Srv_B
role: B- name: Srv_C
role: C

Security Policy File

FlowSieve

Srv_B

User_A

2. Each resource has its own security policy, andassigns a role to each user depending on the policy.

Site A

Site B

Multi-site network resources

Srv_C

4. Future works
User A’s Role isn’t allowed
to access to this device
User A’s Role isn’t allowed toaccess to computational resource

User

Under the current version of FlowSieve, all devices and network
resources are controlled under one security policy file.
In the next plan, we extend FlowSieve so that each device and network resource can have its own security policy towards
multi-site network.
Source Code of FlowSieve: https://github.com/shimojo-lab/flowsieve

User A’s Role isn’t allowed
to access to this switch

3. User’s access to any resource is controlled basedon user’s role.

Contact: Takuya Yamada yamada.takuya@ais.cmc.osaka-u.ac.jp Visit us at www.cmc.osaka-u.ac.jp

Osaka University, Japan
SC16 BOOTH 1437

Hi-IaaS : High Performance Computing Infrastructure as a Service
Hi-IaaS

Back Ground

Software defined HPC platform covering applicationsof IoT/BigData and Science
Providing diversity of HPC Platform dynamically
Hardware (GPU/FPGA, Cluster) Software (SPARK, MPI)

Features

Proposal : Open/Dynamic

Today : Special/Static

Reasonable, scale up, high utility

Expensive, not fit, low utility

Special Protocol

OpenCLJob Scheduler

OpenCLSpecial Topology

GPU/OpenCL

Modify selected node list alongwith the management policy andutilization of compute / network

Database

node12.domain
node07.domain
node18.domain
node02.domain…

Job / Accelerator Managementwith UCLA and Falcon Computing Solutions

Routing nodecommunication path as a
Flow entry

NMM

Selected node list

MapReduce
YARN / Mesos(2) Network information(3) Recommended Nodes

Brain(4) Assigned Nodes(5) Flow Entry list

Job

Job

Job

Job

(6) Network Flow
ControlServerServer

RDStoreRDStoreRDStoreRDStoreMemoryMemoryMemoryMemoryInterconnect Network (ExpEther)

PCI-e NVMe Device PCI-e NVMe Device

acc

acc

Ceilometer(Telemetry)

1. Data base (KVS)
technology

・Decrease data access tothe disaggregated storage
devices by using hash database technology
・Decrease usage of memory
by making server statelessacc

Modify Ironic to control each device

・Proposal:

Server

Node Acc Manager

Heat(Orchestration)

Ironic(BM Mng.)Fabricator

abstraction

Command

ExpEther
Driver
1

2

Host

3

Host

End User

Horizon
(GUI)

Host

4

IaaS
Controller

Nova(Scheduler)

ExpEtherManagerHost

3. New Non Volatile
Memory・I m p l e m e n t c o m p a r e a n dwrite func tion along with
NVMe Rev1.1 specification.
・Decrease exclusive communications between servers

acc

Modify Ironic (Bare metal control) to device level.

Scalable Key-Value store (NOSQL) on Resource Disaggregated Architecture

・Communication among servers degrades performance such as storage capsyty
and access latency.

Server

Node

Node Acc Manager

OpenStack based Resource Management of
Disaggregated Platform

Resource Disaggregated data Store (RDStore) with NEC・Light-weight software defined storage system with technologies
enhancing data access and management
・Sharing NVMe storage devices among server nodes atinterconnection level (ExpEther)

Global Acc Manager

NodeCompute Node

2. Data distribution andredundancy technology

MPI

Distributed File System (HDFS)

OpenFlow
Controller(Trema)Interconnect

・Apply distributed systemtechnology to the resource
disaggregated architecture
・Enhancing performance,
scalability and availability

Spark

Network Control(1) Selected Node/Mngmnt Information(6) Computing
Resource
Assignment

MPI

Global Accelerator Manager : Global resource allocation within acluster to optimize system throughput and accelerator utilization
Node Accelerator Manager : Accelerator sharing/isolation acrossapplications using accelerator-as-a-service

System Administrator

Resource
Assignment
PolicySpark

Reconfigurable H/W

Cluster/MPI

SDN-Enhanced Job Management Systemwith Nara Institute of Science and Technology (NAIST)JMS(OGS/GE)Resource
Manager

S-Defined Subsystem

Fat TreeGPU

Job Script

Job/Resource
Cross
Management

Scale Out/SparkSpecial Device#!/bin/csh#$ -q QUEUE#$ -pe ompi 4#$ -l netprio=policy_namempirun -np 4 ./a.out

MPI

Spark

RDMA

Must Be Reserveduser

Job-resource cross management system
Job management over Accelerators(GPU/FPGA)
Shared NVM+PCIe fabric based scalable High-speed storage
PCIe device level reconfiguration

HPC computing platform going into cloud. however
HPC computing platform needs application specificconfiguration
Ex. #GPU, #FPGA, InfiniBand, FatTree

Ethernet

1

2

3

1

3

Fabric1

2

A B C D E F G H I

4

1

3

J

Contact : system@cmc.osaka-u.ac.jp Visit us at www.cmc.osaka-u.ac.jp

GUI on modified Horizon

Osaka University, Japan
SC16 BOOTH 1437

Towards a Smart Healthcare System for Orthodontic and Dental Treatment
Motivation

Challenges● Increasing demand and costs for healthcare, exacerbated by● A great amount of interests and motivation in providing effective● A relative shortage of doctors or clinical manpower is also a● D o c to r s a r e r e q ui r e d to p r ov i d e i m m e d i a t e a n d a c c u ra t e● Large amounts of heterogeneous medical data have become● The rapidly increasing availability of Big and Complex EHR data isageing populations, are serious concerns worldwide.
problem that leads to increase their workload.

available in various healthcare organizations.
Electronic Healthcare Records (EHR) are the fundamental
resource to support medical practices or help derive healthcare
insights.
M ost of the medical prac tices are c ompleted by medical
professionals backed by their experiences.
Clinical researches are conducted by researchers via
painstaking designed and costly experiments.

healthcare services through Smarter Healthcare Systems.
diagnoses and proper treatments for patients.

bec oming the driving force for the adoption of data- driven
approaches to automate healthcare related tasks.

● It is also a challenge to achieve earlier disease detection, moreaccurate prognosis, faster clinical research advance and the best
ﬁt for patient management.

ClusterSupercomputer
Virtualization
Systems

Aim

Secure VPN
connecting CMC andany offices
in Department● This project aims to enhance the medical operation efficiency and● It is overburdened for doctors to properly manage a sequence of

operations including hearing, diagnosis, surger y, progress
checkup, counselling, treatment, etc. for all patients.
Especially, doctors spend a lot of time and effort to manuallydiagnose by looking at massive number of oral and face photo
images and x-rays.
extract morphological features of face from CT scan and MRI.
make plans of orthodontic procedure or treatment for patients.

Smart Healthcare System

Front End

Login

Login

Dept. of Orthodontics and
Dentofacial Orthopedics

Cybermedia Center (CMC)

Figure 1. Secure and high performance smart healthcare systems

Data Analytics

improve the quality of healthcare services leveraging highperformance computing resources and advanced machine learningtechnologies.

Healthcare Services & Applications
Visualization

Complex Data Analytics Methods
FeatureSelection

Analytic Model
Selection

Predictive
Analysis

Statistical
Learning

Deep Learning

Correlation
Learning

High Performance
Computing Resources● Smar t Healthcare System operates Data Curation and Data

Obtained raw EMR data is probably heterogeneous composed ofstructured data, free-text data (such as doctors’ notes), image
data (such as MRI images) and sensor data. Hence, data
extraction is of great concern for further analysis.
Data cleansing is necessary to remove inconsistencies anderrors.
Data annotation with medical experts’ assistance contributes to
ef fe c t i venes s an d ef ﬁ c i en cy of this wh ol e p ro c es s fromacquisition to extraction and cleansing.
Data integration is employed to combine various sources of data,such as different hospitals’ data for the same patient.
Finally, statistical, descriptive and predictive analysis of different
types will be performed on processed EMR data.
The analysis results are interpreted and visualized, and are used
to construct medical knowledge and ontology for better andmore accurate analysis.

Data Curation

Analytics, supported by High Performance Computing Resources.

● Figure 2 illustrates the pipeline for Big healthcare data analysis.

Structured Data
Data
Extraction

Input Data

Sensor Data

Data
Annotation

Data
Integration

Convolution & Pooling operations

B

G

Multi-layer Perceptron

IOTN(Labels)

R

Convolutional Neural Network
Features / Labels

The specific objectives are to develop a system that automatically

Lookup Table
Word
Embedding● computes Index of Orthodontic Treatment Priority (IOTN), one of

the severity measures for malocclusion and jaw abnormality, which
determines whether orthodontic treatment is necessary;● extracts facial morphological features (e.g., points and measure);

Knowledge base(text documents)● generates medical certificates or checkup lists;from oral and face photo images and x-rays or cephalogram.

Unstructured Data

Data
Cleansing

Figure 2. An overview of system architecture

Example Applications● provides a set of necessary procedure/treatment recommendations;

Medical Knowledge / Ontology

Raw Data Curated Data Knowledge baseWord1/
Phrase1

LSTM

Word2/
Phrase2

Word2/
Phrase2

LSTM

Word3/
Phrase3

LSTM

WordN/
PhraseN@Koseki Orthodontic Office

Text Generating Recurrent Network
Figure 3. An illustration of example applications

Cybermedia Center & School of Dentistry @ Osaka University

http://www.cmc.osaka-u.ac.jp/