Recent Activities in CCS
Masayuki Umemura
Director

Oakforest-PACS
JCAHPC: Joint Center for Advanced High Performance Computing
U. Tsukuba

U. Tokyo Peak Performance 25 PFLOPS
 LINPACK 13.55PFLOPS No. 3 in HPCG list

Intel® Omni-Path Architecture
Full bisection tat-tree network

768 port Director Switch
12台

Node (Fujitsu PRIMERGY CX1640 M1)
CPU Intel Xeon Phi 7250 (Knights Landing)1. . . 2425...

48

26 PB Parallel file system，940 TB High-performance file cash system

48 port Edge Switch
362 台

Oakforest-PACS ranks #1 in IO 500 list
Oakforest-PACS Storage System

Parallel file system (DataDirect Networks ES14KX)
File cache system (Infinite Memory Engine IME14KX ): Improving the storage performance.

IO-500 benchmark in 2017

The file cache system achieved 742 GiB/s[1] for file-per-process write access that parallel
processes access their own file.
and 600 GiB/s for single-shared-file write access that parallel processes access a single
shared file but a different position.

OFP file cache system
IME14KX

Development of Massively Parallel Computer Systems in CCS• 1977 research begins (by Hoshino, Kawai)• 1978 1st machine• 1996 CP-PACS (top of Top500)• 2006 7th machine PACS-CS• 2012 8th machine HA-PACS

CP-PACS First large-scale general-purpose MPP system in Japan Development supported by “Research of Field Physics withDedicated Parallel Computers” funded by the Ministry ofEducation of the Japanese Government.
 ranked as No. 1 system in the November 1996 Top 500 List.
 Collaboration by physicists and computer scientists
 Collaboration with industry, and released as Hitachi SR2201

1996
1978
1st PACS-91980
2nd PAXS-32

Year
1978
1980
1983
1984
1989
1996
2006
2012
2014

1989
5th QCDPAX

6th CP-PACS

2006(top of Top500 list in 1996)

7th PACS-CS

System
PACS-9 (PACS I)
PACS-32 (PACS II)
PAX-128 (PACS III)
PAX-32J (PACS IV)
QCDPAX (PACS V)
CP-PACS (PACS VI)
PACS-CS (PACS VII)
HA-PACS (PACS VIII)COMA (PACS IX)

2012

2014

8th HA-PACS

9th COMA

Performance
7 KFLOPS
500 KFLOPS
4 MFLOPS
3 MFLOPS
14 GFLOPS
614 GFLOPS
14.3 TFLOPS
1.166 PFLOPS
1.001 PFLOPS2007
FIRST(Hybrid Simulator)

36 TFLOPSPACS-X: Hybrid Accelerator System
 GPU&FPGA are embedded withPCIe bas
 Local network system and offloading with FPGA
 Acceleration of hot spot withGPU
Multi-hybrid node
Large-scale network

Pre-PACS-X (PPX) PPX1(8nodes), PPX2(8nodes)
 Optimization of off-loadingwith FPGA

InfiniBand

CPUCPUPCI Express bus
Fast direct communicationGPU

Coase-grained calc.

GPU

FPGA

Fine-grainedcalc.

100Gbps class x N
Local network

Center for Computational Sciences, Univ. of Tsukuba

Timeline of Computing Systems in CCS
Fiscal Year

TPF

2013

2014

2015

2016

HA-PACS (PACS-VIII) 800TF

2017

2018

PPX
64TF

PPX2
64TF

2019

2020

2021

2022

PACS-X ～10PF

HA-PACS TCA +366TF

2MW340kW

UCC+TPFCOMA (PACS-IX) 1PF

T2K-tsukuba

4.5MW

Oakforest-PACS 25PF

630kW

T2K-Todai

JCAHPC

4.2MW

TPF = Technology Path-Forward Machine
UCC = Upscale Commodity Cluster Machine Joint Center for Advanced High Performance Computing (JCAHPC)
筑波大学

東京大学
新 物 質 ・エネルギー

Center for Computational Sciences
U. Tsukuba

数理物質科
学研究科
つくばナノテクアリーナ(TIA)

G30計算科学
英語プログラム

生命環境科
学研究科
シ ステム情報工学研究科

リーディング大学院
ヒューマンバイオロジ

工学部
工学系

大気海 洋研

理学部
理学系

生研

素核宇宙

もの づ くり新領域

計算科学デュアル
ディグリープログラム

計算科学研
究センター

防 災 ・ 減災
地震研

物 性研

医科研

Information Technology Center
U. Tokyo

医療・生命科学

情報理工学系

情報基盤センター Feasibility Study toward Exa-scale Computing (National Project)－7－Organization of CCS
Deputy Director
Steering
Council

Strategic Planning
Headquarters

Director for
Organization Strategy

Project Office for Exascale Computational Sciences
Project Office for Exascale Computing System Development

Research ProjectOffices

Director
AdministrativeCouncil
Personnel
Committee

Project Office for HPCI Strategic Program
Liaison Office for Multidisciplinary Computational Sciences

Researcher
Meeting

Promotion Office for Computational Sciences
Computational Medical
Science Project
Bureau of Public
Relationships and Strategy
Board ofCooperative Research

Faculty 35+2
Professor 12
Associate Prof. 12
Lecturer 5
Assistant Prof. 6
Associate/Assistant Prof. +2

Director for
Cooperative Research

Administrative Committee for
Cooperative Research

Administrative Committee for
Computer Systems

Joint Center of Advanced HPC

ResearchDivisions

Particle
Physics

Astrophysics

NuclearPhysics

Quantum
Condensed
Matter Physics

Life Science

GlobalEnvironmental
ScienceHigh Performance
Computer Systems

ComputationalInformatics

Development of Novel Scientific Fields

Computational Medical Science Project
Development of medical technology by computational science in cooperation with medicine, physics, biology,mathematics, data science, and image processing technology.

Computational Biomolecular
Medical Science
Collaborations： Medicine,
Biology, and Physics
Drug discovery, detection of SNP,and physical mechanism analysis
by bioinformatics and simulations

3DCG Virtual Surgery

Sleep Big Data Analytics &Automatic Sleep Diagnosis

Life Science

Data Science

Computational
Medical Science
ComputationalMedia

Collaborations： Medicine,Engineering, Design
Development of surgical operation support system using
computational surgery in cooperation with Medical, System
Information Engineering, Information Media researches

Collaborations： IIIS,
C-AIR, CCR
System for automatic analysis ofbrain waves in sleep with machinelearning and deep learning

Computational
Physics

Industry

Computational Optical
Bioimaging

Collaborations： Biomedical optics,
Mathematics
Development of non-invasive near-infrared computed
tomography in cooperation with biomedical optics, radiationtransfer physics, and mathematical inverse problem.

Computational Biomolecular Medical Science
Main Target

Current Problems••••••

Medium size drugs
Covalent bond-type drugs
Single Nucleotide Polymorphism

Diversity in Structures (Vast Conformations）
Reaction Control （First-principles Calculations）
Fast Gene Analysis

Drug discovery needs long time（10〜15 year）and a lot offund（$1billion). But the success ratio is less than 1/20000
We needs a breakthrough for in silico drug discovery,i.e. detection of target, design of drug and proteins & etc.

Evolution

Next-generation
sequencer

Fusion of Biology, Medicine,and Physics to solve these problems

Drug Simulation

First-principle

Prof. Y. Inagaki
Department of Biology

Prof. T. Hirokawa
Department of Medicine& AIST

Prof. Y. Shigeta
Department of Physics

ComputationalBiology

Bioinformatics,
In Silico Drug discovery

Biophysics, First-principles
calculations

Influenza HA3First-principles Calculation

Fast Single Nucleotide
Polymorphism detection

Phylogenetic tree analysis by GPU
Mitocondrial protein detection by supercomputer

Gene analysis
SNP analysis

Molecular dynamicsof FtsZ In cell fission

Fragment molecular orbital by GPU architecture
Efficient sampling method by Molecular Dynamics

In Silico drug discovery project by AMED
since 2017.

Post K Computer Project by MEXT
since 2015

Sleep Big Data Analytics & Automatic Sleep Diagnosis•••

Sleep plays an important role in daily life of people. Sleeping problems could trigger various
diseases.
There has not been any means for conveniently measuring the sleep state as accurately as
medical doctors and technicians.
Integrate bio-big data analysis, machine learning, and new sensing technology to realize
automatic analysis and diagnosis of sleep.

Industry
Collaboration

Sleep Big Data Analytics& Machine Learning
Sleep state monitoring, Bio-big data analysis,
Anomaly detection, Automatic diagnosis
Prof. Kitagawa, CCS

Big Sleep Data・ EEG
・ EMG
・ EOG, etc.

EEG

Sleep Measurement
Advanced sensing equipment,
Sensing devices
Prof. Sankai, Center for
Cybernics Research/C-AIR

Sleep Medicine

EMG

HPC environments:
GPUs, Many-cores

FFT&PCASVM

ReEvaluation

Non-REM
REM
WAKE

Principle and mechanism, Big data analytics: Machine Learning, Deep Learning
Diagnosis and treatment ofpositiveIntermediate layers
Input layer
Output layer
Non-REM
sleep disorders
negativeProf. Yanagisawa, IIIS/C-AIR

Collaboration within campus
MEXT: Regional Innovation Eco-system Development
MEXT: Grant-in-Aid for Scientific Research on Innovative Areas

OthersClassifier for Non-REM

3DCG Virtual Surgery• Improving instruction level: Integration of 3D CG simulation and onsite navigation.
• Solving surgeon shortage: VR remote operation, advanced skill passing.
• Boosting medical education: Evaluation of surgical procedures and proficiency.
Collaboration Team
※1 Itaru Kitahara （CCS）※2 Yukio Oshiro （Medical Doctor）※3 LEXI Co., Ltd (CG Surgery)

Surgical Analysis
Pathological Diagnosis[3D Data Analysis ※2]

Surgery Navigation[Mixed Reality ※１,2,3]
3DCG Simulation[Computer Graphics ※2,3][Observation Using
Diverse Sensors ※1]

3D Surgical VideoProcessing[Computer Vision ※1]

Computational Optical Bioimaging
 No explosure, non-invasive, no side effects like CT or MRI, applicable to new-born babies and infants Bed side inspection, low cost, low restraint, high time resolution, automatic and remote diagnosis, monitoring, screening Diagnosis of cancers in thyroid, mammary gland, surface tissue, blood flow, ventricles, subarachnoid, etc.

Imaging of ventricles

Accurate Algorism of Image Reconstruction
Biomedical Optics
Diffuse Optical Tomography (DOT) , Photoacoustic
Imaging (PAI), Fluorescence Tomography

Direct Problem Analysis

Inverse Problem Analysis

Numerical solution of radiative transfer withabsorption, scattering, and reflection.

Mathematical Analysis of non-linear
optimization problem.

Imaging of thyroid gland