HPC基盤の現状と将来
石川裕
東京大学情報理工学系研究科／情報基盤センター
理化学研究所 計算科学研究機構

Outline of This Talk
• Introduction of Key Organizations/Programs/Activities– HPCI• Innovative High Performance Computing Infrastructure– Seamless access to K computer, supercomputers, and user's machines– SPIRE• Strategic Programs for Innovative Research• Feasibility Study for future HPC in Japan– Background– Introduction of “Feasibility study on
advanced and efficient latency core-based architecture forfuture HPCI R&D”

• Post T2K
2012/8/20

University of Tokyo/RIKEN AICS

2

What is HPCI (Innovative High Performance Computing Infrastructure)
proposals
suggestions

HPCI Consortium
Institutional/Universitycomputer centers

Computational Sciencecommunities

RIKEN AICS (Advanced Institute for
Computational Science )Providing proposals/suggestions to thegovernment and related organizations
 Plan and operation of HPCI system
 Promotion of computational sciences Future supercomputing••••

Japanese Government

contract
HPCI system

Operated by RIST, Riken, NII, andU. of Tokyo with 9 supercomputer
centers’ collaboration

 HPCI Operation
 Seamless access to K computer,supercomputers, and user's machinesDistributed shared storage system
 Joint selection of proposals for K andother supercomputers

Preparation of HPCI Consortiam starts, October 2010
Establishment of HPCI Consortium as legal entity 2ndApril 2012, 1st General Assembly 6th June 2012 (39members)
Submission deadline of proposals using HPCI, 15thJune 2012
Start of HPCI operation, the end of September 2012

RIST: Research Organization for Information Science and Technology
NII: National Institute of Informatics
2012/8/20
University of Tokyo/RIKEN AICS

3

SPIRE (Strategic Programs for Innovative Research)• Objectives– Scientific results as soon as K computer starts its operation– Establishment of several core institutes for computational science• Overview– Selection of the five strategic research fields which will contribute tofinding solutions to scientific and social Issues•••••Field 1: Life science/Drug manufacture
Field 2: New material/energy creation
Field 3: Global change prediction for disaster prevention/mitigation
Field 4: Mono-zukuri (Manufacturing technology)
Field 5: The origin of matters and the universe– A nation wide research group is formed by centering the coreorganization of each research area designated by MEXT.
– The groups are to promote R&D using K computer and to constructresearch structures for their own area2012/8/20

University of Tokyo/RIKEN AICS

4Five strategic groups of SPIRE
1.

Computational Life Science and Application Drug
Discovery and Medical Development
Led by Toshio Yanagida, RIKEN

ゲノム

全身タンパク質

細胞組織，臓器

多階層の生命現象

2.

Computational Materials Science Initiative
Led by Shinji Tsuneyuki, University of Tokyo

3.

Projection of Planet Earth Variations for Mitigating
Natural Disasters
Led by Shiro Imawaki, JAMSTEC

4.

Industrial Innovation
Led by Chisachi Kato, University of Tokyo5.

The origin of matters and the universe
Led by Shinya Aoki, University of Tsukuba

2012/8/20

University of Tokyo/RIKEN AICS

5Outline of This Talk• Introduction of Key Organizations/Programs/Activities– HPCI• Innovative High Performance Computing Infrastructure– Seamless access to K computer, supercomputers, and user's machines– SPIRE• Strategic Programs for Innovative Research• Feasibility Study for future HPC in Japan– Background– Introduction of “Feasibility study on
advanced and efficient latency core-based architecture forfuture HPCI R&D”

• Post T2K
2012/8/20

University of Tokyo/RIKEN AICS

6

What happened in FY2011

http://www.open-supercomputer.org/workshop/sdhpc/

MEXT: Ministry of Education, Culture,Sports, Science, and Technologyreport/advise

Council on HPCI Planand Promotion

Discussion on basic
concept and policy

Science roadmap, draft machine requirements toachieve the roadmap, and discussion of challenges tomeet requirements
report

WG for applications

WG for systems

White paper for strategic
direction/development ofHPC in JAPAN,
Published in March 20122012/8/20

University of Tokyo/RIKEN AICS

7

System Requirement for Target Sciences by 2020
• System performance• Small capacity requirement–MD, Climate, Space physics, …

• Small BW requirement– Quantum chemistry, …

• High capacity/BW requirement– Incompressibility fluid dynamics, …

• Interconnection NetworkHigh BW
small capacity

High BW
middle capacity

High BW
High capacity

1.0E+1Requirement of B/F– FLOPS: 800 – 2500PFLOPS– Memory capacity: 10TB – 500PB– Memory bandwidth: 0.001 – 1.0 B/F– Example applications

1.0E+01.0E-1

1.0E-2

1.0E-3Low BW
Middle capacity

1.0E-4
1.0E-3

1.0E-2

1.0E-1

1.0E+01.0E+1

1.0E+2

1.0E+3Requirement of Memory Capacity (PB)

– Not enough analysis has been carried out– Some applications need >1us latency and large bisection BW• Storage• There is not so big demand
Source: Masaaki Kondo’s presentation at IESP Kobe meeting, 20122012/8/20

University of Tokyo/RIKEN AICS

8

Candidate of the Post Peta-scale Architectures• Four types of architectures are considered– General Purpose (GP)• Ordinary CPU-based MPPs
Memory• e.g.) K-Computer, GPU, Blue Gene,capacity
x86-based PC-clusters– Capacity-Bandwidth oriented (CB)• With expensive memory-I/F rather thancomputing capability• e.g.) Vector machines– Reduced Memory (RM)
• With embedded (main) memory• e.g.) SoC, MD-GRAPE4, Anton

CB oriented

MemorybandwidthGeneral
purpose

ReducedMemoryComputeoriented

FLOPS– Compute Oriented (CO)• Many processing units• e.g.) ClearSpeed, GRAPE-DR

Source: Masaaki Kondo’s presentation at IESP Kobe meeting, 20122012/8/20

University of Tokyo/RIKEN AICS

9Gap Between Requirement and Technology Trends• Mapping four architectures onto science requirement• Projected performance vs. science requirement– Big gap between projected and required performance
Mapping of Architectures

Projected vs. Required Perf.
2700

CBRequirement (PFLOPS)

Requirement of B/F

1.0E+1

1.0E+01.0E-1

RM

GP

1.0E-2CO
1.0E-3

1.0E-4
1.0E-3

Gap between
requirements andtechnology trends
1800

900

0
1.0E-2

1.0E-1

1.0E+01.0E+1

1.0E+2

1.0E+3Requirement of Memory Capacity (PB)

CORM

GP

CB

Needs national research project for science-driven HPC systems
GP (General Purpose), Capacity-Bandwidth oriented(CB), Reduced Memory (RM, Compute Oriented (CO)2012/8/20

Source: Masaaki Kondo’s presentation at IESP Kobe meeting, 2012

University of Tokyo/RIKEN AICS

10Plans
here
FY 2011
Council on HPCI Planand Promotion
Feasibility Study ofAdvanced High
Performance Computing

WhitePaper2012

201320142015

2016

20172018

2019

2020

Discussion on
HPC policy

5M USD/2012(436M JPY)
Four teams run

Whether or not the national R&D
project starts depends on results ofthose feasibility studies

R&D of Advanced HPC
Basic Research Programs
CREST:Development of System
Software Technologies for postPeta Scale High Performance
Computing

Deployments
2012/8/20

23M USD (1750M JPY)
17M USD (1410M JPY)
20M USD ?
2014 -- 2015:
Post T2K
University of Tokyo/RIKEN AICS

11Feasibility Study on Future HPC R&D in Japan
Program promotion board
Member: The head of each team and other specialists
Role: To check the progress of the each team and to coordinate
the collaboration among the teams
1 application study team

3 system study teams
TohokuUniv. andNEC

RIKEN AICS and TITECH
Collaboration with application filelds• Identification of scientific and social issues
to be solve in the future• Drawing Science road map until 2020
• Selection of the applications that plays key
roles in the roadmap• Review of the architectures using those
applications
2012/8/20

U. ofTsukuba,
Titech, andHitachi

U. of Tokyo,
Kyushu U.,
Fujitsu,
Hitachi, andNEC• Design of computer systems solving
scientific and social issues• Identification of R&D issues to realize thesystems• Review of the system using theapplication codes• Estimation of the system’s cost

University of Tokyo/RIKEN AICS

12

Outline of This Talk
• Introduction of Key Organizations/Programs/Activities– HPCI• Innovative High Performance Computing Infrastructure– Seamless access to K computer, supercomputers, and user's machines– SPIRE• Strategic Programs for Innovative Research• Feasibility Study for future HPC in Japan– Background– Introduction of “Feasibility study on
advanced and efficient latency core-based architecture forfuture HPCI R&D”

• Post T2K
2012/8/20

University of Tokyo/RIKEN AICS

13

Towards Next-generation General Purpose Supercomputer
Approach:Material and Climate Sciences are the first target applications
Approach from evolution of the K architecture
System Software Stack is designed for both the proposed machineand commodity-based machines

PI: Yutaka Ishikawa, U. of Tokyo Organization
 System Software Stack Performance Prediction and Tuning

Co-PI: Yuichi Nakamura, NEC System Software Stack

Co-PI: Kei Hiraki, U. of TokyoApplications
System Software Stack Architecture Evaluation, Compiler,and Low power technologies(MPI, parallel file I/O, PGAS,
Batch Job Scheduler, Debugging andTuning Tools)

Co-PI: Mutsu Aoyagi, Kyushu U.

Commoditybased
Supercomputer

Co-PI: Naoki Shinjo, Fujitsu

Next-Gen
General Purpose
Supercomputer Network Evaluation Environment Processor, Node,Interconnect Architecture andSystem Software Stack

Co-PI: Tsuneo Iida, Hitachi
 Storage Architecture andSystem Software Stack
2012/8/20

University of Tokyo/RIKEN AICS

14

Co-design• Tightly coupled design of architecture by architects,
software developers, and application developers.
• 1 Cycle / 2 months

Programming Model
Execution Model

Evaluation andTuning of Apps.

Architecture Design:

Examination basedon comparisons

Processor, Node, Network,and Storage

Evaluation ofExisting Systems

Design and prototypingof System Software,and Evaluation
2012/8/20

University of Tokyo/RIKEN AICS

15

Part of Target Applications in FY2012•ALPS(Algorithms and Libraries for Physics Simulations)–•– Total Memory: 10〜100PB, low latency and high radix
network
RSDFT (Real-Space Density Functional Theory)–•A DFT(Density Functional Theory) code with real space discretizedwave functions and densities for molecular dynamics simulations
using the Car-Parrinello type approach– Total Memory: 1PB, Performance: 1EFLOPS(B/F 0.1)
NICAM (Nonhydrostatic ICosahedral Atmospheric Model)–•

Providing high-end simulation codes for strongly correlated
quantum mechanical systems

A Global Cloud Resolving Model (GCRM)– Total Memory: 140 TB, Memory Bandwidth: 300 PB/sec,
Performance: 700 PFLOPS(B/F = 0.4)
COCO (CCSR Ocean Component Model)–ocean general circulation model developed at Center for Climate
System Research (CCSR), the University of Tokyo– Total Memory: 320 TB, Memory Bandwidth: 150 PB/sec,Performance: 50 PFLOPS (B/F = 3)

2012/8/20

University of Tokyo/RIKEN AICS

1 Job:
Exec. Time: 10 hours
Files: 500TB
1 problem:
10 jobs, 5 PB storage
1 Job:
Exec. Time: 24 hours x 10
Files: 1PB x 10
1 problem:
1 job, 10 PB storage
1 Job:
Exec. Time: 720 hours x 100
Files: 10 TB
1 problem:
1 job, 1 PB
16

Outline of This Talk
• Introduction of Key Organizations/Programs/Activities– HPCI• Innovative High Performance Computing Infrastructure– Seamless access to K computer, supercomputers, and user's machines– SPIRE• Strategic Programs for Innovative Research• Feasibility Study for future HPC in Japan– Background– Introduction of “Feasibility study on
advanced and efficient latency core-based architecture forfuture HPCI R&D”

• Post T2K
2012/8/20

University of Tokyo/RIKEN AICS

17

Post T2K
2005

2006

2007

2008

2009

2010

2011

2012

2013

2014

2015

2016

2017

2010

100+ PFlops
Fujitsu FX10 (1PFlops)
Hitach SR16000/M1 (54.9 Tflops)

SR11000

T2K Todai

140TFlops (HA8000 Cluster)

R&D

2019

Exa-scaleSupercomputer“K” Computer
10 PFlops

Market2018

40 to 100 PFlops

Hongo CampusKashiwa Campus

FX10• PRIMEHPC FX10
• 4800 Node (16 core/node)• 1.13 PFlops• 150 TB main memory• Hitachi SR16000/M1• 56 Node (32 Core/Node)• 54.9 TFlops• 11200 GB main memory

HA80002012/8/20

SR16000/M1

University of Tokyo/RIKEN AICS

18

Variations of Many-core based machines
Many-core board connected to PCI-Expresse.g., Intel Knights Ferry, Knights Corner
Many Core Board

memory memory

Host Board
IOH

IOH

PCI-Express
memorymemorymemorymemoryMany-core chip connected to system bus
Not existing so far

Host CPUmemory

Host CPUmemory

memoryMany-core only
Not existing so far

Many-core inside CPU diec.f., Intel Sandy Bridge with GPU

memory memory

IOH

http://pc.watch.impress.co.jp/docs/column/kaigai/20100412_360173.html
2012/8/20

University of Tokyo/RIKEN AICS

memorymemory19

System Software StackIn case of Non-Bootable Many Core

In case of Bootable Many Core

System Software Stack

System Software Stack

Linux APIs, MPI, PGAS, File I/O, File System, Batch Job System, etc.

Linux Kernel

Helper Threads for
uKMC

SMSL
IKCL
AAL-HostDevice Driver
HostuKMC (MicroKernel for

PCI-ExpressInfiniband
Network CardSMSL Many Core)IKCL
AAL-Manycore
Many Core

uKMC (MicrouKMC Kernel(Micro for
SMSLMany Core)
uKMC Kernel(Microfor
Helper Threads for
SMSL
ManyCore)uKMC(MicroKernel
for
IKCLLinux Kernel uKMC
SMSLMany Core)
Kernelfor
IKCL
AAL-Manycore
SMSL
SMSL
ManyCore)IKCL
AAL-Manycore
IKCL
AAL-Host
IKCL
AAL-Manycore
Device Driver
CAL-Manycore
Many CoreLinux APIs, MPI, PGAS, File I/O,
File System, Batch Job System,etc.

Infiniband
Network Card

Design Criteria• AAL (Accelarator Abstraction Layer)• Cache-aware system software stack– Provides low-level accelerator interface• Scalability– Enhances portability of the micro kernel• Minimum overhead of communication facility • IKCL (Inter-Kernel Communication Layer)• Portability– Provides generic-purpose communication and data transfermechanisms• SMSL (System Service Layer)– Provides basic system services
2012/8/20

University of Tokyo/RIKEN AICS20

DCFA: Direct Communication Facility for Accelerator• Limitations of a PCI-Express device– cannot configure another device such as a communication device,and thus it does not know the other device address.
– Cannot receive interrupts from other devices.

• DCFA– The host configures andinitializes an Infiniband HCA,
and informs the HCA address
to an MIC device so that it may
issue commands to that device– The MIC device directly
accesses the Infiniband HCA
registers

2012/8/20

MemoryHOSTCPUMemoryMemoryCommand Issued byMany Core
Many
HOST

Core
PCI-Express
Configured and Initialized byInfiniband
Host CPU
Network Card

CPU

MemoryManyCore
PCI-Express
Infiniband
Network CardMin Si and Yutaka Ishikawa, "Design of Direct Communication Facility for Manycorebased Accelerators, “ to appear at CASS2012 in conjunction with IPDPS2012.

University of Tokyo/RIKEN AICS

21Latency• The same performance as that of host to host datatransfer for large message size
𝑅𝑅𝑅𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 = 𝑅𝑅𝑅𝐷𝐷𝐷𝐷�𝑅𝑅𝑅𝐻𝐻𝐻𝐻
MemoryMemoryMemory2.5
Send/Receive
2

RDMA write

1.5

Memory

1
HOST
CPU

Many
Core
PCI-Express
Infiniband
Network Card

HOST
CPU

MemoryMemoryHOSTCPUHOSTCPU

2012/8/20

PCI-Express
Infiniband
Network Card

Many
Core
PCI-Express
Infiniband
Network Card

0.5
0Message Size (bytes)
Intel Xeon X5680 3.33GHz x 2
PCI-Express
Infiniband
Network Card

Mellanox MT26428
Knights Ferry

University of Tokyo/RIKEN AICS

22

Design Considerations of File I/O System• File I/O functions run on computing core in MIC• File I/Os are delegated to the OSdedicated core in MIC• File I/Os are delegated to the host OS

Yuki Matsuo, Taku Shimosawa, and Yutaka Ishikawa, "A File I/O System for ManyCore Based Clusters,” in conjunction with ICS2012, 2012.

Yutaka Ishikawa, University of Tokyo

2012/8/20

University of Tokyo/RIKEN AICS

23

Performance Differences
IterativeOnce

size = 64KB;
for(n = 0; n < DIVISOR; n++) {for(i = 0; i < size/4; i++) buf[i] = n;write(fd, buf, size);}

size = 64KB; j = 0;
for(n = 0; n < DIVISOR; n++) {for(i = 0; i < size/4; i++) buf[j++] = n;}write(fd, buf, size*DIVISOR);

In the iterative simple benchmark,the write system call is issued
during the user data is located on
L2 cache.
write_comp

Relative Total Execution Timewrite_os

Yutaka Ishikawa, University of Tokyo
2012/8/20

University of Tokyo/RIKEN AICS

24

Summary• Building Nation-wide infrastructure and collaboration
structure– HPCI: Innovative High Performance Computing Infrastructure– SPIRE: Strategic Programs for Innovative Research• Starting Feasibility Study for future HPC in Japan– 1 application and 3 architecture teams have been selected• Studying OS mechanisms for Post T2K– A manycore-based cluster has been considered and beenstudied2012/8/20

University of Tokyo/RIKEN AICS25