The concept of user services
onFugaku
Fumiyoshi Shoji
Operations and Computer Technologies Div., R-CCS, RIKEN@ 2nd R-CCS international symposium
February 17, 2019

K computer retired Aug.2019
2006
2007
2008
I moved to
RIKEN(November 2005)

2009

2010

2011

2012

2013

2014

Conceptual/Detailed design for facility

2016

2017

2018

2019

8 years and 5 months(Apr. 2011-Aug.2019)

Conceptual/Detailed design for K computer

Early access• Achievements:• TOP500 #1• Graph500 #1• HPCG #1• Gordon Bell prize winner

2015

Official Operationx2x 10x3x2

2

Operation/service stats of K computer
Service duration# of job

2,513 days 9 hours(Sep. 28th, 2012 – Aug. 16th, 2019)

4,178,431

Node x time delivered

3,637,258,658

Average job filling rate

75.6%

System availability(for the service duration/for planned service node time)# of user(cumulative/no double counting)# of project(cumulative)

93.6/97.3%
11,095/2,631
1,0153

Fugaku – successor of K computer The Nex-Gen “Fugaku” Supercomptuer
High-Peak --- Acceleration ofLarge Scale Application(Capability)

Mt. Fuji representing
the ideal of supercomputing

ふがく

富 岳

Broad Base --- Applicability & Capacity
Broad Applications: Simulation, Data Science, AI, …
Broad User Bae: Academia, Industry, Cloud Startups, …

Presentation by Satoshi Matsuoka @EEHPC SOP Workshop 2019https://sites.google.com/view/eehpcsop2019/4

Actionl Improving usabilitylaccessibilitylopen source software deploymentldata science platforml Improving efficiencylPre/Post I/Olnode allocationlcheckpoint/restartlpower knob by user and admin
5

Accessibility

CUI and access API(NEWT2.0 based) are availablelinteractive use is also available under batch job schedulingKVM and Singularity will be testedl Front-end/PrePost environmentllMulti architecture basedlx86(w/ GPU), arm TX2(w/ GPU), A64FX(48 nodes)linteractive/batch/OpenStack/Kubernetes (will be tested)Amazon S3 compatible object storage (underprocurement)

OpenStack(will be tested)laccess API

lJobs can be executed via Fujitsu batch job scheduler
SSH

lKubernetes(will be tested)

new!

l Compute nodes

Front-end/PrePostjob control

6Collaboration with commercial service providers
existing users
Fugakupotential users
Service provider #1
Service provider #2...
Service provider #?

Collaborating with service providers, we can provide more
flexible service for wider field of science and engineering users
7Collaboration partners selected

https://www.r-ccs.riken.jp/library/topics/200213.html (in Japanese)×Action• Cool Project name and logo!
• Trial methods to provide computing resources of Fugaku to end-users via service providers• Evaluate the effectiveness of the methods quantitatively as possible and organize the issues• The knowledges gained will be feedbacked to scheme design of Fugaku by the government8

Access APIl We employed NEWT2.0 as a prototype of access API of Fugaku

l We will discuss standardization of API with HPC centers/providersl An implementation of the API on Fugaku will be available August

2020

https://github.com/NERSC/newt-2.09Open source software for Fugakul

For K computerll

Due to special ISA (Sparc based), there was no software eco-system…

For FugakulActivities for open source software on arm ISA- Arm HPC Users Group https://arm-hpc.gitlab.io/- Linaro https://www.linaro.org/- Spack: https://spack.io/lR-CCS Software Center https://www.r-ccs.riken.jp/software_center/l

Official software package manager of the Exascale Computing Project

Activity in R-CCS to develop, deploy and promote high quality applications, libraries,programming tools, etc. make in R-CCS for many HPC platforms including Fugaku.

DL4Fugaku https://github.com/dl4fugaku/dl4fugaku/wiki-R-CCS & Fujitsu collaboration for Deep learning framework on Fugaku-Target: PyTorch, TensorFlow, Chainer, etc.

10

Data science platformSensor

atdnepoGenomeWeather

Trafficaobject storage（S3 compatible）

Analysis

file storage（Lustre based）

under procurement
11Actionl Improving usability
laccessibilitylopen source software deploymentldata science platforml Improving efficiency
lPre/Post I/Olnode allocationlcheckpoint/restart

lpower knob by user and admin
12

Sharing pain for efficiency
l Average job filling rate : 75.6% (= node allocation loss : 24.4%)

A complicated Pre/Post I/O implementation and operation rule
IRREGULAR DOWN TIME

1.

300

200

1000
FY2012(Sep.-Mar.)FY2013

FY2014File system

FY2015FY2016job scheduler

MPI

FY2017FY2018

misc

2.

An inefficient node allocation rule (2-3%)

3.

Resource compensation rule for system failure (1-2%)

FY2019(Apr.-Aug.)

13

Sharing pain for efficiency (Pre/Post I/O)

l Pre/Post I/Ol

Asynchronous Pre/Post I/O could be available on the K computer
asynchronous type(K computer)

Pre

job A
Pre

Post

job B
PrePresynchronous type(Fugaku)

job A

Post

job C

Post

Post

Pre

job B

Post

Pre

job C

Post

An asynchronous Pre/Post I/O was much more difficult to implement and its complexity might induce
many serious bugs in system software. à We adopt a synchronous type for FugakulTo optimize I/O requests, Pre/Post I/O will be counted as user time
K computer

no-counted
Precounted

job

no-counted
PostFugaku

countedPrecounted

job

countedPost

14

Sharing pain for efficiency (node allocation)l K computerlllà

A block-wise (≠distributed) node allocation policy due to a directconnection network topology
node allocation unit is 2x3x2 = 12 nodes
User can run a job with any node size (even not a multiple of 12nodes)

232

node allocation lossàby the gap between user request and system assigned

àby scheduling difficultyl Fugakull

A block-wise policy and node allocation unit is 2x3x8 = 48 nodes
User can choose node size in a multiple of 2x3x8 (job with morethan 48 node case)15

Sharing pain for efficiency (checkpoint/restart)l K computerlNode hours lost by system failure was compensated.

planned elapsed timenode hours spent till the failurel Fugakulsystem failure occurs and the job
are terminated by system

An user level checkpoint/restart tools (e.g. ECP-VeloC/VELOC) will beavailable on Fugakul

https://github.com/ECP-VeloC/VELOC

à Itʼs time to finish resource compensation for system failure…
16

New functions of Fugaku for energy saving
Fujitsuʼs presentation @ Hot Chips30 https://www.fujitsu.com/jp/Images/20180821hotchips30.pdf

Average power consu
mption in the idle decr
ease up to 5-10%

User can use the power knob via Power API

17

How can we motivate users for energy saving?
l Which policy is better?

all power knobs are turn off at default (start from minimum saving)1.
ladmin finds out jobs that are wasting power from profiling datal

admin requests user to turn on the knob

l

Pros

: Less user complaints

l

Cons

: Less energy saving

all power knobs are turn on at default (start from maximum saving)2.
luser shows to admin that using the knob reduces (keeps) energy-to-solution for
his/her job by trialladmin allow the user to turn off the knob

l

Pros

: More energy saving

l

Cons

: More user complaints
18

How can we motivate users for energy saving? (contʼd)l Grant incentives depending on the contribution to the power savingll

additional node hours, higher priority, etc.
Concern: How can we fairly evaluate “contributions” for energy saving(“as-is” --> tuned)?

l Change resource allocation unitllnode x hours -> energy (watt hour)
Concern: How can we keep fairness between applications which havedifferent power profile?

19

Summaryl Easy to uselaccessibility by collaboration with commercial service providerslopen source software deployment by Spackldata science platform by object/file storages with analysis env.

l Sharing pain for efficiencylPre/Post I/Olnode allocationlaggressive use of power knob for power saving

20Thank you for your attention

21