Activity Report 2019 / Supercomputer Center, Institute for Solid State Physics, The University of Tokyo

1

OUTLINE

1.1

Supercomputer System

In SY2019, the ISSP supercomputer center provided users with System B - SGI
ICE XA/UV hybrid system and System C - HPE SGI 8600 system. System B isa massively-parallel supercomputer with three types of compute nodes: 19 “Fat”
nodes, 1584 “CPU” nodes, and 288 “ACC” nodes. “Fat” nodes are each comprised of four Intel Xeon E5-4627v3 CPUs (10 cores/CPU) and 1 TB of memory.
“CPU” nodes have two Intel Xeon E5-2680v3 CPUs (12 cores/CPU) and 128 GBof memory. “ACC” nodes have two nVIDIA Tesla K40 GPUs in additon to two
Xeon E5-2680v3 CPUs and 128 GB of memory. System B achieves 2.6 PFLOPS
in theoretical peak performance with high power eﬃciency. The subsystem comprised of only CPU nodes ranks 61st on the November 2015 Top 500 List, which isa ranking based on total performance measured by the HPL benchmark. The subsystem of ACC nodes ranks 104th on the Top 500 List, and it also ranks 23rd on
the Green 500 List, which is a ranking based on performance per watt of electrical
power consumption. The compute nodes communicate to each other through FDR
Inﬁniband. The Fat nodes are interconnected in fat tree topology, while the CPU
and ACC nodes are connected in enhanced hypercube topology. System B enteredoﬃcial operation on Aug. 21, 2015. SY2019 was the ﬁfth year of the operation of
the current System B. System C is a massively-parallel supercomputer with 252“CPU” nodes, which have two Intel Xeon Gold 6148 CPUs (20 cores/CPU) and
192 GB of memory. System C achieves 774 TFLOPS in theoretical peak performance. SY2019 was the second year of the operation of the current System C. For
further details, please contact ISSP Supercomputer Center (SCC-ISSP).
[Correspondence: center@issp.u-tokyo.ac.jp]

1.2

Project Proposals

The ISSP supercomputer system provides computation resources for scientists
working on condensed matter sciences in Japan. All scientiﬁc staﬀ members (including post-docs) at universities or public research institutes in Japan can submit
proposals for projects related to research activities on materials and condensed
matter sciences. These proposals are peer-reviewed by the Advisory Committee
members (see Sec. 1.3), and then the computation resources are allocated basedon the review reports. The leader of an approved project can set up user accounts
for collaborators. Other types of scientists, including graduate students, may also
be added. Proposal submissions, peer-review processes, and user registration are
all managed via a web system.
The computation resources are distributed in a unit called “point”, determined
as a function of available CPU utilization time and consumed disk resources. There
were calls for six classes of research projects in SY 2019. The number of projects
and the total number of points that were applied for and approved in this school
year are listed in Table 1.

1

Activity Report 2019 / Supercomputer Center, Institute for Solid State Physics, The University of Tokyo

Table 1: Classes of research projects in SY 2019

Class

A
B
C
D
E
S
SCCMS
Total

Maximum
PointsSys–B Sys–C
100
100
1k
500
10k
2.5k
10k
2.5k
30k
2.5k
–
–

Application

# of
Proj.

any time
twice a year
twice a year
any time
twice a year
twice a year

17
88
182
5
17
0
36
345

Total points
AppliedApproved
Sys–B Sys–C
Sys–B Sys–C
1.7k
0.9k
1.7k
0.9k
83.8k
4.7k
58.3k
4.1k
1416.2k88.2k
608.0k
63.0k
26.9k
2.0k
26.9k
2.0k
475.0k
44.0k
253.0k
35.6k
0
0
0
0
253.0k
23.5k253.0k
23.5k2255.6k 163.2k 1200.9k 128.8k• Class A is for trial use by new users; proposals for Class A projects are accepted
throughout the year.
• Proposals for projects in Classes B (small), C (mid-size), E (large-scale), and S(exceptional) can be submitted twice a year. Approved projects in Classes A, B,
C, and E continue to the end of the school year.
• In Class D, projects can be proposed on rapidly-developing studies that need toperform urgent and relatively large calculations. An approved project continues
for 6 months from its approval.
• Class S is for projects that are considered extremely important for the ﬁeld ofcondensed matter physics and requires extremely large-scale computation. The
project may be carried out either by one research group or cooperatively by several
investigators at diﬀerent institutions. A project of this class should be applied withat least 10,000 points; there is no maximum. We require group leaders applying
for Class S to give a presentation on the proposal to the Steering Committee of
the SCC-ISSP. Class S projects are carried out within one year from its approval.
• Project leaders can apply for points so that the points for each system do not
exceed the maximum point shown in this table.

2

Activity Report 2019 / Supercomputer Center, Institute for Solid State Physics, The University of Tokyo

SGI ICE XA/3744 CPU + 576 GPU
2.62PFlops 240.4TB memory

SGI UV/
UV/76 CPU3
31.6TFlops
1
19TB memoryd array
disk
1.29 PB
Front-end WS
SGI Rackabe Standard-Depthe Server C2112-GP2

System B
System C

HPE SGI 8600
774TFlops 47.25TB memory

disk array 500TB

Front-end WS
HPE

YOU

Figure 1: Supercomputer System at the SCC-ISSP

In addition, from SY 2016, ISSP Supercomputer is providing 20% of its computational resources for Supercomputing Consortium for Computational Materials
Science (SCCMS), which aims at advancing parallel computations in condensed
matter, molecular, and materials sciences on the 10-PFlops K Computer and theexascale post-K project. Computational resources have also been allotted to Computational Materials Design (CMD) workshops, as well as for Science Camps heldin ISSP for undergraduate students.

1.3

Committees

In order to fairly manage the projects and to smoothly determine the system
operation policies, the Materials Design and Characterization Laboratory (MDCL)of the ISSP has organized the Steering Committee of the MDCL and the Steering
Committee of the SCC-ISSP, under which the Supercomputer Project Advisory
Committee (SPAC) is formed to review proposals. The members of the committees
in SY 2019 were as follows:

Steering Committee of the MDCLHIROI, Zenji

ISSP (Chair person)3

Activity Report 2019 / Supercomputer Center, Institute for Solid State Physics, The University of TokyoKAWASHIMA, Naoki
OZAKI, Taisuke
NOGUCHI, Hiroshi
UWATOKO, Yoshiya
SUGINO, Osamu
TOHYAMA, Takami
ONO, Tomoya
YAMAURA, Jun-ichi
TAKAHASHI, Hiroki
MOTOME, Yukitoshi
HOSHI, Takeo
KIMURA, Kaoru
HASEGAWA, Masashi
NAKATSUJI, Satoru

ISSP
ISSP
ISSP
ISSP
ISSP
Tokyo Univ. of Sci.
Kobe Univ.
Tokyo Tech.
Nihon Univ.
Univ. of Tokyo
Tottori Univ.
Univ. of Tokyo
Nagoya Univ.
ISSP

Steering Committee of the SCC-ISSP
KAWASHIMA, Naoki
NOGUCHI, Hiroshi
OZAKI, Taisuke
SUGINO, Osamu
TSUNETSUGU, Hirokazu
KATO, Takeo
KONDO, Takeshi
MORITA, Satoshi
HIGUCHI, Yuji
FUKUDA, Masahiro
KAWAMURA, Mitsuaki
NAKAJIMA, Kengo
TSUNEYUKI, Shinji
HATANO, Naomichi
MOTOME, Yukitoshi
ONO, Tomoya
TOHYAMA, Takami
YOSHINO, Hajime
SUZUKI, Takafumi
YOSHIMOTO, Yoshihide
OKUMURA, Hisashi
HOSHI, Takeo
YATA, Hiroyuki
FUKUDA, Takaki

ISSP (Chair person)
ISSP
ISSP
ISSP
ISSP
ISSP
ISSP
ISSP
ISSP
ISSP
ISSP
Univ. of Tokyo
Univ. of Tokyo
Univ. of Tokyo
Univ. of Tokyo
Kobe Univ.
Tokyo Univ. of Sci.
Osaka Univ.
Univ. of Hyogo
Univ. of Tokyo
NINS-RSCS
Tottori Univ.
ISSP
ISSP

Supercomputer Project Advisory Committee
KAWASHIMA, Naoki

ISSP (Chair person)4

Activity Report 2019 / Supercomputer Center, Institute for Solid State Physics, The University of Tokyo

NOGUCHI, Hiroshi
SUGINO, Osamu
OZAKI, Taisuke
TSUNETSUGU, Hirokazu
KATO, Takeo
KONDO, Takeshi
HIGUCHI, Yujihi
MORITA, Satoshi
FUKUDA, Masahiro
KAWAMURA, Mitsuaki
NAKAJIMA, Kengo
TSUNEYUKI, Shinji
IMADA, Masatoshi
HATANO, Naomichi
SUZUKI, Takafumi
YOSHIMOTO, Yoshihide
OKUMURA, Hisashi
HOSHI, Takeo
ONO, Tomoya
YOSHINO, Hajime
MOTOME, Yukitoshi
TOHYAMA, Takami
KITAO, Akio
ARITA, Ryotaro
IKUHARA, Yuichi
SHIBATA, Naokazu
AKAGI, Kazuto
YANASE, Yoichi
HATSUGAI, Yasuhiro
OKADA, Susumu
KOBAYASHI, Nobuhiko
NAKAYAMA, Takashi
HOTTA, Takashi
MATSUKAWA, Hiroshi
YAMAUCHI, Jun
HAGITA, Katsumi
KONTANI, Hiroshi
SAITO, Mineo
KAWAKAMI, Norio
YUKAWA, Satoshi
SUGA, Seiichiro
TATENO, Masaru
YASUDA, Chitoshi
OGATA, Masao
WATANABE, Satoshi

ISSP
ISSP
ISSP
ISSP
ISSP
ISSP
ISSP
ISSP
ISSP
ISSP
Univ. of Tokyo
Univ. of Tokyo
Toyota Riken/Waseda Univ.
Univ. of Tokyo
Univ. of Hyogo
Univ. of Tokyo
NINS-RSCS
Tottori Univ.
Univ. of Tsukuba
Osaka Univ.
Univ. of Tokyo
Tokyo Univ. of Sci.
Tokyo Tech.
Univ. of Tokyo
Univ. of Tokyo
Tohoku Univ.
Tohoku Univ.
Kyoto Univ.
Univ. of Tsukuba
Univ. of Tsukuba
Univ. of Tsukuba
Chiba Univ.
Tokyo Metropolitan Univ.
Aoyama Gakuin Univ.
Keio Univ.
National Defense Academy
Nagoya Univ.
Kanazawa Univ.
Kyoto Univ.
Osaka Univ.
Univ. of Hyogo
Univ. of Hyogo
Univ. of the Ryukyus
Univ. of Tokyo
Univ. of Tokyo

5

Activity Report 2019 / Supercomputer Center, Institute for Solid State Physics, The University of Tokyo

OKAMOTO, Yuko
HUKUSHIMA, Koji
NEMOTO, Koji
YABANA, Kazuhiro
HIDA, Kazuo
WATANABE, Kazuyuki
FURUKAWA, Nobuo
KUROKI, Kazuhiko
TAKANO, Hiroshi
YASUOKA, Kenji
TANAKA, Yukio
MASUBUCHI, Yuichi
KAWAMURA, Hikaru
KUSAKABE, Koichi
SHIRAI, Koun
SAKAI, Toru
ISHIBASHI, ShojiOTANI, Minoru
TOMITA, Yusuke
MIYASHITA, Seiji
SHIRAISHI, Kenji
OGUCHI, Tamio
KAWAKATSU, Toshihiro
KOBAYASHI, Kazuaki
TATEYAMA, Yoshitaka
KIM, KangOTSUKI, TomiMORIKAWA, Yoshitada
ODA, Tatsuki
OTSUKI, Junya
KOGA, Akihisa
SHIMOJO, Fuyuki
TAKETSUGU, Tetsuya
TSURUTA, Kenji
HAMAGUCHI, Satoshi
NISHIDATE, Kazume
KAGESHIMA, Hiroyuki
SATO, Tetsuya
ISHII, Fumiyuki
TATETSU, Yasutomi
YANAGISAWA, Susumu
SHUDO, Ken-ichi
OHMURA, Satoshi
NOGUCHI, Yoshifumi
NAKAMURA, Kazuma

Nagoya Univ.
Univ. of Tokyo
Hokkaido Univ.
Univ. of Tsukuba
Saitama Univ.
Tokyo Univ. of Sci.
Aoyama Gakuin Univ.
Osaka Univ.
Keio Univ.
Keio Univ.
Nagoya Univ.
Nagoya Univ.
Osaka Univ.
Osaka Univ.
Osaka Univ.
Univ. of Hyogo
AIST
AIST
Shibaura Inst. Tech.
JPS
Nagoya Univ.
Osaka Univ.
Tohoku Univ.
NIMS
NIMS
Osaka Univ.
Sophia Univ.
Osaka Univ.
Kanazawa Univ.
Okayama Univ.
Tokyo Tech.
Kumamoto Univ.
Hokkaido Univ.
Okayama Univ.
Osaka Univ.
Iwate Univ.
Shimane Univ.
Keio Univ.
Kanazawa Univ.
Meio Univ.
Univ. of the Ryukyus
Yokohama Natl. Univ.
Hiroshima Inst. Tech.
Shizuoka Univ.
Kyushu Inst. Tech.

6

Activity Report 2019 / Supercomputer Center, Institute for Solid State Physics, The University of Tokyo

GOHDA, Yoshihiro
HAMADA, Ikutaro
RAEBIGER, Hannes
WATANABE, Hiroshi
TADA, Tomofumi
TODO, Synge

1.4

Tokyo Tech.
Osaka Univ.
Yokohama Natl. Univ.
Keio Univ.
Tokyo Tech.
Univ. of Tokyo

Staﬀ

The following staﬀ members of the SCC-ISSP usually administrate the ISSP Supercomputer.
KAWASHIMA, Naoki
NOGUCHI, Hiroshi
OZAKI, Taisuke
SUGINO, Osamu
IDO, Kota
FUKUDA, Masahiro
HIGUCHI, Yuji
KAWAMURA, Mitsuaki
MORITA, Satoshi
YOSHIMI, Kazuyoshi
MOTOYAMA, Yuichi
YATA, Hiroyuki
FUKUDA, Takaki
ARAKI, Shigeyuki

Professor (Chair person)
Associate Professor
Professor
Professor
Research Associate
Research Associate
Research Associate
Research Associate
Research Associate
Project Researcher
Project Researcher
Technical Associate
Technical Associate
Technical Associate

2

STATISTICS (SCHOOL YEAR 2019)

2.1

System and User Statistics

In the following, we present statistics for operation time taken in the period from
April 2019 to March 2020 (SY 2019). In Table 2, we show general statistics ofthe supercomputer system in SY 2019. The total numbers of compute nodes in
System B and System C are 1891 and 252, respectively. Consumed disk points
amount to about 3% and 2% of the total consumed points in System B and SystemC, respectively.
In the left column of Fig. 2, availabilities, utilization rates, and consumed points
in Systems B and C are plotted for each month. Throughout the school year, theutilization rates were very high, exceeding 80% throughout most of the year.
The user statistics are shown in the right column of Fig. 2. The horizontal
axis shows the rank of the user/group arranged in the descending order of theexecution time (hour×nodes). The execution time of the user/group of the ﬁrst
rank is the longest. The vertical axis shows the sum of the execution time up to7

Activity Report 2019 / Supercomputer Center, Institute for Solid State Physics, The University of Tokyo

Table 2: Overall statistics of SY 2019
System B System C
total service time (×10 node·hours)
16242
2165
number of executed jobs
75681751444
3
total consumed points (×10 point)656
59
CPU points (×103 point)635
58
3
disk points (×10 point)
21
1
3
total exec. time (×10 node·hours)
14529
1843
availability
98.04%
98.06%
utilization rate
89.39%
85.13%
3

the rank. From the saturation points of the graphs, the numbers of “active” users
of Systems B and C are around 400 and 100, respectively. The maximum ranks in
the graphs correspond to the number of the users/groups that submitted at least
one job.

2.2

Queue and Job Statistics

Queue structures of Systems B and C in SY2019 are shown in Table 3. In System
B, users can choose from three types of compute nodes; jobs submitted to queues
with “cpu”, “acc”, and “fat” at the end of their queue names are submitted toCPU, ACC, and Fat nodes, respectively, while only CPU nodes are available in
System C. See Sec. 1.1 for a description of each type of compute node. The user
then has to choose the queue according to the number of nodes to use and theduration of their calculation jobs. Queue names starting with “F” are for jobs
taking 24 hours or less, while those starting with “L” can run much longer up
to 120 hours. More nodes are allotted to “F” queues in order to maximize theturnaround time of user jobs. The queue names starting with “i” are used forinteractive debugging of user programs and the elapsed time limit is 30 minutes.
The number following “F”, “L”, or “i” correspond to the number of nodes that canbe used by one user job. Note that F/B72acc queue were abolished in November
2018.
To prevent overuse of the storage, points are charged also for usage of disk
quota in the three systems, as shown in Table 4. Disk points are revised often for
optimal usage of the resources by examining usage tendencies each year.
Although we do not mention here in detail, to promote utilization of themassively parallel supercomputer, background queues for system B (“B4cpu”,“B36cpu”, “B144cpu”, “B18acc”, and “B2fat”) and background queues for system C (“B4cpu”, “B9cpu”, “B36cpu”) which charge no points for the jobs havealso been open.
The number of jobs, average waiting time, and total execution time in each8Activity Report 2019 / Supercomputer Center, Institute for Solid State Physics, The University of Tokyo

System B

14

106 node·hours

12
10
8
6
4
user exec. timegroup exec. time

2
0
0

100

200
300
400
Rank of user/group

500

600

System C

1.8
1.6

106 node·hours

1.4
1.2
1.0
0.8
0.6

user exec. timegroup exec. time

0.4
0.2

0

25

50
75
100
125
Rank of user/group

150

175

Figure 2: Left: Availabilities, utilization rates and point consumptions of eachmonth during SY 2019. Right: User statistics. The horizontal axis shows therank of the user/group arranged in the descending order of the execution time(hour×nodes). The vertical axis shows the sum of the execution time up to the
rank.

9

Activity Report 2019 / Supercomputer Center, Institute for Solid State Physics, The University of Tokyo

queue are shown in Table 5. In System B, a large portion of jobs have been executedin “F” queues. The largest amount of the execution time has been consumed in
the large-scale “F144cpu” queue, but substantial number of jobs were run in every
queue, suggesting that a wide variety of user needs are met by this queuing scheme.
In most of these queues, the queue settings meet the user’s tendencies in that thewaiting times are on the order of the elapsed-time limit.

2.3

Project for Advancement of Software Usability in Materials Science

From School Year 2015, the supercomputer center (SCC) has started “Project
for advancement of software usability in materials science”. In this project, forenhancing the usability of the supercomputer system in ISSP, we perform some
software-advancement activity such as implementing a new function to an existing
code, releasing a private code on Web, writing manuals. Target programs are
publicly oﬀered in December and selected in the review by the Steering Committeeof SCC. The projects are carried out by the software development team composedof three members in ISSP. In SY 2019, two projects were selected as listed inTable 6.

2.4

GPGPU Support Service

As noted in Sec. 1.1, ACC nodes with graphics processing units (GPU) were introduced in System B in School Year 2015. Since GPUs were introduced in the
ISSP Supercomputer center for the ﬁrst time, many programs developed or utilized by users of this center have not been programmed for GPU computing. To
help users take advantage of GPUs, the supercomputer center has started a service
for porting users’ materials science software to General Purpose GPUs (GPGPU).
After a call for proposals (which will usually be in December), target programs forthe next school year are selected by the Steering Committee of SCC. The porting
service is carried out on each program for about two months; the coding is performed by engineers from the computer vendor supplying the ISSP supercomputer
system, and ISSP staﬀ oversee the progress of the project and manage necessarycommunications with the proposer. Copyrights of the resulting software basicallybelong to the proposers, but the supported contents might be published underagreement with the proposer. In SY 2019, one project was selected as listed inTable 7.

Acknowledgments
The staﬀs would like to thank Prof. Takafumi Suzuki (now at University of Hyogo)
for developing WWW-based system (SCM: SuperComputer Management System)
for management of project proposals, peer-review reports by the SPAC committee,
and user accounts. We also thank Ms. Reiko Iwafune for creating and maintaining
a new WWW page of the ISSP Supercomputer Center.

10Activity Report 2019 / Supercomputer Center, Institute for Solid State Physics, The University of Tokyo

Table 3: Queue structures in SY 2019
System B
Memoryjob points
queue
Elapsed time # of nodes # of nodes
namelimit (hr)/job/queuelimit (GB)/(node·day)
F4cpu
24
1–4
216
120/node
1
L4cpu
120
1–4
108
120/node
1
F36cpu24
5–36
288
120/node1
L36cpu120
5–36
144
120/node
1
F144cpu24
37–1441008
120/node 72(144)/(# of nodes)∗L144cpu120
37–144144
120/node 72(144)/(# of nodes)∗i18cpu0.5
1–18
72
120/node1
F18acc24
1–18
180120/node2
L18acc120
1–18
90120/node
2
i9acc
0.5
1–9
36
120/node
2
F2fat
24
1–2
17
1000/node
4
L2fat
120
1–2
6
1000/node
4
i1fat
0.5
1
2
1000/node
4∗

For F/L144cpu queue, the number of occupied node increases in increments of 72
nodes.

System C
queue Elapsed time # of nodes # of nodes
namelimit (hr)/job/queue
F4cpu
24
1–4
54
L4cpu
120
1–4
18
0.5
1–4
18
i4cpu
F9cpu
24
5–9
36
L9cpu
120
5–9
18
F36cpu24
10–36
144
L36cpu
120
10–36
36
∗

Memorylimit (GB)170/node
170/node
170/node
170/node
170/node
170/node
170/node

job points/(node·day)
1
1
1
1
1
18(36)/(# of nodes)∗18(36)/(# of nodes)∗For F/L36cpu queue, the number of occupied node increases in increments of 18 nodes.

11Activity Report 2019 / Supercomputer Center, Institute for Solid State Physics, The University of Tokyo

Table 4: Disk points of Systems B and C

System B
System C∗∗

point/day/home
0.001 × θ(q − 300)
/work 0.0001 × θ(q − 3000)/home
0.001 × θ(q − 150)
/work 0.0001 × θ(q − 1500)

q is denoted in unit of GB.
θ(x) is equal to the Heaviside step function H(x) multiplied by x, i.e., xH(x).

Table 5: Number of jobs, average waiting time, total execution time, and average
number of used nodes per job in each queue.

queue# of Jobs

F4cpu
L4cpu
F36cpuL36cpuF144cpu
L144cpu
i18cpuF18acc
L18acc
i9acc
F2fat
L2fat
i1fat

169958
9051
26234
2295
9950
194
76809
201751
6661
6519
7310
753
3504

System B
Exec. Time # of nodes
Waiting Time(hour) (×103 node·hour)
12.85
1223.79
1.38
26.04
570.601.71
30.71
1342.06
8.54
64.56
812.259.57
33.30
7208.66
90.65
83.32
829.66109.860.52
106.568.74
3.93
831.483.07
11.86
536.87
3.28
0.15
2.31
3.94
24.27
75.56
1.28
30.50
26.75
1.37
0.22
0.54
1.00

System C
queue# of Jobs Waiting Time
Exec. Time # of nodes
3(hour) (×10 node·hour)
F4cpu
25640
12.06
268.51
1.54
L4cpu
1718
79.26
110.23
1.61
i4cpu
10297
0.06
2.86
2.63
F9cpu
2863
40.39
159.26
7.43
L9cpu
203
31.61
46.64
6.36
F36cpu2588
19.52
652.5824.94
L36cpu130
70.88
160.2424.37

12

Activity Report 2019 / Supercomputer Center, Institute for Solid State Physics, The University of Tokyo

Table 6: List of Project for advancement of software usability in materials science
for SY 2019.
Project Proposer
Shusuke Kasamatsu
Yamagata University
Naoki Kawashima
The University of Tokyo

Project Name
Coupling of extended ensemble sampling
and ﬁrst-principles solvers
Tensor Network Solver
for Quantum Lattice Problems

Table 7: List of supported software and project proposers for the GPGPU support
service for SY 2019.
Project Proposer
Project Name
Junya Otsuki
GPU acceleration of matrix calculations
Okayama University in the solver of Bethe-Salpeter equation

13