Activity Report 2021 / Supercomputer Center, Institute for Solid State Physics, The University of Tokyo

1

OUTLINE

1.1

Supercomputer System

In School year 2021 (SY 2021), the ISSP supercomputer center provided users with
System B - Dell PowerEdge C6525/R940 system named “ohtaka” and System
C - HPE SGI 8600 system named “enaga”. System B is a massively-parallel
supercomputer with two types of compute nodes: 8 “Fat” nodes and 1680 “CPU”
nodes. “Fat” nodes are each comprised of four Intel Xeon Platinum 8280 CPUs
(28 cores/CPU) and 3 TB of memory per node. “CPU” nodes have two AMD
EPYC 7702 CPUs (64 cores/CPU) and 256 GB of memory per node. System B
achieves about 6.881 PFLOPS in theoretical peak performance with high powereﬃciency. The subsystem comprised of only CPU nodes ranks 87st in the Nov.
2020 Top 500 List, which is a ranking based on total performance measured by
the HPL benchmark. The compute nodes communicate to each other throughHDR100 Inﬁniband and are connected in fat tree topology. SY 2021 was the second
year of the operation of the current System B. System C is a massively-parallel
supercomputer with 252 “CPU” nodes, which have two Intel Xeon Gold 6148
CPUs (20 cores/CPU) and 192 GB of memory. System C achieves 774 TFLOPS
in theoretical peak performance. SY 2021 was the last year of the operation of thecurrent System C. For further details, please contact ISSP Supercomputer Center(SCC-ISSP).
[Correspondence: center@issp.u-tokyo.ac.jp]

Figure 1: Supercomputer System at the SCC-ISSP

1

Activity Report 2021 / Supercomputer Center, Institute for Solid State Physics, The University of Tokyo

1.2

Project Proposals

The ISSP supercomputer system provides computation resources for scientists
working on condensed matter sciences in Japan. All scientiﬁc staﬀ members (including post-docs) at universities or public research institutes in Japan can submit
proposals for projects related to research activities on materials and condensed
matter sciences. These proposals are peer-reviewed by the Advisory Committee
members (see Sec. 1.3), and then the computation resources are allocated basedon the review reports. The leader of an approved project can set up user accounts
for collaborators. Other types of scientists, including graduate students, may also
be added. Proposal submissions, peer-review processes, and user registration are
all managed via a web system.
The computation resources are distributed in a unit called “point”, determined
as a function of available CPU utilization time and consumed disk resources. There
were calls for six classes of research projects in SY 2021. The number of projects
and the total number of points that were applied for and approved in this school
year are listed in Table 1.
• Class A is for trial use by new users; proposals for Class A projects are
accepted throughout the year.
• Proposals for projects in Classes B (small), C (mid-size), E (large-scale), andS (exceptional) can be submitted twice a year. Approved projects in Classes
A, B, C, and E continue to the end of the school year.
• In Class D, projects can be proposed on rapidly-developing studies that needto perform urgent and relatively large calculations. An approved projectcontinues for 6 months from its approval.
• Class S is for projects that are considered extremely important for the ﬁeldof condensed matter physics and requires extremely large-scale computation.
The project may be carried out either by one research group or cooperatively
by several investigators at diﬀerent institutions. A project of this class should
be applied with at least 10,000 points; there is no maximum. We requiregroup leaders applying for Class S to give a presentation on the proposal to
the Steering Committee of the SCC-ISSP. Class S projects are carried out
within one year from its approval.
• Project leaders can apply for points so that the points for each system do
not exceed the maximum point shown in this table.
In addition, from SY 2016, ISSP Supercomputer has been providing 20% of
its computational resources for Supercomputing Consortium for Computational
Materials Science (SCCMS), which aims at advancing parallel computations incondensed matter, molecular, and materials sciences on the 10-PFlops K Computer
and the exascale post-K project. From SY 2020, about 10% of the computational
resources have been provided for SCCMS. Computational resources have also beenallotted to Computational Materials Design (CMD) workshops, as well as CCMS
hands-on workshops.

2Activity Report 2021 / Supercomputer Center, Institute for Solid State Physics, The University of Tokyo

Table 1: Classes of research projects in SY 2021. Total points listed in this table
are rounded. In Class D, we collect information about the projects ended in eachsemester. Note that the maximum points of Classes A and D in the ﬁrst semester
had been changed to those listed in the second semester since May 2021.

Class

A
B
C
D
E
S
SCCMS
Total

Class

A
B
C
D
E
S
SCCMS
Total

Maximum
PointsSys–B Sys–C
100
50
800
100
8k
1k
10k
1k
24k
3k
–
–

Maximum
PointsSys–B Sys–C
40
100
800
40
8k
400
10k
400
24k
1.2k
–
–

First semester (Apr.-Sep.)
Application # ofTotal points
Proj.
AppliedApproved
Sys–B Sys–C Sys–B Sys–C
any time
12
1.2k
0.5k
1.2k
0.5k
twice a year
58
40.9k
4.5k
23.2k
3.4k
twice a year
116 725.7k
71.1k 372.6k 50.8k
any time
00000
twice a year
8 187.0k
24.0k 110.0k 17.3k
twice a year
0
0
0
0
0
11
26.1k
5.0k
26.1k
5.0k
205 980.9k 105.1k 533.1k 77.0k

Second semester (Oct.-Mar.)
Application # ofTotal points
Proj.
AppliedApproved
Sys–B Sys–C Sys–B Sys–C
any time
5
0.5k
0.2k
0.5k
0.2k
twice a year
38
27.2k
0.8k
14.7k
0.7k
twice a year
53 327.6k
8.5k 159.8k
6.6k
any time
3
16.0k
0
13.5k
0
twice a year
6 136.0k
7.2k
83.0k
5.5k
twice a year
0
0
0
0
0
11
28.0k
2.4k
28.0k
2.4k
116 535.2k
19.1k 299.5k 15.3k

3Activity Report 2021 / Supercomputer Center, Institute for Solid State Physics, The University of Tokyo

1.3

Committees

In order to fairly manage the projects and to smoothly determine the system
operation policies, the Materials Design and Characterization Laboratory (MDCL)of the ISSP has organized the Steering Committee of the MDCL and the Steering
Committee of the SCC-ISSP, under which the Supercomputer Project Advisory
Committee (SPAC) is formed to review proposals. The members of the committees
in SY 2021 were as follows:

Steering Committee of the MDCLKAWASHIMA, Naoki
HIROI, Zenji
OZAKI, Taisuke
NOGUCHI, Hiroshi
UWATOKO, Yoshiya
SUGINO, Osamu
KUBO, Momoji
ONO, Tomoya
YAMAURA, Jun-ichi
TAKAHASHI, Hiroki
MOTOME, Yukitoshi
HOSHI, Takeo
ISHIWATA, Shintaro
HASEGAWA, Masashi
NAKATSUJI, Satoru

ISSP (Chair person)
ISSP
ISSP
ISSP
ISSP
ISSP
Tohoku Univ.
Kobe Univ.
Tokyo Tech.
Nihon Univ.
Univ. of Tokyo
Tottori Univ.
Osaka Univ.
Nagoya Univ.
ISSP

Steering Committee of the SCC-ISSP
KAWASHIMA, Naoki
NOGUCHI, Hiroshi
OZAKI, Taisuke
SUGINO, Osamu
TSUNETSUGU, Hirokazu
KATO, Takeo
YAMASHITA, Minoru
MORITA, Satoshi
HIGUCHI, Yuji
FUKUDA, Masahiro
IDO, Kota
KAWAMURA, Mitsuaki
NAKAJIMA, Kengo
HATANO, Naomichi
MOTOME, Yukitoshi
ONO, Tomoya

ISSP (Chair person)
ISSP
ISSP
ISSP
ISSP
ISSP
ISSP
ISSP
ISSP
ISSP
ISSP
ISSP
Univ. of Tokyo
Univ. of Tokyo
Univ. of Tokyo
Kobe Univ.

4

Activity Report 2021 / Supercomputer Center, Institute for Solid State Physics, The University of Tokyo

TODO, Synge
KUBO, Momoji
OBA, Fumiyasu
WATANABE, Hiroshi
YOSHINO, Hajime
OKUMURA, Hisashi
HOSHI, Takeo
YOSHIMI, Kazuyoshi
YATA, Hiroyuki
FUKUDA, Takaki

Univ. of Tokyo
Tohoku Univ.
Tokyo Tech.
Keio Univ.
Osaka Univ.
NINS-RSCS
Tottori Univ.
ISSP
ISSP
ISSP

Supercomputer Project Advisory Committee
KAWASHIMA, Naoki
OZAKI, Taisuke
NOGUCHI, Hiroshi
SUGINO, Osamu
TSUNETSUGU, Hirokazu
YAMASHITA, Minoru
KATO, Takeo
MORITA, Satoshi
HIGUCHI, Yuji
FUKUDA, Masahiro
IDO, Kota
KAWAMURA, Mitsuaki
NAKAJIMA, Kengo
HATANO, Naomichi
MOTOME, Yukitoshi
ONO, Tomoya
TODO, Synge
KUBO, Momoji
OBA, Fumiyasu
WATANABE, Hiroshi
YOSHINO, Hajime
OKUMURA, Hisashi
HOSHI, Takeo
TSUNEYUKI, Shinji
SUZUKI, Takafumi
YOSHIMOTO, Yoshihide
TOHYAMA, Takami
KITAO, Akio
ARITA, Ryotaro
IKUHARA, Yuichi
SHIBATA, Naokazu
AKAGI, Kazuto

ISSP (Chair person)
ISSP
ISSP
ISSP
ISSP
ISSP
ISSP
ISSP
ISSP
ISSP
ISSP
ISSP
Univ. of Tokyo
Univ. of Tokyo
Univ. of Tokyo
Univ. of Tsukuba
Univ. of Tokyo
Tohoku Univ.
Tokyo Tech.
Keio Univ.
Osaka Univ.
NINS-RSCS
Tottori Univ.
Univ. of Tokyo
Univ. of Hyogo
Univ. of Tokyo
Tokyo Univ. of Sci.
Tokyo Tech.
Univ. of Tokyo
Univ. of Tokyo
Tohoku Univ.
Tohoku Univ.

5

Activity Report 2021 / Supercomputer Center, Institute for Solid State Physics, The University of TokyoYANASE, Yoichi
HATSUGAI, Yasuhiro
OKADA, Susumu
KOBAYASHI, Nobuhiko
NAKAYAMA, Takashi
HOTTA, Takashi
MATSUKAWA, Hiroshi
YAMAUCHI, Jun
HAGITA, Katsumi
KONTANI, Hiroshi
SAITO, Mineo
KAWAKAMI, Norio
YUKAWA, Satoshi
SUGA, Seiichiro
TATENO, Masaru
YASUDA, Chitoshi
OGATA, Masao
WATANABE, Satoshi
HUKUSHIMA, Koji
NEMOTO, Koji
YABANA, Kazuhiro
FURUKAWA, Nobuo
KUROKI, Kazuhiko
YASUOKA, Kenji
TANAKA, Yukio
MASUBUCHI, Yuichi
KUSAKABE, Koichi
SHIRAI, Koun
SAKAI, Toru
ISHIBASHI, ShojiOTANI, Minoru
TOMITA, Yusuke
SHIRAISHI, Kenji
OGUCHI, Tamio
KAWAKATSU, Toshihiro
KOBAYASHI, Kazuaki
TATEYAMA, Yoshitaka
KIM, KangOTSUKI, TomiMORIKAWA, Yoshitada
ODA, Tatsuki
OTSUKI, Junya
KOGA, Akihisa
SHIMOJO, Fuyuki
TAKETSUGU, Tetsuya

Kyoto Univ.
Univ. of Tsukuba
Univ. of Tsukuba
Univ. of Tsukuba
Chiba Univ.
Tokyo Metropolitan Univ.
Aoyama Gakuin Univ.
Keio Univ.
National Defense Academy
Nagoya Univ.
Kanazawa Univ.
Kyoto Univ.
Osaka Univ.
Univ. of Hyogo
Univ. of Hyogo
Univ. of the Ryukyus
Univ. of Tokyo
Univ. of Tokyo
Univ. of Tokyo
Hokkaido Univ.
Univ. of Tsukuba
Aoyama Gakuin Univ.
Osaka Univ.
Keio Univ.
Nagoya Univ.
Nagoya Univ.
Osaka Univ.
Osaka Univ.
Univ. of Hyogo
AIST
AIST
Shibaura Inst. Tech.
Nagoya Univ.
Osaka Univ.
Tohoku Univ.
NIMS
NIMS
Osaka Univ.
Sophia Univ.
Osaka Univ.
Kanazawa Univ.
Okayama Univ.
Tokyo Tech.
Kumamoto Univ.
Hokkaido Univ.

6

Activity Report 2021 / Supercomputer Center, Institute for Solid State Physics, The University of Tokyo

TSURUTA, Kenji
Okayama Univ.
HAMAGUCHI, Satoshi
Osaka Univ.
NISHIDATE, Kazume
Iwate Univ.
KAGESHIMA, Hiroyuki
Shimane Univ.
ISHII, Fumiyuki
Kanazawa Univ.
TATETSU, Yasutomi
Meio Univ.
YANAGISAWA, Susumu Univ. of the Ryukyus
SHUDO, Ken-ichi
Yokohama Natl. Univ.
OHMURA, Satoshi
Hiroshima Inst. Tech.
NOGUCHI, Yoshifumi
Shizuoka Univ.
NAKAMURA, Kazuma
Kyushu Inst. Tech.
GOHDA, Yoshihiro
Tokyo Tech.
HAMADA, Ikutaro
Osaka Univ.
RAEBIGER, Hannes
Yokohama Natl. Univ.
KAWARABAYASHI, Tohru Toho Univ.
KATO, Yusuke
Univ. of Tokyo
NASU, Joji
Tohoku Univ.
HOTTA, Chisa
Univ. of Tokyo
ISOBE, Masaharu
Nagoya Inst. Tech.
HARADA, Ryuhei
Univ. of Tsukuba
TAMURA, Ryo
NIMS
TANAKA, Shu
Keio Univ.
TADA, Tomofumi
Kyushu Univ.

1.4

Staﬀ

The following staﬀ members of the SCC-ISSP usually administrate the ISSP Supercomputer.
KAWASHIMA, Naoki
NOGUCHI, Hiroshi
OZAKI, Taisuke
SUGINO, Osamu
IDO, Kota
FUKUDA, Masahiro
HIGUCHI, Yuji
KAWAMURA, Mitsuaki
MORITA, Satoshi
YOSHIMI, Kazuyoshi
MOTOYAMA, Yuichi
YATA, Hiroyuki
FUKUDA, Takaki
ARAKI, Shigeyuki

Professor (Chair person)
Associate Professor
Professor
Professor
Research Associate
Research Associate
Research Associate
Research Associate
Research Associate
Project Researcher
Project Researcher
Technical Specialist
Technical Specialist
Project Academic Specialist

7Activity Report 2021 / Supercomputer Center, Institute for Solid State Physics, The University of Tokyo

2

STATISTICS (SCHOOL YEAR 2021)

2.1

System and User Statistics

In the following, we present statistics for operation time taken in the period from
April 2021 to March 2022 (SY 2021). In Table 2, we show general statistics of thesupercomputer system in SY 2021. The total numbers of compute nodes in System
B “ohtaka” and System C “enaga” are 1688 and 252, respectively. Consumed disk
points amount to about a few percent of the total consumed points in both System
B and System C.
Table 2: Overall statistics of SY 2021

total service time (×103 node·hours)
number of executed jobs
total consumed points (×103 point)
CPU points (×103 point)
disk points (×103 point)
total exec. time (×103 node·hours)
availability
utilization rate

System B
ohtaka14396
456984
498
480
18
12980
97.34%
90.14%

System C
enaga
1601
46166
30
29
2
1367
96.23%
85.47%In Fig. 2, availabilities, utilization rates, and consumed points in Systems B
and C are plotted for each month. Throughout the school year, the availability
and the utilization rates were very high : the availability and the utilization rates
exceed about 90% and 80% throughout most of the year, respectively.
The user statistics are shown in Fig. 3. The horizontal axis shows the rank of theuser/group arranged in the descending order of the execution time (hour×nodes).
The execution time of the user/group of the ﬁrst rank is the longest. The verticalaxis shows the sum of the execution time up to the rank. From the saturation
points of the graphs, the numbers of “active” users of Systems B and C are around
400 and 100, respectively. The maximum ranks in the graphs correspond to thenumber of the users/groups that submitted at least one job.

2.2

Queue and Job Statistics

Queue structures of Systems B and C in SY2021 are shown in Tables 3 and 4,
respectively. In System B “ohtaka”, users can choose from two types of computenodes; jobs submitted to queues with “cpu” and “fat” at the end of their queue
names are submitted to CPU and Fat nodes, respectively, while only CPU nodes
are available in System C “enaga”. See Sec. 1.1 for a description of each type ofcompute node. The user then has to choose the queue according to the number

8Activity Report 2021 / Supercomputer Center, Institute for Solid State Physics, The University of Tokyo

Table 3: Queue structures of System B in SY 2021

queue
name
F1cpu
L1cpu
F4cpu
L4cpu
F16cpuL16cpuF36cpuL36cpuF72cpuL72cpuF144cpu
L144cpu
i8cpu
F2fat
L2fat
i1fat

Elapsed time
limit (hr)
24
120
24
120
24
120
24
120
24
120
24
120
0.5
24
120
0.5

System B, ohtaka# of nodes # of nodes/job/queue
1
600
1
300
2–4
216
2–4
108
5–16
288
5–16
144
17–36
72
17–36
36
72
576
72
288
144
432
144
144
1–8
72
1–2
7
1–2
3
1
1

Memorylimit (GB)230/node
230/node
230/node
230/node
230/node
230/node
230/node
230/node
230/node
230/node
230/node
230/node
230/node
2900/node2900/node2900/nodejob points/(node·day)
1
1
1
1
1
1
1
1
1
1
1
1
1
4
4
4

of nodes to use and the duration of their calculation jobs. Queue names startingwith “F” are for jobs taking 24 hours or less, while those starting with “L” canrun much longer up to 120 hours. More nodes are allotted to “F” queues in orderto maximize the turnaround time of user jobs. The queue names starting with“i” are used for interactive debugging of user programs and the elapsed time limit
is 30 minutes. The number following “F”, “L”, or “i” correspond to the numberof nodes that can be used by one user job. Although we do not mention here indetail, to promote utilization of the massively parallel supercomputer, backgroundqueues (queue name starting with “B”) for Systems B and C which charge nopoints for the jobs have also been open.
To prevent overuse of the storage, points are charged also for usage of disk
quota in the three systems, as shown in Table 5. Disk points are revised often for
optimal usage of the resources by examining usage tendencies each year.
The number of jobs, average waiting time, and total execution time in eachqueue are shown in Tables 6 and 7. In System B, a large portion of jobs havebeen executed in “F” queues. The largest amount of the execution time has been
consumed in the large-scale “F72cpu” queues for ohtaka, respectively. However,substantial number of jobs were run in every queue, suggesting that a wide variety
of user needs are met by this queuing scheme. In most of these queues, the queue
settings meet the user’s tendencies in that the waiting times are on the order ofthe elapsed-time limit.

9

Activity Report 2021 / Supercomputer Center, Institute for Solid State Physics, The University of Tokyo

System B

System C
Availability, Utilization [%]

100

90
80
Availability
Utilization

70

90
80

60

60

80

8

60
40
20
0
04

05

06

07

08

09 10
Month

11

12

01

02

6
4
2
0
04

03

Availability
Utilization

70

Point consumption[×103 points]

Point consumption[×103 points]

Availability, Utilization [%]

100

05

06

07

08
Month

09

10

11

12

Figure 2: Availabilities, utilization rates and point consumptions of each month
during SY 2021.

System B

System C
1.4

12
1.2
10

8

106 node·hours

106 node·hours

1.0

6

0.8

0.6
4
user exec. timegroup exec. time2

user exec. timegroup exec. time

0.4

0.2
0

100

200
300
400
Rank of user/group

500

600

0

20

40

60
80 100 120
Rank of user/group

140

160

Figure 3: User statistics. The horizontal axis shows the rank of the user/grouparranged in the descending order of the execution time (hour×nodes). The verticalaxis shows the sum of the execution time up to the rank.

10Activity Report 2021 / Supercomputer Center, Institute for Solid State Physics, The University of Tokyo

Table 4: Queue structures of System C in SY 2021

queue
name
F4cpu
L4cpu
i4cpu
F9cpu
L9cpu
F36cpuL36cpu∗Elapsed time
limit (hr)
24
120
0.5
24
120
24
120

System C, enaga# of nodes # of nodes/job/queue
1–4
54
1–4
18
1–4
18
5–9
36
5–9
18
10–36
144
10–36
36

Memorylimit (GB)170/node
170/node
170/node
170/node
170/node
170/node
170/node

job points/(node·day)
1
1
1
1
1
18(36)/(# of nodes)∗18(36)/(# of nodes)∗For F/L36cpu queue, the number of occupied node increases in increments of 18 nodes.

Table 5: Disk points of Systems B and C

System B ohtaka
System C enaga

∗
∗

/home
/work
/home
/work

point/day
0.001 × θ(q − 600)
0.0001 × θ(q − 6000)
0.001 × θ(q − 150)
0.0001 × θ(q − 1500)

q is denoted in unit of GB.
θ(x) is equal to the Heaviside step function H(x) multiplied by x, i.e., xH(x).

2.3

Project for Advancement of Software Usability in Materials Science

From School Year 2015, the supercomputer center (SCC) has started “Project
for advancement of software usability in materials science”. In this project, forenhancing the usability of the supercomputer system in ISSP, we perform some
software-advancement activity such as implementing a new function to an existing
code, releasing a private code on Web, writing manuals. Target programs are
publicly oﬀered in December and selected in the review by the Steering Committeeof SCC. The projects are carried out by the software development team composedof three members in ISSP. In SY 2021, three projects were selected as listed inTable 8.

11

Activity Report 2021 / Supercomputer Center, Institute for Solid State Physics, The University of Tokyo

Table 6: Number of jobs, average waiting time, total execution time, and average
number of used nodes per job in each queue of System B.

queue# of Jobs

F1cpu
L1cpu
F4cpu
L4cpu
F16cpuL16cpuF36cpuL36cpuF72cpuL72cpuF144cpu
L144cpu
i8cpu
F2fat
L2fat
i1fat

211125
10781
73536
5619
21718
1621
1633
87
7773
133
3023
134
91096
3673
282
522

System B, ohtaka
Waiting Time
Exec. Time # of nodes(hour) (×103 node·hour)
16.62
320.551.00
60.45
289.72
1.00
11.69
1175.58
2.87
17.66
400.572.53
15.21
1505.98
10.18
79.48
663.189.93
67.66
315.88
27.74
51.37
93.18
21.63
33.00
3494.1072.00
68.15
373.0472.00
144.0017.02
2062.21141.08677.79144.00
0.09
42.54
3.23
28.21
24.77
1.10
39.31
15.96
1.23
0.50
0.06
1.00

Table 7: Number of jobs, average waiting time, total execution time, and average
number of used nodes per job in each queue of System C.

queue# of Jobs

F4cpu
L4cpu
i4cpu
F9cpu
L9cpu
F36cpuL36cpu

23663
727
7775
1434
77
1551
62

System C, enaga
Waiting Time
Exec. Time # of nodes(hour) (×103 node·hour)
5.24
185.74
1.78
50.21
80.59
1.59
0.05
2.64
2.33
6.11
68.23
7.22
11.94
22.05
6.68
8.13
301.11
29.58
10.02
28.94
20.32

12

Activity Report 2021 / Supercomputer Center, Institute for Solid State Physics, The University of Tokyo

Table 8: List of Project for advancement of software usability in materials science
for SY 2021.
Project Proposer
Minoru Otani
AIST CD-FMat
Takeo Hoshi
Tottori University

2.4

Project Name
Improvement of Quantum ESPRESSO
implementing the ESM RISM method
Uniﬁed platform of experiment-data analysis
for 2D material structure

ISSP Data Repository

From School Year 2021, the supercomputer center (SCC) has started to operate
ISSP Data Repository (ISSP-DR) for accumulating and utilizing research data inmaterials science. GitLab is used as the data management system, and a portalsite is provided as a data registration and search system for the registered data.
By using ISSP-DR, it is possible to store and publish research data used in papers and datasets useful in the ﬁeld of condensed matter science. Users of ISSP
Supercomputer are welcome to apply for and use ISSP-DR.

Acknowledgments
The staﬀs would like to thank Prof. Takafumi Suzuki (now at University of Hyogo)
for developing WWW-based system (SCM: SuperComputer Management System)
for management of project proposals, peer-review reports by the SPAC committee,
and user accounts. We also thank Ms. Reiko Iwafune for creating and maintaining
a new WWW page of the ISSP Supercomputer Center.

13