Institute for NanoScience Design, Osaka University
大阪大学ナノサイエンスデザイン教育研究センター
Masaaki Geshi
下司 雅章
35th CMD workshop, supercomputer course
2019, September 2nd (Mon.)

Contents
 Necessity of large-scale computing
 Methods of the large-scale computing(Mainly

parallelization)
 Today’s supercomputers Future direction of large-scale computingNecessity of large-scale computing
 Systems we want to study have many atoms.
 The computational cost increases N3, N6, or

exponentially.(N is the number of atoms, electrons,…)
 Some physical quantities needs a huge computational
cost in spite of a small system(high accuracy).
 There is scientific and/or industrial significanceobtained by large-scale computing.

Problems raised by large-scalecomputing Because of huge date must be handled, huge HDD andfast IO is needed. (~30PB in K computer, ~150PB in
Fugaku(post-K))
 The problem of data analysis including visualization.
 In some cases, data analysis costs more time thancomputational time.
…

When we perform large-scalecomputing,…
 It costs huge computational time.

 In some cases, huge physical memory of CPU isneeded.

 In order to solve these problems, we have to improve

software codes to reduce the computational time andmemory.

 Parallel computing is definitely necessary.

Methods for large-scale computing
 Speed-up techniques Vectorization→SIMD(many cores) Parallelization
 Order-N method(reduce the cost o(Np)→o(N)(p=1))
 Use of specialized machine or accelerator(GPU).
 These are used in a code depending on one’s necessity.
The easiest way to speed up is to be able to realize it bychoosing proper compile options(vectorization, Automaticparallelization, optimization,…).
Since it is not possible to speed up the code
automatically, we have to tune the code significantly.

Concept of Flynn’s taxonomy
Single data

Multiple data

SISD

SIMD

Singleinstruction

DATA

DATA

instruction

instructionPE

MISD
Multiple
instruction

PE

PE

PE

MIMD
DATA

DATA
instruction

instruction

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

Vectorization
 A vector processor, or array processor, is a CPU that

implements an instruction set containing instructions
that operate on one-dimensional arrays of data called
vectors(row, column, or diagonal elements, etc…).
 Vectorization is the method that we program a code toconstruct regularly-arrayed data, and process the data
at a time by the vector CPU.
 Vectorization realizes the reduction of a CPU time.
 Today, this concept is included in SIMD.

Vector processing
do i=1,imax
a(i) = c(i) ＋d(i)b(i) = c(i) ×d(i)enddo

Scalarprocessinga(1) = c(1) ＋d(1)b(1) = c(1) ×d(1)a(2) = c(2) ＋d(2)b(2) = c(2) ×d(2)：：a(imax) = c(imax) ＋d(imax)b(imax) = c(imax) ×d(imax)

We tune to increase the partbeing processed at the deepest
do-loop in multi do-loop.
Vector processinga(1) = c(1) ＋d(1)a(2) = c(2) ＋d(2)::a(imax) = c(imax) ＋d(imax)
b(1) = c(1) ×d(1)b(2) = c(2) ×d(2)：：b(imax) = c(imax) ×d(imax)“a” array“b” arrayOrder-N
 Realized by algorithm→No approximation Ex.: Screened-KKR
 Realized by approximation of Hamiltonian→Approximation by localization of interaction. This type of
methods were actively studied in 90’s.
 Roughly speaking, matrices are block-diagonalized or including

similar method.
 N×N→N×Nlocal(fixed). If N increases, Nlocal not increase. (likea tight-binding approximation) The accuracy of the approximation can be adjustabledepending on a required accuracy.
 One of the famous method is a Divide and conquer method（分割統治法）. This is used in “OpenMX(Ozaki)”, “DC(Kobayashi
and Nakai)”,CONQUEST(Bowler, Miyazaki, Gillan)…

Parallelization
 This is the method to shorten the elapse time bysharing task with more than one CPU withoutchanging the processing time of the task.
 This is used to share big memory(array).
 SIMD(Single instruction, multiple data),
MIMD(Multiple instruction, multiple data)Elapse time

Concept of parallelization
rank0

rank1

rank2

rank3

WORK1

WORK1

WORK1

WORK1

WORK2-1

WORK2-2 WORK2-3 WOEK2-4

Adjacent Communication
Global Communication

WORK3 WORK3 WORK3 WORK3
Point: Increase parallelizable time anddecrease communication time.

Redundant
computation (allCPUs compute thesame calculation)
Parallelcomputation（different CPU
computes
different
calculation)
Redundant
computation

Problems in parallelization
 The ratio of parallelizable part in an elapse time on

1CPU sequential job is conclusive（Amdahl’ law）.
 When we parallelize a software code, the data
communication between nodes must occur. If thenumber of nodes increases, the communication time
between nodes costs more than the computational
time.(In the case of classical MD simulation, this is a
crucial problem.)Amdahl’s Law
Speed up ratio

αP ≡t1 Elapse time of one job on 1 coret N core

Elapse time of one job on N core

 1 
1


(α P ) max = t1 ×=
 tN 
P
 core  max (1 − P) +N coreP: parallelizable time in a sequential job

Upper bound of parallelizable ratio

For examples,
Not parallelizable（1 sec.）{

parallelizable（99 sec.）
1+0.99=1.99 sec.(almost 50 times faster)
Even weWe
usehave
100 processors,
to increase we
thecannot
parallelizableobtain 100
faster speed up.
parttimes
as possible.
We have to decrease the nonparallelizable part as possible.

…

100 processors{

We feel that the merit of parallelization
does not increase even if the number ofprocessors increases more and more.

…

1000 processors

1+0.099=1.099 sec.（almost 91 times
faster）
Although we uses 1000
processors, the efficiency does
not become 100 times even!!!

Gustafson-Barsis’ law
Speed-up ratio ≤ n + (1 − n) s
n: the number of coress: the ratio of sequential computation(this partcannot parallelize.)

In this law, the communication time
among nodes does not considered.

Indices of parallelization efficiency
 Strong scaling－The overall problem size (number ofatoms or electrons etc...) is fixed and the number ofprocessors is increased(based on Amdahl’s law).
 Weak scaling－The problem size per processor is fixed
and the overall problem size is increased withincreasing processors(based on Gustafson’s law). The
processing of one processor is not changed. The ideal
situation is the computation time should be constanteven if the increase of the number of processors. If thecomputation time increases, we can see the nonparallel part remains, and if the adjacentcommunication time increases, the method of thecommunication has a problem.
Reference: http://www.r-ccs.riken.jp/library/event/tokuronB_180406.html

Example of Amdahl’s law(1)Ordinary calculations using PC clusterExample of Amdahl’s law(2)Calculations using high-end PC cluster or supercomputers

Example of Amdahl’s law(3)K computer: 705,024 cores

Calculations using high-end supercomputers

Example of Amdahl’s law(4)Speed up ratio αp

Calculations using top class supercomputers in the world

Number of cores
K computer: 705,024 cores

Summit: 2,397,824 cores
Sunway TaihuLight: 10,649,600 cores

Even if you are just a user, you need to be aware of the parallelization efficiencyof the software you use because you must effectively
use machine time(machine point, or cost(usage fee))you can use.
 you need to understand the characteristics of thecomputer and make efforts to increase the calculation
efficiency.
 you need to choose the best compile option to makeexecutable file.
 you need to choose the best parameters for the best
performance of the software to obtain the best
scientific results.

Parallel programming languages
 MPI(message passing interface) The most versatile method.
 Both core-to-core and node-to-node communication canbe handled.
 The programming may not be easy.
 OpenMP Only core-to-core communication.
 The programming may be easier than that of MPI.
(within non-complex case)
 PVM

｝

 HPF(High Performance Fortran)

Now, these are seldom used.

Example of parallelization with MPI
program main
include (mpif.h)…
call MPI_INIT(IERR)
call MPI_COMM_SIZE(MPI_COMM_WORLD,NPROCS,IERR)
call MPI_COMM_RANK(MPI_COMM_WORLD,MYRANK,IERR)…
ista, iend are the first arraycall para_range(1, n, nprocs, myrank, ista, iend)
numbers allocated to eachdo i=ista, iendprocessor.
…
Each processor has only allocated elements ofa(i)=a(i)+value
the array a() .
enddo
call MPI_ALLREDUCE(a,a1,n_element,
Sum up all elements of the array a()
for all processors and distributed it& MPI_DOUBLE_PRECISION,
for all processors (all processors have
& MPI_SUM, MPI_COMM_WORLD, IERR)
the same value of a()).
…
call MPI_FINALIZE(IERR)
It is necessary to rewrite so as to calculate only the part…
allocated to each node and to communicate the data.
end
Execution: mpirun (or mpiexec) –np 4(# of parallel) ./a.out

Example of parallelization with OpenMP
program main
implicit none
integer omp_get_thread_num,IFor simple parallelization, we only need to insert
double precision z(100), a, x(100), y
directives, so if we compile without parallelization
do i= 1, 100z(i) = 0.0
options the directives are commented out, sox(i) = 2.0
basically we do not need to change the originalend docode.
a = 4.0
y = 1.0
print *, “Welcome to the parallel world”
Directive of the start of parallel processing!$omp parallel
print *, “Excuted on”, omp_get_thread_num()call daxpy(z, a, x, y)!$omp end parallel
Directive of the end of parallel processing
end program mainsubroutine daxpy(z, a, x, y)
integer I
double precision z(100), a, x(100), y!$omp parallel dodo i = 1, 100z(i) = a * x (i) + y
enddo
returnend

Do parallel of just below do loop

Execution: We do not use a special command.
Set environmental value“OMP_NUM_THREADS=4” (# of parallel)

The difference between MPI and
OpenMP• OpenMP can be parallelizable only for cores whichshare the memory. We cannot make a large-scaleparallelization code by using OpenMP only.
• The typical software codes which correspond to thelarge-scale parallelization are tuned by using MPI for
its main part.
• The role of OpenMP is to assist the speed up of thecode which is parallelized by using MPI.
• Even if you do not develop the software code, you haveto know the details of the code in order to get
maximum performance.

The other environments to develop
parallel computing codes using GPU
 CUFFT, CUBLAS,…
 Only call libraries to accelerate specific parts. The other parts do not be
accelerated.
 OpenACC
 a programming standard for parallel computing developed by Cray,
CAPS, NVIDIA and PGI, designed to simplify parallel programming ofheterogenious CPU/GPU system.
 Very similar with OpenMP. In near future, OpenMP and OpenACC
may be merged.
 Fortran and C are supported.
 CUDA a parallel computing platform and programming model created by
hard
NVIDIA and implemented by the graphics processing units (GPUs)
that they produce. （only PGI Fortran can used.）） OpenCL
 a framework for writing programs that execute across heterogeneous
platforms consisting of central processing units (CPUs), graphics
processing units (GPUs), DSPs and other processors.
easy

OpenACC(Open Accelerator)
program picalc
implicit none
integer, parameter :: n=1000000
integer :: ireal(kind=8) :: t, pi
pi = 0.0!$acc parallel loop
do i=0, n-1
t = (i+0.5)/npi = pi + 4.0/(1.0 + t*t)
end do!$acc end parallel loop
print *, 'pi=', pi/n
end program picalc

As the development
environment, the best results
can be obtained by improving
according to the message issued
by the compiler.

http://www.nvidia.co.jp/object/openacc-gpu-directives-jp.html
http://www.cms-initiative.jp/ja/events/2014-haishin
http://www.r-ccs.riken.jp/library/event/tokuronB_180406.html(The 14th lecture given by Dr. A. Naruse is useful.)The cutting-edge techniques Avoid memory wall problem(cache control) Computing power>data transfer from memory tocomputing unit(CPU)→reuse the data on caches
 Pipeline processing
We have to know details of Continuous access in do loop
hardware to make highly loop unrolling
optimized software codes.
 Divide data into blocks
 Use the highly-optimized libraries
…
Please see these sites; http://www.r-ccs.riken.jp/library/event/tokurona_170406.html
http://www.r-ccs.riken.jp/library/event/tokuronB_180406.html(only Japanese)

Memory wall problem
Old computer

CPU
MemoryToday’s computer
Computation
performance(number ofclocks) was slow（~10MHz)The memory can
not be accesseduntil a certain
time passes

Sending data from memory to the
CPU was slow, and the computation
speed was also slow, so it was not the
problem at “data transfer speed ≈calculated speed”.

CPU
MemoryThe computation
performance
becomes extremely
fast （~a few GHz）
The fact thatmemory cannot be
accessed only for a
fixed time does notchange much.

Sending data from the memory to the CPU
is slow, but since the computing speed has
become extremely fast, the CPU cannot wait“the transfer speed << compute speed”(supplement with cache etc…).
30

Memory wall problem

Impotant!: Data transfer is themost expensive at all levels.

Today’s computer

CPU

メモリ

High speed Main memory
O(ns)
L1

L1

Co
re

Co
re

L1

L1

L1

L1

L1

L1

L1

Co
re

O(10 ns)

Co
re

Co
re

Co
re

Co
re

Co
re

Co
re

L1

register ~ KB

L2 cacheCo
reO(100 ns)

cache

Co
re

Co
re

Co
re

Co
re

Co
re

Co
re

Co
re

Co
re

Co
re

Co
re

L1

L1

L1

L1

L1

L1

L1

L1

L1

L1

O(10 ms)

10KB~10MB
．．．
L1

Co
re

Main memory

L2 cache

HDD

GB~100GB．．． Core

L1

100GB~TB

Main memory

Large
Once the data is sent from the memory to
If you do not understand and develop acapacity
the CPU, the code is programmed to reuse
complex CPU configuration, theit as much as possible and reduce theperformance will not be achieved.
number of calls to the memory as much asThe time required for data transfer frompossible.Data transfer between thethe main memory to the CPU is 100
memory and the register or cache is fast, times longer than access to the data on
but the capacity is small.

the register.

31

High performance Python
 In place of Fortran and C / C ++, which have been used inscientific computing in the past, HPC software development
has been advanced in the USA by using Python.
 Python has overwhelming code readability compared to
Fortran and C / C ++.
→It is more likely that the maintenance of the code will be
continued without reading by experts.
 We do not need write everything in Python. If we know that
another language is faster than Python for a certain part, weshould use the part in the original. We combine Python andthe other language and improve the ease of maintenance.（Itis said that Python is a "glue language" because it is alanguage that "sticks" programs in multiple languages.）Analysis of the computational time inorder-N tight-binding method
A electronic
structure partoccupies 97.6%
of total elapse
time in asequential job.

Highparallelizability
is expected.

Parallelization by means of MPI
 We used MPI_ALLREDUCE and MPI_BCAST tocommunicate data.（Although this is quite simple andincludes useless data communication, the speed-up ratio
was good.) We divided do-loop as outer as possible.
 In this method, we can make all matrix elements as long as
we determine the atomic positions. These data determinedfrom 1 MD step are stored in CPU memory of all node.
Although almost all data are useless, we do not need tocommunicate the data among nodes and there is no loss ofdata communication.

Elapse time of each part as afunction of the number of CPU

SGI Origin 3800

Elapse time as a function of thenumber of CPU
M. Geshi et al, J. Phys. Soc. Jpn. 72 ,2880 (2003).

Parallelization ratio
M. Geshi et al, J. Phys. Soc. Jpn. 72 ,2880 (2003).

P=0.988
This is not enough today. To achieve the best
performance, it is necessary to increase theparallelization efficiency to the limit.

Today’s supercomputers

Classes of parallel computers Multicore computing
 a processor that includes multiple execution units ("cores") on
the same chip.
 Symmetric multiprocessing
 a computer system with multiple identical processors thatshare memory and connect via a bus.
 Distributed computing
 a distributed memory computer system in which the processing
elements are connected by a network.(Cluster computing,Massive parallel processing, Grid computing) Specialized parallel computers Within parallel computing, there are specialized parallel
devices that remain niche areas of interest.
(GPGPU,Application-specific integrated circuits, Vector
processors)
From http://en.wikipedia.org/wiki/Parallel_computingThe latest top 500 supercomputer
list (2018.11)
http://www.top500.org/

IBM
Cray

USA

GPU

USA

GPU

China

Designed for data and AI

ChinaIntel Xeon(many-cores)

USA

GPU
Intel Xeon(many-cores)

Japan

GPU
By the way, K-computer is 18,
Oakforest PACS is 14.

USA
USA

GPU
40

The latest top 500 supercomputer
list (2019.6)(Latest)
http://www.top500.org/

IBM製

USA

GPU

USA

GPU
By the way, Oakforest-PACS is 16,
and K-computer is 20.
(The usage of the K-computer
stopped at August 16, and it was
shutdown at the end of August.)

China
China

Cray製

USA

GPU
USA
Japan

USA

Intel XeonPhi(This architecture was gone.)

GPU
GPU

41

Linpack
 A software library for performing numerical linear

algebra on digital computers.
 It is used for the performance assessment ofsupercomputers. In practice, High-Performance
Linpack(HPL) is used.
 It makes use of the BLAS libraries for performing basic
vector and matrix operations.
 The benchmark used in the LINPACK Benchmark is tosolve a dense system of linear equations.

This benchmark shows nothing more than
one side of performance of supercomputers.

HPC Challenge
The HPC Challenge benchmark consists of basically 7 tests:
 HPL - the Linpack TPP benchmark which measures the floating point rate of
execution for solving a linear system of equations.
 DGEMM - measures the floating point rate of execution of double precision
real matrix-matrix multiplication.
 STREAM - a simple synthetic benchmark program that measures sustainable
memory bandwidth (in GB/s) and the corresponding computation rate forsimple vector kernel.
 PTRANS (parallel matrix transpose) - exercises the communications wherepairs of processors communicate with each other simultaneously. It is a useful
test of the total communications capacity of the network.
 RandomAccess - measures the rate of integer random updates of memory(GUPS)
 FFT - measures the floating point rate of execution of double precisioncomplex one-dimensional Discrete Fourier Transform (DFT).
 Communication bandwidth and latency - a set of tests to measure latency andbandwidth of a number of simultaneous communication patterns; basedon b_eff (effective bandwidth benchmark).

http://www.hpcchallenge.org/

HPCG Benchmark(From 2014)
The High Performance Conjugate Gradients (HPCG) Benchmark project is an effort to create
a new metric for ranking HPC systems. HPCG is intended as a complement to the High
Performance LINPACK (HPL) benchmark, currently used to rank the TOP500 computing
systems. The computational and data access patterns of HPL are still representative of some
important scalable applications, but not all. HPCG is designed to exercise computational anddata access patterns that more closely match a different and broad set of important
applications, and to give incentive to computer system designers to invest in capabilities thatwill have impact on the collective performance of these applications.
HPCG is a complete, stand-alone code that measures the performance of basic operations ina unified code:•Sparse matrix-vector multiplication.
•Vector updates.
•Global dot products.
•Local symmetric Gauss-Seidel smoother.
•Sparse triangular solve (as part of the Gauss-Seidel smoother).
•Driven by multigrid preconditioned conjugate gradient algorithm that exercises the key kernels
on a nested set of coarse grids.
•Reference implementation is written in C++ with MPI and OpenMP support.

http://www.hpcg-benchmark.org/index.html

HPCG ranking(2019.6)(Latest)
USA
USA
Japan
USA
Japan

China

Japan
USA

45

Green500(省エネスパコンリスト)
 The purpose of the Green500 is to provide a ranking of themost energy-efficient supercomputers in the world. For
decades, the notion of "performance" has beensynonymous with "speed" (as measured in FLOPS). Thisparticular focus has led to the emergence ofsupercomputers that consume egregious amounts ofelectrical power and produce so much heat thatextravagant cooling facilities must be constructed to ensure
proper operation. In addition, the emphasis on speed as theultimate metric has caused other metrics such as reliability,
availability, and usability to be largely ignored. As a result,
there has been an extraordinary increase in the total cost ofownership (TCO) of a supercomputer.

http://www.green500.org/

Green 500(2019.6)(Latest)
Japan
GPU

USA

GPU

USA

GPU

Japan

GPU
GPU

USA
Japan

This list fluctuates
violently every time.
That is whyemphasis is placedon the developmentof this technology.

GPU
GPU

GPU

USA

These machines balanceenergy saving andcomputing performance.

https://www.top500.org/green500/

47

Graph500 (graph analysis for big data)

http://www.graph500.org/

Graph500 (2019 June)(Latest)
Performance evaluation of "graph analysis" used for analysis of big data

http://www.graph500.org/

post-K “Fugaku”project
このイメージは、現在表示できません。

https://www.r-ccs.riken.jp/jp/post-k/project.html

Published specifications of post-K
Programming Environment

Nodes

System software

Storages
1st level

2nd level
3rd level

HPCI supercomputer situation
 The highest use hope isthe name FX 100. The
highest use hope is thename FX 100. At present,
it is scheduled tosuspend operation in FY
2018, but extension for
one year is being studied.
 Since there aresupercomputers besides
these (supercomputer of
ISSP etc…), there is stillmuch computing
resources.

K computer is working on 24h/365days
 K computer was active for public use on 24hours/365 days

for general users.

 Different from K-computer, almost all the top classmachines of the top500 ranking are for closed users and areused for limited purposes.

 The several rankings showed that K computer is almighty.
 Users may optimize their code for K computer easier thanfor the other ones(in the sense that it does not use a
special architecture like GPU.).

Fugaku will be operated in the same way as K-computer.

Future direction of large-scalecomputingFrom the above rankings
 From now on, both performance and energy-saving areimportant (Summit makes energy saving and computing
performance highly compatible).
 The trend of many-core system with not so high frequency
CPU is still going on. →Fugaku project is proceeding in thisdirection.
 Or many supercomputers have GPU and realize energy
saving.
 As GPU is prominent in the field of AI and machine learning,
this architecture will continue to be adopted in the future.
 However, I do not know whether supercomputers made up
of GPU are effective in all fields or not.
 Probably, many supercomputers designed for data analysis
and AI will appear after now.

However,…
 Effective performances measured by LINPACK or HPCG

benchmarks are not necessarily accordance with theperformance of our software codes. Usually, we spend somuch effort to get better performance on supercomputersthan typical PC clusters.
→The post-K machine, Fugaku, will be constructed forgetting better performance for scientific application thanbenchmark programs (Co-design). (However, it may be thenational mission to get the first prize of top 500 for
taxpayers.)
 The effort goes beyond knowledge and techniques ofphysicists, chemist, biologists,… We need those of computerscientists and numerical mathematicians and need tocollaborate them.
 Fugaku will be also a computer for general purpose. It may beinferior to a dedicated machine.(cf. Anton (supercomputer for
performing classical MD) is 100~1000 faster than normal
supercomputers.)

Future direction1
 The tendency of architecture is multi cores or many

cores (including SIMD,GPU, …).
 It is not easy to use all cores effectively.

(As personal opinion, I think that I do not have to worry about using all the cores and it isenough to get the best efficiency to obtain the calculation result.) Is it reasonable and proper to devote many researchers’

effort to develop parallel efficiency? (very serious
problem!!!) For DFT calculations, it may be difficult to speed-up
the software by using GPU without breakthrough.
 We have to perform feasibility studies continuously tofollow a leading-edge architecture.

Future direction2
 The United States develops both architectures andprogramming language frameworks. We should
contribute those.
 It is too much hard-pressed for physicists and chemists
to cover computer science or numerical analysis.
→it is needed to establish a system to promote to work
together with computational scientists and computerscientists or experts of numerical analysis.
 Still now, the guarantee of accuracy of calculation isinadequacy. There is no standard for summation of datafrom cores (threads) on MPI or OpenMP,… We must
treat it carefully, otherwise our calculation may become
meaningless.

Future direction3
 Today’s amassed skills through K computer can be usedduring the next 10 years. (In Japan, Fugaku project, andthe next project)
 However, the development of supercomputer must be
stop in near future.(Prof. Hiraki said it was 2029. The
reason is the necessity of huge electric power, failure ofMoors’ law, etc…)
 We should consider a new idea to get high performanceof our own calculations for both software and hardware
as soon as possible. We may undergo a paradigm shift
before 2029.
 For the time being, data transfer will continue tobe the most costly at all levels.

MateriApps

This portal site provides the information of applications that are used in materials
science. You can find your desired application. “MateriApps LIVE” gives you anenvironment to perform open-source software codes without any installation
process.

Finally, We should advance the development of human resources

to be able to use supercomputers and/or HPC machines
effectively.
 The relation between the ranking of top500 and thesoftware code we use is not necessarily clear. We need tohave the knowledge of the architecture of thesupercomputers and/or the HPC machines if we’d like tomaximize the performance of our software codes.
 We should cultivate people who can develop the softwarecode.
 We should cultivate people who can effectively use
scientific software codes to progress the researches.
（CMD-WS, distance learning etc…)

Related texts• These books are for developers.
• Integrates technologies for development on massively parallel computerssuch as supercomputers.
• Lecture videos and texts are available on the following site(Japanese only).
https://www.r-ccs.riken.jp/library/event/tokuronA_2019.html
https://www.r-ccs.riken.jp/library/event/tokuronB_180406.html
62