Supercomputing Japan 2024
2024年3月12日 16:00 - 16:30 @ タワーホール船堀

ガラスとグラフニューラルネットワーク- 究極の長時間ダイナミクスへのデータ駆動科学による挑戦
glassy systems

芝隼人

専門：計算物性物理・分子シミュレーション

兵庫県立大学 大学院情報科学研究科
E-mail: shiba@gsis.u-hyogo.ac.jp「富岳」横 !

Acknowledgements
Supercomputing + Data Science
Information Technology Center, Univ. of Tokyo

Masatoshi
Hanai

Toyotaro
Suzumura

Takashi
Shimokawabe

Kengo
Nakajima

Fundings‣ 科学研究費補助金 基盤研究 (S) FY2019-2023「 （計算＋データ＋学習）融合によるエクサスケール時代の革新的シミュレーション手法」(h3-open-bdec)‣ 学際大規模情報基盤共同利用・共同研究課題 FY2019-2023 etc…

AI for HPC & HPC for AI
実験・観測

supercomputerSimulation

AI‣ AI による帰納的予測の精度向上‣ 予測結果の再利用 または 併用したシミュレーション技術開発‣ （実験・観測とのデータ同化）

3

本日の話

・物理学の研究対象としてのガラス̶ 超長時間のダイナミクス

・ガラスダイナミクスの機械学習予測

̶ この数年間の発展

・ガラスのGNN での（簡単な）学習速度測定- NVIDIA & AMD GPUs

4

ガラス =「流れなくなった液体」?「乱れた固体」?
“乱れ” (featureless structure) を扱う難しさ、深遠さ

2021年ノーベル賞 Giulio Parisi 「レプリカ対称性の破れ」

6

ピッチドロップ実験
＠Trinity College Dublin

begun in 1944 by an unknown colleague of theNobel prizewinner Ernest Walton while he was in
the physics department of Trinity College (Wikipedia).
2013 broadcasted by Prof. Shane Bergin7⌘ ⇠ 2 ⇥ 10 Pa · s
十数年おきの落下• Nature News
• The Independent (UK)• Radio Lab• Wall Street Journal• The Register• Circa• Huffington Post
• The Slate• Irish Times• RTE News• International Business Times• The Atlantic• Discover Magazine• The Australian• Scientific American• Times of India• NBC Bay Area
• National Geographic• Die Zeit
• Fox News
• NewScientist• Daily Mail
• Business Insider• Phys.org• CNET• ScienceNews• The Verge• GizMag• New Scientist

7

Science & Engineering of glasses© DeepMind 2020 ( V. Bapst et al. )https://deepmind.google/discover/blog/towards-understanding-glasses-with-graph-neural-networks/

Glass→ extremely long time
experiment > hours
simulation ~ μs8ガラス物理の（ひとつの）中心課題
構造（＝たった１枚の写真）の中から、究極的に長時間の「未来」が引き出せるか？

Ludovic Berthier and Giulio Biroli: Theoretical perspective on t596
9080
70
60
5040302010

初期構造
(now)

t=0？
t = τβ ∼ 0.03τα00102030405060708090

L. Berthier & G. Biroli, RMP (2011)

FIG. 7 (color online). Spatial map of single-particle displacementsin the simulation of a binary mixture of Lennard-Jones mixture in
two dimensions. Arrows show the displacement of each particle in a
trajectory of length comparable to the structural relaxation time.
The map reveals the existence of particles with different mobilities
during relaxation, but also the existence of spatial correlations
between these dynamic fluctuations.

不均一運動(future)single-particle displacements recorded during the simulationof a binary Lennard-Jones mixture in two dimensions. This
type of measurement, out of reach of most experimental
techniques that study the liquid state, reveals that dynamics
might be different from one particle to another (transiently).
More importantly, Fig. 7 also unambiguously reveals theexistence of spatial correlations between these dynamict = τα(1) The relaxation isrelaxation time scaletion or response functtial averaging over ttimes. (2) The relaxat
exponential, even loca
(Ediger, 2000) sugge
play, but definitely coerogeneous, with regi
average. Since superc
slow region will evenphysical characterizat
the determination ofities, as well as their t
A clearer and mor
neous character of thestudies. For example,
squared displacement
Fig. 5), time signalsspecific features that
resolved in both spacein Fig. 8. What do we
trajectories are not sm
sion of long periods oaround well-definedVibrations were prev
served at intermediate
ments of Fig. 5, but t
statistically widely dis
averaged quantities on
and direct measureme
fluctuations around th
A simple type ofmuch detail. When lolog(t)10ガラス物理の（ひとつの）中心課題
構造（＝たった１枚の写真）の中から、究極的に長時間の「未来」が引き出せるか？

Ludovic Berthier and Giulio Biroli: Theoretical perspective on t596
908070605040302010

初期構造(now)t=0？
t = τβ ∼ 0.03τα00102030405060

708090L. Berthier & G. Biroli, RMP (2011)

FIG. 7 (color online). Spatial map of single-particle displacementsin the simulation of a binary mixture of Lennard-Jones mixture in
two dimensions. Arrows show the displacement of each particle in a
trajectory of length comparable to the structural relaxation time.
The map reveals the existence of particles with different mobilities
during relaxation, but also the existence of spatial correlations
between these dynamic fluctuations.

不均一運動(future)single-particle displacements recorded during the simulationof a binary Lennard-Jones mixture in two dimensions. This
type of measurement, out of reach of most experimental
techniques that study the liquid state, reveals that dynamics
might be different from one particle to another (transiently).
More importantly, Fig. 7 also unambiguously reveals theexistence of spatial correlations between these dynamict = τα(1) The relaxation isrelaxation time scaletion or response functtial averaging over ttimes. (2) The relaxat
exponential, even loca
(Ediger, 2000) sugge
play, but definitely coerogeneous, with regi
average. Since superc
slow region will evenphysical characterizat
the determination ofities, as well as their t
A clearer and mor
neous character of thestudies. For example,
squared displacement
Fig. 5), time signals
specific features that
resolved in both spacein Fig. 8. What do we
trajectories are not sm
sion of long periods oaround well-definedVibrations were prev
served at intermediate
ments of Fig. 5, but t
statistically widely dis
averaged quantities onand direct measureme
fluctuations around th
A simple type ofmuch detail. When lolog(t)13ガラスの未来予測の機械学習構造（＝たった１枚の写真）の中から、究極的に長時間の「未来」が引き出せる？

Ludovic Berthier and Giulio Biroli: Theoretical perspective on t596
9080706050403020100

初期構造(now)

SVMs, GNNs,
Physics-based ML etc…

0102030405060708090L. Berthier & G. Biroli, RMP (2011)

FIG. 7 (color online). Spatial map of single-particle displacementsin the simulation of a binary mixture of Lennard-Jones mixture in
two dimensions. Arrows show the displacement of each particle in atrajectory of length comparable to the structural relaxation time.
The map reveals the existence of particles with different mobilities
during relaxation, but also the existence of spatial correlations
between these dynamic fluctuations.

不均一運動(future)single-particle displacements recorded during the simulationof a binary Lennard-Jones mixture in two dimensions. This
type of measurement, out of reach of most experimental
techniques that study the liquid state, reveals that dynamics
might be different from one particle to another (transiently).
More importantly, Fig. 7 also unambiguously reveals theexistence of spatial correlations between these dynamic(1) The relaxation isrelaxation time scaletion or response functtial averaging over t
times. (2) The relaxat
exponential, even loca
(Ediger, 2000) sugge
play, but definitely coerogeneous, with regi
average. Since superc
slow region will evenphysical characterizat
the determination ofities, as well as their t
A clearer and mor
neous character of thestudies. For example,
squared displacement
Fig. 5), time signals
specific features that
resolved in both spacein Fig. 8. What do we
trajectories are not sm
sion of long periods oaround well-definedVibrations were prev
served at intermediate
ments of Fig. 5, but t
statistically widely dis
averaged quantities on
and direct measureme
fluctuations around th
A simple type ofmuch detail. When lo15are thengroupedin this lower-dimensional space using Gaussian25
imensionality
reducer.
Notethatthisalgorithm
allowsfor
nonutional neural networks and graph neural networks . How- mixture models. A full description of this algorithm is given inr, in order
to train these algorithms,
data dimension.
linking structure
to theneartransformations
to a lower
Third,
theparticles Methods.
Supplementary
urethendynamics
had to beused.
Methods
that can autonomously
ThisGaussian
UML algorithm has two main parameters that need to be
regroupedin
thislower-dimensional
space
using
ect purely structural heterogeneities offer an unbiased fresh chosen: (i) the dimensionality of the bottleneck of themixturemodels. A full description of this algorithm is given in
k at the problem.
autoencoder c and (ii) the number of Gaussian components NG
Here
we show that a simple,
efﬁcient UML algorithm that we used to ﬁt the distribution in the lower-dimensional space. To
heSupplementaryMethods.
10 for detecting crystalline structure can beentlydeveloped
This UML algorithm has two main parameters thatneedbechoose
the todimensionalityof the bottleneck, we require that thernessed to detect structural heterogeneities in glasses. Our autoencoder reproduces at least 75% of the variance of the input
教師つき学習教師なし学習hosen:(i)the dimensionality
ofas input—uses
the bottleneck of theorithm—which
requiresonly a single snapshot
data. In Fig. 2a–c we show the fraction of variance explained
utoencoderc and(BOPs)(ii) thenumberof environments
Gaussian components
NGnd-order parameters
to encode
the local(FVE, see “Methods”)
by the autoencoder as a function of c forthe particles.
Combininga neural
network-based
autoencodersed
to ﬁt thedistribution
in the lower-dimensional
space. snapshots
To
動力学間のフィッティング
the chosenfrom all three 構造models. Basedon this例
ー構造の自動分類
or抽出
h Gaussian mixture models, it then autonomously designs a analysis, we choose c = 2 for both the binary hard spheres and thehooseof ofthedetecting
bottleneck,we require that theuctural theorderdimensionality
parameter capable
the largest
Wahnström
models, and c = 4 for the Kob–Andersen model. To例) 粒子運動距離
utoencoder
reproduces
atleast75%
ofthevarianceoftheinput
次元縮約,
クラスタ解析,GradCAM,
etc…
uctural variation in the system.
determine the number of Gaussians NG, we measure the BayesianWe
performanceof thealgorithm
three archetypical
ata.testIntheFig.
2a–c weshow
the onfractionof varianceexplained
Information
Criterion (BIC, see Supplementary Methods) as aLabels: Dynamics
ss forming systems: binary hard spheres, Wahnström, and function of N , and plot it for each snapshot in Fig. 2d–f. TheFVE, see “Methods”) by the autoencoder as a function of c forGE. Boattini et al., Nat. Commun.
Model
Input: Structure
he chosen snapshots from all three models. Based on thisTraining
次元縮約SVC, GNN,nalysis, we choose c = 2 for both thebinary
hard spheres and theq1
MLP, CNN,
Wahnström models, and c = 4 forqthe
Kob–Andersen model. To2rij
Q =and others
etermine the numberof Gaussians
NG, we measure the Bayesian
nformation Criterion (BIC, see SupplementaryMethods) as aq8Prediction
Theunction of NG, and plot it for each snapshot in Fig. 2d–f.
ガウス混合モデル

ガラスへの機械学習

arXiv:2311.11452 (2023)

1 Schematic representation of the unsupervised machine learning method. In this method, the local environment of a particle is encoded in a vectorof bond-order parameters, which is used as the input for an artiﬁcial neural network trained to reduce its dimensionality. The resulting distribution ofticle environments in the lower dimension is clustered using a Gaussian mixture model. Finally, particles are assigned a probability of belonging to one oftwo clusters, and colored accordingly.

「結晶らしさ」の強弱の分布抽出

NATURE COMMUNICATIONS | (2020)11:5479 | https://doi.org/10.1038/s41467-020-19286-8 | www.nature.com/naturecommunications. In this method, the local environment of a particle is encoded in a vector
network trained to reduce its dimensionality. The resulting distribution of

線形モデル ̶ SVM, LR, etc..
非線形モデル ̶ various NNs【他、半教師あり学習・強化学習など】

17

© DeepMind 2020
https://deepmind.google/discover/blog/towards-understanding-glasses-with-graph-neural-networks/

DeepMind の「挑戦」
基礎的な科学の問題への人工知能によるアプローチ
囲碁(AlphaGo)

脳科学

タンパク質構造 (AlphaFold)
ガラス

行列積演算 (AlphaTensor)
材料データ (GNoME)
18

頂点
粒子の種類
INPUT

GNNs by DeepMind [Nature Physics, 2020]

NATURE PHYSICS

頂点
粒子運動

GNNs

OUTPUTE PHYSICS

辺aa 位置関係
S(q, t )= 0.9 b0.80.1
S(q, t )= 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 1.00.50

学習モデル
SVM, CNN
物理モデル
0 SM
2 = 振動解析
10 10103tc

1010.70.60.50.40.30.20.13D KALJ @ T = 0.44Methods
GNN
CNN
機械学習
SVM
SM
DW
物理手法
PE

他の手法を圧倒的凌駕
ρ

ρ

1.0b4

100.50

T = 0.56
010 102d 1.00

1030.50ARTICLE

0.47

t （予測時間）

1040.44

T = 0.56

19

Our GNN — BOnd TArgeting Network (BOTAN)
頂点
粒子の種類

GNNs

INPUT

頂点
粒子運動
OUTPUT

辺
位置関係

辺
距離変化
10.03τα

BOTAN

0.8

ταρ

0.6

DeepMind GNN0.4

予測精度の「劇的」向上(Bapst et al.)0.2(a) T = 0.44
10-11003D KALJ
101210t10341010520

ガラスの未来予測の機械学習構造（＝たった１枚の写真）の中から、究極的に長時間の「未来」が引き出せる？

Ludovic Berthier and Giulio Biroli: Theoretical perspective on t5969080706050403020100

初期構造(now)

SVMs, GNNs,
Physics-based ML etc…

0102030405060708090

L. Berthier & G. Biroli, RMP (2011)

FIG. 7 (color online). Spatial map of single-particle displacementsin the simulation of a binary mixture of Lennard-Jones mixture in
two dimensions. Arrows show the displacement of each particle in a
trajectory of length comparable to the structural relaxation time.
The map reveals the existence of particles with different mobilities
during relaxation, but also the existence of spatial correlations
between these dynamic fluctuations.

不均一運動(future)single-particle displacements recorded during the simulationof a binary Lennard-Jones mixture in two dimensions. This
type of measurement, out of reach of most experimental
techniques that study the liquid state, reveals that dynamics
might be different from one particle to another (transiently).
More importantly, Fig. 7 also unambiguously reveals theexistence of spatial correlations between these dynamic→ 深層学習による新しい物理の開拓、予測技術の活用(1) The relaxation isrelaxation time scaletion or response functtial averaging over t
times. (2) The relaxat
exponential, even loca
(Ediger, 2000) sugge
play, but definitely coerogeneous, with regi
average. Since superc
slow region will evenphysical characterizat
the determination ofities, as well as their t
A clearer and mor
neous character of thestudies. For example,
squared displacement
Fig. 5), time signals
specific features that
resolved in both spacein Fig. 8. What do we
trajectories are not sm
sion of long periods oaround well-definedVibrations were prev
served at intermediate
ments of Fig. 5, but t
statistically widely dis
averaged quantities onand direct measureme
fluctuations around thA simple type ofmuch detail. When lo21

その後 — “Roadmap” review, preprint arXiv:2311.14752 (2023, submitted)
G. Jung, R. Alkemade, V. Bapst, D. Coslovich, L. Filion, F. Landes, A. J. Liu, F. S. Pezzicoli,
H. Shiba, G. Volpe, F. Zamponi, L. Berthier, and G. Biroli

Latest models
supervised machine learning
model
GlassMLP

MLPs with[Jung, Biroli, & Berthier]complex (input + loss)

Linear Regression[Boattini & Filion]

CNN[Fan & Ma]

SE(3)-equivariant GNN[F. Landes et al]

Geo-GNN[Z. Jiang et al]

BOTAN[H. Shiba et al]# params.
~ 650

linear fitting~ 1000CNN (ResNet)~ O(105)

GNN (node targets)~ O(106)

GNN (node targets)+ self attention
GNN (node + edgetargets)~ O(106)~ O(106)

効率な特徴抽出— 様々な物理的制約・拘束条件
・forward model の利用— 少ないデータ量

高い表現力（ ~ MB)

過学習の問題

大量の学習データセット必要（ ~ GB)

22

その後 — “Roadmap” review, preprint arXiv:2311.14752 (2023, submitted)
G. Jung, R. Alkemade, V. Bapst, D. Coslovich, L. Filion, F. Landes, A. J. Liu, F. S. Pezzicoli,
H. Shiba, G. Volpe, F. Zamponi, L. Berthier, and G. Biroli

Latest models

Prediction accuracy

supervised machine learning
model
GlassMLP

MLPs with[Jung, Biroli, & Berthier]complex (input + loss)[Boattini & Filion]

CNN[Fan & Ma]

SE(3)-equivariant GNN[F. Landes et al]

Geo-GNN[Z. Jiang et al]

BOTAN[H. Shiba et al]linear fitting# params.
~ 650

10.8~ 10000.6CNN (ResNet)

DeepBOCGlas

latest GNNs & LR

~ O(105)

DeepMind GNN⇢PLinear Regression(Pearson coeffs.)

SB
D

0.4

GNN (node targets)
GNN (node targets)+ self attention
GNN (node + edgetargets)~ O(106)
0.2

unsupervised ML~ O(106)~ O(106)

010 1

100

101

102
tt

103

104

105

23

Roadmap review — 将来展望
G. Jung, R. Alkemade, V. Bapst, D. Coslovich, L. Filion, F. Landes, A. J. Liu, F. S. Pezzicoli,H. Shiba, G. Volpe, F. Zamponi, L. Berthier, and G. Biroli, preprint arXiv:2311.14752 (2023, submitted)‣ ガラス動力学の「未来予測」による超長時間のサンプリング?

(超低温ガラス, “ultrastable” glass)input

prediction

predictionglass?

生成生成

生成AIAIAI- 2020 年頃からいくつかの生成モデル利用の試み

VAE, 自己回帰モデル
計算量的な困難、扱えるシステムサイズが極めて限定- 拡散モデル, Flow Matching など最新手法の活用？
24

GPU 学習速度評価- １基単体の学習のみ- NVIDIA Tesla V100 SXM (32 GB) @ UHyogo- NVIDIA A100 SXM (40 GB) on Wisteria-A @ UTokyo- NVIDIA A100 PCIe (40 GB) @ UTokyo- NVIDIA H100 HBM3 @ ITC, UTokyo- AMD Instinct MI100, MI210 PCIe @ ITC, UTokyo- DataLoader は測定から除外- 今回は、推論のベンチマークはなし
25

BOnd TArgeting Network (BOTAN)[Shiba et al., JCP 2023]

backpropagation

node input

node output
粒子運動

粒子の種類

ENDE
nodenodeABDE

ENedgeedge

edge input
相対位置x7edge output

粒子間距離変化(neighbors rij < 2.0σA )26

BOTAN GNN (MPNN) Benchmark
PyTorch 2.2.1 + PyTorch Geometric 2.5x 3.4x 2.6

ROCm
6.0.2(bulid)

CUD 11.8 , 12.1, or 12.3

27

NVIDIA H100 (HBM3) benchmarking

xgemm

28

NV H100 (HBM3)
TF32 tensor 989 TFLOPsvsAMD MI210 (PCIe)FP32/64 matrix 45.3 TFLOPs29

ガラスの深層学習

まとめ（にかえて）・超長時間のMDデータのGNN 学習 → 「最高精度」の予測- 何をAIが「理解」しているのか- どういった新たな「計算」「サンプリングに」使えるか

・今後、重要になりうること- 表現力 + 少データでの学習 ̶ 教師なし学習との組み合わせ- 大規模MD データへの活用- 生成モデルとのドッキング
H. Shiba, et al., J. Chem. Phys. 158, 184503 (2023).

・GNN の性能評価- multi-arch Benchmarking に簡便

Code & Dataset https://github.com/h3-Open-BDEC/pyg_botan
G. Jung, R. Alkemade, V. Bapst, D. Coslovich, L. Filion, F. Landes,A. J. Liu, F. S. Pezzicoli, H. Shiba, G. Volpe, F. Zamponi, L. Berthier, and G. Biroli,
preprint arXiv:2311.14752 (2023)
Dataset on Zenodo: https://zenodo.org/records/10118191

33

Thank you !