Recent approaches to machine learning of interatomic potentials seen from a perspective of plasma material interaction and primary r

Recent approaches to machine learning ofinteratomic potentials seen from a perspective ofplasma material interaction and primary radiation
damageBastiaan J. Braams, Centrum Wiskunde & Informatica (CWI),
Amsterdam, Netherlands

Presentation at 4th MoD-PMI, NIFS, 2019-06-19

Recent approaches to machine learning of interatomic potentials seen from a perspective of plasma material interaction and primary r
OutlineIntroduction: problems with force fields for materials and PMIAtomistic Modelling
New approaches from big data and machine learning
Related developments in machine learning
Supplement: Other uses of potential energy surfaces

Recent approaches to machine learning of interatomic potentials seen from a perspective of plasma material interaction and primary r
Introduction: problems with force fields for materials and PMI

Hydrogen retention in irradiated tungsten
IAEA Coordinated Research Project (CRP) on Plasma-wallinteraction with irradiated tungsten and tungsten alloys in fusion
devices (2013-2018). See https://www-amdis.iaea.org/CRP/.
Need to understand effect of radiation om microstructure andeffect of microstructure on hydrogen retention and migration.
Must use surrogate irradiation; need modelling to interpret
experimental data.
Most basic computations: primary radiation damage and hydrogenmigration.
Relatively short timescale. (Long timescale: segregation,corrosion.) Molecular dynamics is the main tool.

Recent approaches to machine learning of interatomic potentials seen from a perspective of plasma material interaction and primary r
Introduction: problems with force fields for materials and PMI

Problems with potentials for tungsten
Talk by A. Sand (Helsinki, with Kai Nordlund) at IAEA,
2017-11-16: “Energetic cascades in tungsten: sensitivity tointeratomic potentials and electronic effects.”“Potentials with largely similar point defect formation andmigration energies disagree regarding clustered fraction of defects
for high PKA energies. Some potentials predict only very smallclusters, others show formation of clusters of > 100 point defects.”“Why the different predictions, despite extensively fitted ’good’potentials??”
Discuss blending to short-range Ziegler-Biersack-Littmark (ZBL)
potential.
Many-body effects beyond embedded atom (EAM) approach.

Recent approaches to machine learning of interatomic potentials seen from a perspective of plasma material interaction and primary r
Introduction: problems with force fields for materials and PMI

Vacancy Clustered Fraction

Recent approaches to machine learning of interatomic potentials seen from a perspective of plasma material interaction and primary r
Introduction: problems with force fields for materials and PMI

Self-Interstitial Atom Clustered Fraction

Recent approaches to machine learning of interatomic potentials seen from a perspective of plasma material interaction and primary r
Introduction: problems with force fields for materials and PMI

Average Number Frenkel Pairs per CascadeRecent approaches to machine learning of interatomic potentials seen from a perspective of plasma material interaction and primary r
Introduction: problems with force fields for materials and PMI

Force fields for fusion materials and plasma-materialinteraction
The potential is (almost) everything; and that needs to bereflected in the effort.
Keep in mind the following target application: Primary radiation
damage in W-H-He. PKA event, melt region, resolidification. Moredifficult than pure W (see above); not as difficult as steel.
Quantum effects on the nuclear motion: barely ever relevant.
Electronic excitation beyond simple stopping: can be important,could be taken into account. (Langevin approach, potential
depends on electron temperature.)Recent approaches to machine learning of interatomic potentials seen from a perspective of plasma material interaction and primary r
Atomistic Modelling

Molecular dynamics: “F=ma”
N atoms; classical nuclei, positions x(i), 1 ≤ i ≤ N.
Interaction potential V (X ) (X ∈ R3N ).
Force F = −∇V .
1 ∂Vd 2 x(i)=−dt 2mi ∂x(i)Electron dynamics is gone. This is the Born-Oppenheimer(adiabatic) approximation.
For large molecules Pand condensed phase a local representation
may be used: V = i V0 (Xi ) where Xi are the collective nuclear
coordinates for a local environment of the i-th atom.

Recent approaches to machine learning of interatomic potentials seen from a perspective of plasma material interaction and primary r
Atomistic Modelling

Other uses of potential energy surfaces
Born-Oppenheimer approximation allows quantum nuclei; it is not
limited to semiclassical molecular dynamics.
Molecular spectroscopy: Eigenvalue problem HΨ = E Ψ for thenuclear wavefunction. Tractable for small molecules.
Diffusion Monte Carlo for the ground state nuclear wavefunction:
Random walk with birth and death processes.
1tr(Ae −βH ). Averages
Quantum statistics: < A >β = Z (β)
calculated using Path Integral Monte Carlo.

Ring Polymer Molecular Dynamics and variants; PIMC plus timeevolution. Model for nuclear quantum effects.
∂Quantum scattering: i~ ∂tΨ(X , t) = HΨ(X , t). Application toreaction dynamics is pretty much limited to 4-atom systems.

Recent approaches to machine learning of interatomic potentials seen from a perspective of plasma material interaction and primary r
New approaches from big data and machine learning

New Approaches from Big Data and Machine Learning
Machine Learning has brought specific methods, e.g. deep
convolutional neural networks (Vision, Go).
Machine Learning is also bringing a change of attitude...
Nothing wrong with optimizing over very many variables(Stochastic Gradient Descent).
Nothing wrong with lots of local minima, even inequivalent ones.
Don’t ask for a guaranteed global optimum.
(NN with 20 layers and 256 nodes per layer and a ReLU
nonlinearity has multiplicity of about 1010139 , being (256!)20 ; withtanh nonlinearity about 1011680 , being (2256 × 256!)20 .)Recent approaches to machine learning of interatomic potentials seen from a perspective of plasma material interaction and primary r
New approaches from big data and machine learning

Fitting or learning in the presence of symmetries
Example: Want to fit or learn f : RN → R, f (x) = z, using dataf (Xα ) = zα . Typical point x = (x1 , ..., xN ). Say that theunderlying true function is totally symmetric in the (xi )i . Options:(a) Ignore the symmetry, use any plausible model. (Maybe
replicate the data using symmetry.) Obtain symmetry via accuracy.
(b) Use explicit invariants of a good functional form. Example:
yk = pk (x) (where the pk are elementary symmetric polynomials
for 1 ≤ k ≤ N), then f (x) = g (y (x)) with some plausible modelfor g . Efficient; technically difficult for more complicatedsymmetries. (Braams+Bowman at Emory University.)(c) Use explicit invariants of an easily generalizable form. Example:
y = Sort(x), then f (x) = g (y (x)). Introduces nonsmoothness,often discontinuities. Obtain smoothness via accuracy.

Recent approaches to machine learning of interatomic potentials seen from a perspective of plasma material interaction and primary r
New approaches from big data and machine learning

GAP-SOAP approach of G. Csányi, B. Bartók et al.
Key reference: Bartók, Kondor, Csányi (2013) Phys Rev B 87.
Gaussian Approximation Potential (GAP), also referred to as
Kernel Ridge Regression, high-dimensional version of Radial Basis
Functions. Authors use both language of Machine Learning andlanguagePof function fitting, regression analysis.
f (X ) = α wα K (X , Xα ).
Smooth Overlap of Atomic Potentials (SOAP) kernel K (X , X 0 ).
R
X 7→ ρ, S(ρ, ρ0 ) = ρ(r )ρ0 (r )dr .
RK (X , X 0 ) = |S(ρ, R.ρ0 )|n dR, R ∈ O(3).
Integrals evaluated via spherical harmonic expansion.

Recent approaches to machine learning of interatomic potentials seen from a perspective of plasma material interaction and primary r
New approaches from big data and machine learning

SchNet, Deep Tensor network from TU Berlin
Key reference: Schütt, Sauceda, Kindermans, Tkatchenko, Müller(2018) J Chem Phys 148. Also Nature (2017).
Say N atoms. Each NN layer contains atomic feature vectors xi foreach atom (1 ≤ i ≤ N); positions are global parameters.
Transitions between layers, l → l + 1 (before the nonlinearity):
Dense atom-wise, xil+1 = W l .xil + b l ;Convolution feature-wise: xil+1 = (X l ? W 0l )i . Convolutions
depend on relative distances.
Smooth shifted SoftPlus instead of ReLU.
Weights to be fitted as functions of relative positions.

Recent approaches to machine learning of interatomic potentials seen from a perspective of plasma material interaction and primary r
New approaches from big data and machine learning

Spherical Wavelet Expansion approach led by S. Mallat
Key reference: Eickenberg, Exarchakis, Him, Mallat, Thiry (2018)
J Chem Phys 148.
General with respect to chemical elements.
Global transform, no explicit reference to local environments.
PSmoothed densities ρ(r ) = i ni g (r − ri ), smooth kernel g ;separate densities for core and valence electrons. (Optional bonddensities; ignore those.)m ; convolutions with
Solid harmonic wavelet basis functions ψj,ldensities ρ.

Now symmetrize with respect to rotations and translations ...
Finally multilinear regression.

Recent approaches to machine learning of interatomic potentials seen from a perspective of plasma material interaction and primary r
New approaches from big data and machine learning

Atomic cluster expansion by R. Drautz, Bochum
Key reference: Drautz (2019) PRB 99.
PLocal environment of atom i, descriptors Aiv = j φv (~rji ), wherethe φv are a family of basis functions: v = (nlm) and thenφv (~r ) =√4πRnl (k~r k)Ylm (~r /k~r k).

Cluster products involving Clebsch-Gordon coefficients(K )=i,~n,~lB

X~ × Ai,n1 ,l1 ,m1 · · · Ai,nK ,lK ,mK .
CG(~l, m)~mFinally Ei =P(K ) (K )c ~ B ~.
K ,~n,~l n~ ,li,~n ,lDrautz (2019) also describes a nonlinear version to overcome slow
convergence of the cluster expansion.

Recent approaches to machine learning of interatomic potentials seen from a perspective of plasma material interaction and primary r
New approaches from big data and machine learning

DeepMD approach led by E and Car, Princeton Univ.
Key reference: Zhang, Han, Wang, Car, E (2018) PRL 120.
PRecall V = i V0 (Xi ) where Xi are the collective nuclear
Coordinates for a local environment of the i-th atom.
Local environment is shifted to i-th atom, rotated in a
problem-dependent manner. Let j enumerate neighbouring atoms
within a cut-off distance.
Environment descriptor D(Xi ) = Sort{Dij } sorted by chemical
species and by distance Rij .
Dij = (1/Rij , xij /Rij2 , yij /Rij2 , zij /Rij2 )Obtain V0 as output of a deep neural network with inputs theordered (Dij )j .

Recent approaches to machine learning of interatomic potentials seen from a perspective of plasma material interaction and primary rNew approaches from big data and machine learning

DeepPot-SE, deep potential smooth edition, fromPrinceton University and IAPCM
Key reference: Zhang, Han, Wang, Saidi, Car, E, NIPS 2018.
Follow-on to the DeepMD work, but now with due respect forcontinuity, energy conservation and vector covariance.
Environment descriptors without rotation or sort:
Dij = sij (rij ) × (1, xij , yij , zij ); sij → 0 for large rij .
Two-stage NN: an encoding network and a fitting network.
Encoding network maps local environment to a feature spacepreserving point group and permutation symmetry.
Fitting network is fully connected feed-forward neural network withskip connections.

Recent approaches to machine learning of interatomic potentials seen from a perspective of plasma material interaction and primary r
Related developments in machine learning

Perspective from Machine Learning .. Introl .v (l−1) ).
Basic feed-forward neural network: v (l) = φ(Wl−1v ∈ R b ; φ e.g. pointwise ReLU; W ∈ Mat(b, b) to be learned.
Structured network: v ∈ X b , X ∼ RN with some structure. W hasa corresponding structure.
Example (Convolutional Neural Network, CNN): X ∼ Rm×n ,pixelized greyscale image. Elements of W are convolutions withcompact kernel.
Parameterized network: Data depend on parameters p, weights tobe learned as a function of p. (Linear regression formulation.)

Recent approaches to machine learning of interatomic potentials seen from a perspective of plasma material interaction and primary r
Related developments in machine learning

Invariants and Covariants
Vector spaces U and V , group G with representations on U andV . For g ∈ G we write g .u or g .v for the action of g on a typicalelement u ∈ U or v ∈ V .
Function f : U → V is covariant (equivariant) if for all g ∈ G andu ∈ U, f (g .u) = g .f (u).
(Invariants are the special case g .v ≡ v .)Example, the dipole moment d(X ), X ∈ R 3N . Invariant underSym(N), covariant under O(3).
P
Represent it by effective charge model: d(X ) = i wi (X )ri ; thenthe weight vector w ∈ R N is covariant under Sym(N) andinvariant under O(3).

Recent approaches to machine learning of interatomic potentials seen from a perspective of plasma material interaction and primary r
Related developments in machine learning

Point Cloud Convolutional Networks
Permutation equivariant neural networks (“Deep Sets”): Zaheer,Kottur, Ravanbhakhsh et al., NIPS 2017.
Structured and parameterized: X ∼ RN , v ∈ RN×b ; interpret asfeature vector of size b for each of N points in R3 . Parameters p:
positions ri (1 ≤ i ≤ N). Weights W represent local convolutions;depend on local distances.
Point cloud convolutional networks are covariant (equivariant)under Sym(N).
See also: [SpiderCNN convolutional filters on point sets: Xu, Fan,Xu, Zeng, Qiao, Proc ECCV 2018], [PointConv deep convolutions
on point sets: Wu, Qi, Fuxin, Arxiv 2018], other work onpermutation equivariant NN.

Recent approaches to machine learning of interatomic potentials seen from a perspective of plasma material interaction and primary r
Related developments in machine learning

Point Clouds with Additional Structure
Basic point cloud network has v ∈ RN×b ; feature vector of size bfor each of N points. Group is Sym(N); feature vectors are
unstructured. Imagine additional structure in the feature vector.
Additional group G of transformations on v , especiallypermutation group or SO(3); parameters and feature vector havedefinite transformation properties under G .
Group-Equivariant CNN: Cohen and Welling, ICML 2016.
Spherical Convolutional Neural Networks: Cohen, Geiger, Köhler,Welling, ICLR 2018.
Tensor Field Networks: Thomas, Schmidt, Kearnes et al., Arxiv,2018.
Gauge Equivariant CNN: Cohen, Weiler, Kicanaoglu, Welling,Arxiv, 2019.

Recent approaches to machine learning of interatomic potentials seen from a perspective of plasma material interaction and primary r
Related developments in machine learning

Conclusions
There are promising new approaches to force fields with inspirationfrom big data and machine learning. (No assessment here ofrelative merits.)
There are valuable related developments from machine learning
community inspired at least in part by application to atomistic
force fields.
BJB wish list for developments ...
Simultaneous learning of energy, dipole, quadrupole moment
through local charges with long-range interactions.
Fit or learn bands in solids invariant under SL(3, Z).
Learn local effective Hamiltonians for excited states.

Recent approaches to machine learning of interatomic potentials seen from a perspective of plasma material interaction and primary r
Supplement: Other uses of potential energy surfaces

Other uses of potential energy surfaces

Supplementary slides.

Recent approaches to machine learning of interatomic potentials seen from a perspective of plasma material interaction and primary r
Supplement: Other uses of potential energy surfaces

Molecular spectroscopy
Eigenvalue problem HΨ = E Ψ:
−X ~2i2mi∆i Ψ(X ) + V (X )Ψ(X ) = E Ψ(X )Configuration interaction approach (Hartree products):
Ψ(X ) =

Xcα Ψα (X )α(i)Ψα (X ) = Πi ψα(i) (x(i))
This provides the ro-vibrational spectrum. Tractable for small
molecules: e.g. 2(H2 O), CH3 OH; up to 9 atoms in our work.

Recent approaches to machine learning of interatomic potentials seen from a perspective of plasma material interaction and primary r
Supplement: Other uses of potential energy surfaces

Diffusion Monte Carlo
Ground state wavefunction: HΨ = E0 Ψ.
−X ~22mii∆i Ψ(X ) + V (X )Ψ(X ) = E0 Ψ(X )Steady state for reaction-diffusion equation:∂Ψ X ~2−∆i Ψ(X ) + V (X )Ψ(X ) = E0 Ψ(X )∂t2miiCan be solved in many dimensions using random walk with birthand death processes.
Result is ground state energy E0 ; plus sample from the groundstate wavefunction. (Sample |Ψ2 | via descendant weighting.)

Recent approaches to machine learning of interatomic potentials seen from a perspective of plasma material interaction and primary r
Supplement: Other uses of potential energy surfaces

Quantum statistics
Partition function Z (β) = tr(e −βH ). Thermal averages:< A >β =1tr(Ae −βH )Z (β)

Use e −βH = (e −(β/n)H )n ; H = T + V ; n → ∞. Let βn = β/n,ωn = 1/βn ~;Vn (X) =X
1(V (Xi ) + mωn2 (Xi+1 − Xi )2 )2iZZn (βn ) =Path Integral Monte Carlo.

e −βn Vn (X) dx

Recent approaches to machine learning of interatomic potentials seen from a perspective of plasma material interaction and primary r
Supplement: Other uses of potential energy surfaces

Ring Polymer Molecular Dynamics
Due to David Manolopoulos (Oxford). PIMC plus time evolution.
Classical hamiltonian:
Hn (x, p) = Vn (x) +X p2ii2mdx∂Hndp∂Hn=,=−dt∂pdt∂xSeen as a model for calculating the quantum Kubo correlation
function.
1c̃A,B (t) =βZ (β)

Z β0tr(e −(β−λ)H A(0)e −λH B(t))dλRecent approaches to machine learning of interatomic potentials seen from a perspective of plasma material interaction and primary rSupplement: Other uses of potential energy surfacesQuantum scattering

Time-dependent Schrödinger equation:i~∂Ψ(X , t) = HΨ(X , t)∂t

Wavepacket propagation in an unbounded domain. Application toreaction dynamics is pretty much limited to 4-atom systems.