Lattice QCD at the Exascale
The 2nd R-CCS International Symposium
K to Fugaku
RIKEN Center for Computational ScienceFebruary 17-18, 2020

N.H. Christ
Columbia University

Outline• Overview of Lattice QCD• Algorithms and architectures• Recent calculations– Muon g-2– KL  + - / 0  e+ e– KL-KS mass difference• Exascale plans

2nd R-CCS Symposium - 02/17/2020(2)

The RBC & UKQCD collaborations
BNL and BNL/RBRC
Yasumichi Aoki (KEK)
Taku Izubuchi
Yong-Chull Jang
Chulwoo Jung
Meifeng LinAaron Meyer
Hiroshi Ohki
Shigemi Ohta (KEK)Amarjit Soni
UC Boulder
Oliver Witzel
CERN
Mattia Bruno
Columbia University
Ryan Abbott
Norman Christ
Duo Guo
Christopher Kelly
Bob Mawhinney
Masaaki Tomii
Jiqun Tu

University of Liverpool
Nicolas Garron

Bigeng Wang
Tianle Wang
Yidi Zhao
University of Connecticut
Tom Blum
Dan Hoying (BNL)
Luchang Jin (RBRC)
Cheng Tu
Edinburgh University
Peter Boyle
Luigi Del Debbio
Felix Erben
Vera Gülpers
Tadeusz Janowski
Julia Kettle
Michael MarshallFionn Ó hÓgáin
Antonin Portelli
Tobias Tsang
Andrew Yong
Azusa Yamaguchi

MITDavid Murphy
Peking University
Xu Feng
University of Regensburg
Christoph Lehner (BNL)
University of Southampton
Nils Asmussen
Jonathan Flynn
Ryan Hill
Andreas Jüttner
James Richings
Chris Sachrajda
Stony Brook University
Jun-Sik Yoo
Sergey Syritsyn (RBRC)

2nd R-CCS Symposium - 02/17/2020(3)

Particle Physics – Overview• Search for new phenomena
at the highest energies.
• Goal of the Large Hadron
Collider at CERN.
• High precision results at lower energies may havegreater reach: Belle II at KEK.
• To compare with Standard Model need lattice QCD
predictions: HPC plays a critical role
2nd R-CCS Symposium - 02/17/2020(4)

Standard Model•  (photon), g (gluon) and W + (weak) exchange
2nd R-CCS Symposium - 02/17/2020(5)

Standard Model• Proton and neutron inatomic nucleus composedfor u and d quarks.
• These are bound by theexchange of gluons g.

• u and d quarks nearly massless and highly relativistic• Copious virtual pair creation  many body problem
2nd R-CCS Symposium - 02/17/2020(6)

Lattice QCD
2nd R-CCS Symposium - 02/17/2020(7)

Lattice QCD• Introduce a space-time lattice• Evaluate the Euclidean Feynman
path integral– Study eHQCD t– Precise non-perturbative formulation• Use Monte Carlo importance samplingwith hybrid, MD/Langevin evolution• ~40 samples give sub-percent errors
2nd R-CCS Symposium - 02/17/2020(8)

Challenges• Numerical control of the path integral– 963 x 192 lattice  5 x 109 dimensional integral– Long autocorrelation times  misleading errors• Extend Schrodinger quantum mechanics into
Euclidean time
– Unphysical time evolution– Time-like processes challenging• Fermion sign problem–  baryon  0  path integral weight not positive– Poor signal to noise ratio in baryon correlators

2nd R-CCS Symposium - 02/17/2020(9)

Historical overview• 1974 Gauge-invariant lattice theory
formulated. [Ken Wilson]• 1980 String tension computed withphysical dependence on the gauge
coupling. [Mike Creutz]• 1990 Fermion loops routinelyincluded.
• 1992 Fermion doubling problemsolved. [David B. Kaplan]• 2002 Routine use of chiral quarks.
• 2014 Routine calculation withphysical u, d and s quark masses.
2nd R-CCS Symposium - 02/17/2020(10)

Lattice QCD – 2020
• Physical quark masses (ChPT not needed)• Chiral quarks (doubling problem solved)• Large physical volumes: (6 -10 fm)3
• Small lattice spacing: 1/a = 2.774 GeV• Some quantities with ~0.1% precision• Now an essential tool for many standard
model tests.
2nd R-CCS Symposium - 02/17/2020(11)

Algorithms
andArchitectures
2nd R-CCS Symposium - 02/17/2020(12)

Architecture• Regular lattice problem:
x easy for parallel computer.
• Difficulty scales as ~ L6.
• Weak scaling not sufficient• Large demand on network
bandwidth:– Local 4D volume grows with nodeflops– Off-node bandwidth grows as (nodeflops)3/4– Mira  Summit
0.2 TF/s /20 GB/s  42 TF/s /25 GB/sQCDSP – 600 Gflops••

RIKEN/Columbia/BNLGordon Bell Prize 1998

• Serious algorithm challenge
2nd R-CCS Symposium - 02/17/2020(13)

Numerical methods• Use low eigenmodes to solve D Gn = hn for multiple
right-hand sides (deflation).
• 963 x 192 volume requires 5K eigenvectors (160 TB).
• Compress using local coherence: 30x (Clark, Jung &Lehner, arXiv:1710.06884)• Use All-Mode-Averaging technique (Blum, Izubuchi & Shintani,arXiv:1208.4349 [hep-lat])– Loosen CG stopping condition 108  104.
– Obtain accurate result for 8 out of 128 time slices– O(108) accurate result = G104 G10  G104
– Achieve 5-20 x speed-up.

2nd R-CCS Symposium - 02/17/2020(14)

Numerical methods - cont• Split-Grids (Boyle and Jung)– Deflate multiple right-hand sides on full machine– Perform separate CG inversions on many subpartions– 4x speed-up on Cori at NERSC (Cray XC40/KNL)• Multi-splitting preconditioned conjugate gradient(Tu, Guo, and Mawhinney, arXiv:180408593)– Precondition with Dirac operator without off-node comms.
– Use many local inversions, exploit large node performance.
– Use tensor cores on Summit
and obtain 1.5x speedup– Must include halo sites inproduct Dpcond Dpcond.

2nd R-CCS Symposium - 02/17/2020(15)

New Directions• Use machine learning to infer theresults for complex quantities fromthose of simple ones. (Boram Yoon,LANL)

Correct small errors with a few exact results.
• Use machine learning to generate Markov samples.
(Albergo, Kanwar and Shanahan, arXiv:1904.12072)

2nd R-CCS Symposium - 02/17/2020(16)

New Directions• Use machine learning to simplify local MSPCG
approximation (Jiqun Tu: Columbia  NVIDIA)• Learn optimal 1536 parameters in T

• Use Quantum Computing to generate real-time
evolution?
2nd R-CCS Symposium - 02/17/2020(17)

Muon anomalous
magnetic moment
2nd R-CCS Symposium - 02/17/2020(18)

g - 2 for the muon
• Anomalous moment: a = (g ‐ 2)/2• BNL E821 expt:a = 11699208.9  6.3 x 10-10x• 3 difference between thestandard model prediction and experiment: FNAL E989
a EXP - a SM = 27.4(2.7)(2.6)(6.3) x 10-10• Effects of quark and gluons enter at order EM2:

HVP
692.5(2.7) x 10-10

HLbL
10.5(2.6) x 10-10
2nd R-CCS Symposium - 02/17/2020(19)

Brookhaven, Columbia,
Connecticut, Nagoya, RIKEN
Thomas Blum (Connecticut)
Norman Christ (Columbia)
Masashi Hayakawa (RIKEN & Nagoya)Taku Izubuchi (BNL & RIKEN BNL)
Luchang Jin (Columbia  Connecticut)
Chulwoo Jung (Brookhaven)
Christoph Lehner (Regensburg & BNL)

2nd R-CCS Symposium - 02/17/2020(20)

Use lattice QCD to calculate HLbL
• Compute connected and leading
disconnected parts.
quarks ingauge fieldbackgroundphotonsmuon• Important challenge for lattice QCD– Treat E&M through an expansion in – Massless photon introduces new problems• Sum stochastically over x and y
2nd R-CCS Symposium - 02/17/2020(21)

Use lattice QCD to calculate HLbL
• Use Lattice QCD for quark loop and gluons• Evaluate photons and muon parts analyticallyFinite volume
E&M

Infinite volume
E&M

quarkphotonmuon• With physical mass and continuum limit• μ = 7.20(3.98)stat(1.65)sys × 10−10 from QEDLX(arxiv:1911.081230)
2nd R-CCS Symposium - 02/17/2020

(22)

KL+–
  0+  e e
2nd R-CCS Symposium - 02/17/2020(23)

Physics of KL  + –• A second order weak, ``strangeness changing
neutral current’’(Cirigliano, et al. , Rev. Mod. Phys., 84, 2012)• KL  + – decay rate is known:– BR(KL  + –) = (6.84 ± 0.11 ) x 10-9

• Large ``background’’ fromtwo-photon process:2nd R-CCS Symposium - 02/17/2020(1)

Calculation    e+ e –• Avoid Euclidean calculation• Evaluate in Minkowski space• Wick rotate internal time
integral:E&M (complex)

QCD (real)

2nd R-CCS Symposium - 02/17/2020(3)

Lattice Results for    e+ e –(Yidi Zhao)• Lattice result is complex:– Exponentially small FV corrections– Physical kinematics, 1/a  1.73 GeV :• Im(A) = 35.94(1.01)(1.09) [Expt: 35.07(37)]• Re(A) = 20.39(72)(70).
[Expt: 21.51(2.02)]• First step in predicting KL + -– Lattice QCD calculation needed to remove
two-photon background.
– Would allow ~10% test of standard model
prediction for rare, 2nd order weak decay
2nd R-CCS Symposium - 02/17/2020(4)

KL – KS
massdifference
2nd R-CCS Symposium - 02/17/2020(27)

K0 – K0 system• K0 – K0 are distinct anti-particles:(s d ) and (d s) bound states• These are mixed by thestrangeness-violating weakcouplings:• MKL- MKS = 3.483(6) x 10-12 MeV• Sensitive to 1000 TeV energy scale, 1000 x LHCenergies• Evidence for charm quark energy scale first foundhere.
• Effects of QCD can now be computed from firstprinciples!
2nd R-CCS Symposium - 02/17/2020(28)

Lattice Version• Evaluate standard, Euclidean, 2nd order K0 – K0
amplitude:

tf - tb

tb - tatb - ti
2nd R-CCS Symposium - 02/17/2020(29)

MK Preliminary Results(Bigeng Wang)

MK x 10MeVMKFV

6.7(0.6)Expt.

3.483(6)0.27(18)

t2 - t1•••••

Physical light, strange and charm masses
643 x 128, 1/a = 2.36 GeV
Integrate: 0    10
152 configurations
a2 errors 20% ?
2nd R-CCS Symposium - 02/17/2020(30)

Many new lattice results• Study two-nucleon problem (S. Aoki, et al. HAL QCD)
• Use Euclidean lattice data to constrain spectral
functions in inclusive decays (S. Hashimoto, et al.)• Results for Kl3 decay on largest lattice to date(K.-I. Ishikawa, et al., PACS Collaboration)• Calculate long-distance part (5%) of direct CP
violation in K   decay:  (J. Karpie):• K decay and direct CP violation (RBC/UKQCD)
re (’/) = (1.38 ± 7) x 10-4 (lattice)(16.6 ± 0.23) x 10-4 (Expt)

2nd R-CCS Symposium - 02/17/2020(4)

Lattice QCD at the Exascale• Important opportunities to discover newphysics beyond the standard model.
• Much work still needed to compensate forthe increasingly weak inter-node network.
• Target smaller lattice spacing– Increase accuracy of charm quark physics– Use open boundary conditions (Luscher & Schaefer)– Fourier accelerate: HMC  RMHMC (Girolami &Calderhead)– Studying QCD at =100:

2nd R-CCS Symposium - 02/17/2020(32)