Neural Networks
and Quantum Field Theory
Jim Halverson
Deep Learning and Physics Seminar

Based on: 2008.08601 with Maiti and Stoner

The Gang

Anindita Maiti
targeting physics postdocs, Fall 2022

Keegan Stonertargeting ML labs, Fall 2022

+ Three Other Connections @ Physics / ML Interface

Institute for Artificial Intelligenceand Fundamental Interactions (IAIFI)

Physics Meets ML

Feel free to contact me!

one of five new NSF AI research institutes,
this one at the interface with physics! MIT,
Northeastern, Harvard, Tufts.

virtual seminar series at the interface,“continuation” of 2019 meeting atMicrosoft Research.

e-mail: jhh@neu.edu
Twitter: @jhhalverson
web: www.jhhalverson.com

ML for physics / math discoveries?
Can physics / math help ML?

Bi-weekly seminars from physicists
and CS, academia and industry.

ML for Math:Colloquia begin in Spring!
www.iaifi.org.

Sign up at www.physicsmeetsml.org.

e.g. “Learning to Unknot”: 2010.16263

ML for Strings:

e.g. “Statistical Predictions in String Theory
and Deep Generative Models”: 2001.00555

Legend:

What is learning?

Randomly initialized NN
Trained NN

Physics Language:
Learning is a data-induced flow from an initialization
function-space distribution to a trained distribution.

Bayesian Language:
Learning is approximating the posterior over functions
given a prior and a likelihood.

Distribution atInitialization

21

Learning

TrainedDistribution

33

function space12

Then what is supervised learning?
the evolution of the 1-pt function E[f] until convergence.

Outline-

What is QFT? (physically? origin of Feynman diagrams. statistically?)-NN-QFT Correspondence: model NN distributions with QFT techniquesi) asymptotic NNs, GPs, and free field theoryii) NNs, non-GPs and Wilsonian “effective” field theory.
iii) renormalization: removes divergences in higher correlators, simplifies NN dist.

-Experiments: verify approach in simple examples-Discussion and Outlook: parameter-space / function-space duality, training

What is QFT?
physically?
what are Feynman diagrams?
statistically?

What is QFT, physically?
●

quantum theory of fields,
and their particle excitations.

●for both fundamental particles, (e.g. Higgs)
and quasiparticles (e.g. in superconductors)●

a single QFT predicts radioactive decay rates,
strength of particle scattering, etc.

●●two main perspectives:“canonical quantization” (bra-ket approach)Feynman’s path integral (today).
Many Nobel prizes. (Could easily rattle off 5-10?)

Example: Higgs boson discovery

The QFT = Standard Model(SM) of Particle Phys.
2012: Discovered Higgs
boson at CERN, e.g., in
diphoton channel @ left.
Amazing science press.
2013: Nobel to Higgs, Englert.

Origin of Feynman diagrams?
Pictures useful for computing momentsof Gaussian or near-Gaussian distributions

Example: Near-Gaussian Moments
via Perturbation Theory

Example: Gaussian Moments

for small λ, truncate

Feynman rules: a picture-expression dictionary

Additions and extra widgets may arise, but
Essence: approximate non-Gaussian moments interms of Gaussian moments, diagrammatically.

Sounds like QFT is physics widgets
on top of a statistics backbone.

What is QFT, statistically?
●

defined by distribution on field space,
the so-called Feynman path integral.
log-probability S[Φ] is “action”●

Experiments measure n-pt
correlation functions and amplitudes.

●●Free QFT: no interactions, Gaussian.
Perturbative QFT:
distribution is near-Gaussian, computeapproximate moments perturbatively.

NN-QFT Correspondencei) asymptotic neural nets, GPs, and free QFTii) finite N neural nets, non-GPs, interacting QFTiii) Wilsonian renormalization
A way to model NN distributions
using QFT techniquesAsymptotic Neural Networks
neural network has a discrete hyperparameter Nthat enters into its architecture.
asymptotic limit = N → ∞ limit
crucial property: want to add infinite number ofparameters, which themselves are random variables!
example:infinite width limit of single-layer or deep
feedforward networks

NN-GP Correspondence and Central Limit Theorem
Add N iid random variables,
take N → ∞,sum is drawn from a Gaussian distribution.
If some step in a neural net does this,that step drawn from Gaussian.
e.g., if NN output does, it’s drawn from a Gaussian.
then NN is drawn from Gaussian distribution on
field space, known as a Gaussian Process (GP).

“Most” architectures admit GP limit
Single-layer infinite width feedforward networks are GPs.

[Neal], [Williams] 1990’s

Deep infinite width feedforward networks are GPs. [Lee et al., 2017], [Matthews et al., 2018]
Infinite channel CNNs are GPs. [Novak et al., 2018] [Garriga-Alonso et al. 2018]
Tensor programs show any standard architecture admits GP limit.

[Yang, 2019]

tons of examples citedin our paper admit GP limits

GP property persists under appropriate training.

[Jacot et al., 2018] [Lee et al., 2019]

Gaussian Processes and Free Field Theory
Gaussian Process:Free Field Theory:“free” = non-interacting
Feynman path integral:

distribution:

where:
K is the kernel of the GP.
log-likelihood:n-pt correlation
functions:
Crucial note:P[f] can also have one or zero integrals,“local” and “ultra-local” cases, respectively.

From P.I. perspective, free theories
are Gaussian distributions on field space.
e.g., free scalar field theory

GP Predictions for Correlation Functions
if asymptotic NN drawn from GP and
GP “=” free QFT, should be able to use Feynman
diagrams for correlation functions.

Right: analytic and Feynman diagram expressions
for n-pt correlations of asymptotic NN outputs.
Physics analogy: mean-free GP is totally
determined by 2-pt statistics, i.e. the GP kernel.
kernel = propagator, so GP = a QFT where all
diagrams rep particles flying past each other.

What about ﬁnite N nets?

Non-Gaussian Processes (NGPs), EFTs, and Interactions
Punchline: finite N networks that admit a GP limit
should be drawn from non-Gaussian process. (NGP)

where, e.g., could have a model:

Wilsonian EFT for NGPs:

determines NGP “effective action” = log likelihood.
Some art in this, but done for decades by physicists.

Experiments below: single-layer finite width networks
such non-Gaussian terms are interactions in QFT.
their coefficients = “couplings.”
odd-pt functions vanish → odd couplings vanish.
𝜅 is 1/N suppressed rel. λ, somes more irrelevant(Wilsonian sense), gives even simpler NGP distribution.

NGP Correlation Functions from Feynman Diagrams
Correlation functions defined by NGP distribution:

Feynman Rules:

use usual physics trick

to compute diagrammatically as Feynman diagrams.
Essentials from QFT reviewed in paper,e.g. cancellation of “vacuum bubbles” (components with noexternal points) by expanding the denominator.

these rules are a picture toanalytic expression dictionary.
note: in our experiments, GP kernel happens tobe exact all-width 2-pt function.

2-pt, 4-pt, and 6-pt Correlation Functions

point: theory equations that
actually enter our NN codes.

At this point you should object!

(very impressive attention to detail if you actually did.)
Input space integrals often diverge at large input.
QFT prescription: “regularization.”
Various varieties, we use a “hard cutoff” Λ, replaceso any input integral is over a box of size Λ.

Making sense of divergences: Renormalization
Experiments: the central insight in renormalization.

Theory: NGP action corrects GP action by[Zee] for beautiful textbook discussion.

Evaluate set of NNs on inputs
the old S had Λ→ ∞ and computing n-pt gives
divergences. Λ finite regulates those divergences,
input is now in a box.
and measure experimental correlation functions,
For any Λ sufficiently big, measure couplings,
make predictions, verify with experiments.
these just are what they are! One set of corr fns.
Goal of theory is to explain them.

But there’s an infinite number of, and onlyone set of experiments for them to describe!
How does this make sense?

Essence of Renormalization
the infinity of effective actions must makethe same experimental predictions, requiring, e.g.

Extracting 𝛽-functions from theory
NN effective actions (distributions) with different
Λ may make the same predictions by absorbing
the difference into couplings, “running couplings.”

Encoded in the β-functions, which capture how
the couplings vary with the cutoff.
Induces a “flow” in coupling space as Λ varies,
Wilsonian renormalization group flow. (RG)
Extract from hitting n-pt functions with derivatives.

Our examples:𝜅 more irrelevant than 𝜆, in sense of Wilson.
Means as Λ gets large, 𝜅 goes to zero faster than 𝜆,so you can ignore it.
Extract β-function for 𝜆 from deriv. of 4-pt.

Experiments
in single-layer networksi) sanity check fall-off to GP at large Nii) measure 4-pt non-Gaussianities, predict 6-pt and verifyiii) cutoff dependence of 4-t non-Gaussianities verifies renormalization

Parametric Control in 1/N
Also of interest are connected n-point functions,which receive tree-level contributions from n-point couplings.
Results:

G4|con ~ N-1

G6|con ~ N-2

expect in general

Comes from parameter space description.
Simplest function space implication?
n-point coupling have same scaling, give parametric control in 1/N.

G2k|con ~ N1-k

Controlling correlators with 1/N-expansion

Subtlety of Networks with Linear Output: Independence●

any network with linear output looks like:●the bias and weight terms are independent draws from different processes!
QFT language: two fields that don’t interact with each other.

●For Gaussian bias, fb is drawn from constant GP for all N.
However, fW is from NGP at finite N, GP only in asymptotic limit.
QFT language: fW has self-interactions
Therefore: must use NGP for fW, but not fb.

Experiments with single-layer networks
Erf-net:Gauss-net:

ReLU-net:

Deviations from GP Predictions

Question: can we measure theexperimental falloff to the GP
prediction as N → ∞ ?

Measuring Falloff to GP Predictions @ Large N
Some details of the experiments:

10 experiments of 106 neural nets each, computeensemble average to get correlation functions.

Background := average (across the 10 expts)
standard deviation of mn
Experimentally determined scaling:Simple Extracting of Couplings from Experiments
intentionally left the coupling 𝜆 inside the interaction
integrals. Only pull out if constants!

Case, 𝜆 function: write as constant + space varying

but aren’t couplings constant?
e.g., physics: proton collisions have same interactions
at Fermilab (Illinois) and CERN (Switzerland).

then we havebut Standard Model is T-invt. NGPs? not always.

and expression from before not constant,Case, 𝜆 constant: measure from 4-pt function expts
but when variance is small relative to mean havecall denominator integrand 𝚫1234y.

our definition of “measuring 𝜆.”Include 4-pt Contribution to 6-pt functions
The use of EFT:

Experimental verification:i.e. effective corrections from including 4-pt contr. to 6-pt.
Effectiveness: Expt 6-pt - GP prediction = NGP correction

measured 𝜆 enters
6-pt prediction,corrects bad GP
prediction to very
close to experiment.
EFT of NN is effective!

More Precision: Quartic Coupling Predictions
Three-parameter model:

Fitting the models:
Inputs form a grid.
Train on interior, Test on exterior.

Three interactions:●●●

Constant local quartic
Input-dependent local quartic
Constant quartic with symmetric non-locality

Results are fresh, see details in journal version.

Renormalization Theory vs. Experiment: ReLU-net

experimentally measured din-dependent slope matches theory
predictions from Wilsonian RG

Discussion and Outlook
summary,
parameter-space / function-space duality,
supervised learning in QFT language

Summary of Results
asymptotic NN’s “=” Free QFT

b/c drawn from GPs
NNs “=” QFT

b/c drawn from NGPs
central idea: model NGP / NN distribution using
Wilsonian effective field theory. (EFT)

fairly general: any “standard architecture” (Yang)
admits a GP limit. persists under some training.
therefore, away from limit, NGP. use EFT to model.
import QFT ideas directly into NNs.
EFT treatment of NN distribution yields:
1) output correlation functions as Feynman diagrams.
2) measure some couplings (non-Gaussian coeffs) inexperiments, predict, verify in experiments.
3) Wilsonian RG induces flow in couplings, simplifies themodel of the NN distribution.
Verified all of this experimentally, single layer networks,
indeed QFT gives function-space perspective on NNs.

What does this treatment get you?
Duality:

Training:

In physics, means two perspectives on a single system,
where certain things are easier from one.

Our formalism only requires being “close” to GP,
where measure of closeness determined
experimentally and in examples is relatively low N.

Parameter-space / function-space duality:
at large N, parameter-space complexity explodes.
but in function-space complexity decreases due torenorm. and 1/N suppression of non-Gaussianities.
Acute example: single number in NGP dist. was sufficient
to approximate NGP 6-pt corrections, despite losing an∞ number of params in moving from GP.

Some training preserves GP at large N, in principle
allowing QFT treatment of NGP during training.
Supervised learning:in QFT language, it is just learning the 1-pt function.
in general this will break symmetry of NGP (seepaper next week for priors), bring in even more QFT.

Thanks!
Questions?
And seriously, feel free to get in touch:e-mail: jhh@neu.edu
Twitter: @jhhalverson
web: www.jhhalverson.com

Constants or Functions? Use Technical Naturalness
Recalling thatwhen should spatially varying

A principle of ‘t Hooft:

be small?

Because sometimes QFT corrections to anarbitrary coupling g give:it’s true sometimes in physics, e.g.
is small relative to weak scale, protected by chiralsymmetry that is restored when e- mass goes to zero.
Applied to our case? i.e., when isfor NNs?

but sometimes

or at least in that case it might be small.

holds true in our cases, GP kernel of Gauss-net is the only
T-invt one, and only example with coupling constants.