Computa(onal	
  Nuclear	
  Physics
Joe	
  Carlson	  	
  Nuclear,	  Par0cle,	  Astrophysics	
  and	  Cosmology	
  Group
Theore0cal	
  Division,	
  LANL

Building	
  Complex	  Structuresand	
  phenomenon	
  fromsimple	  interac(ons
QUCS	
  2012,	
  NARAThe Nuclear Landscape and the Big Questions (NAS report)• How did visible matter come into being and how does it
evolve?
• How does subatomic matter organize itself and what
phenomena emerge?
• Are the fundamental interactions that are basic to thestructure of matter fully understood?
• How can the knowledge and technological progressprovided by nuclear physics best be used to benefit
society?

Experimental relevance:
FRIB, RIBF,ATLAS, NSCL,LENP Facilities, NNSA
facilities, JLab, JINA, SNS, ...

Nuclear	  Physics	
  SciDAC	  projects
SCIDAC	  projects	
  :	
  UNEDF	
  	
  ➭	
  	
  NUCLEI
Na0onal	
  (and	  interna0onal)	
  eﬀortPhysicists,	
  Computer	
  Scien0sts	
  and	  Applied	
  Mathema0cians15	
  ins0tu0ons,	
  2011	
  19	
  postdocs	
  +	
  11	
  students
Both	
  People	
  andComputa7onal	  Resources	
  are	
  cri7cal~120M	
  core-­‐hours	
  in	
  2012For a popular description of UNEDF, see:
SciDAC Review Winter 2007

http://www.scidacreview.org/0704/pdf/unedf.pdf
Nucl. Phys. News 21, No. 2, 24 (2011)
Office of Science “Highlight Series”:
http://science.energy.gov/news/in-focus/2011/03-28-11-s/

3

Junior	
  Scien(sts	
  in	
  SciDACPOST-DOCTORAL ASSOCIATES (2010)
Christopher Calderon, LBNL (staff, Numerica co.)Joaquin Drut (Professor, UNC)
Stefano Gandolfi, LANL (staff, LANL)
Kai Hebeler, OSU (TRIUMF)
Heiko Hergert, MSU (OSU)
2010: Early
Career Award
Jason Holt, UTK/ORNL
Eric Jurgenson, LLNL (staff, LLNL)
Markus Kortelainen, UTK (U. Jyväskylä)
Plamen Krastev, UCSD (research, Harvard)
Pieter Maris, ISU (Research Prof. ISU)
Eric McDonald, MSU (staff scientist, MSU)
Gustavo Nobre, LLNL (BNL, NNDC) 2011: Faculty
Junchen Pei, UTK (Prof., Pekin U.) UNC/Chapel
Hill
Nicolas Schunck UTK (staff, LLNL)
Roman Senkov, CMU
Ionel Stetcu, UW (staff, LANL)
Jun Terasaki, UNC (staff, U. Tsukuba)
2012: Faculty
Stefan Wild, ANL (staff, ANL)
GuelphRelevant instruction (workshops, courses)
is crucial for the future of the field

Effect of UNEDF on workforce

Year-1: 9 students, 17 postdocs;
Year-2: 12 and 12;
Year-3: 10 and 18;
Year-4: 11 and 19
2010:
Math/CS
Staff ANL

2010:Staff
LLNL

2012:
HarvardResearch
Computing

Why	
  is	
  Nuclear	
  Physics	
  Important?
Many-­‐body	  physics:
Strong	  coupling	
  of	
  spin	  to	
  space	  (tensor,	
  spin-­‐orbit)
Strong	
  Pairing	
  (	
  Δ	
  /	
  EF	
  from	
  0.03~0.3)
Compe00on	
  between	
  single-­‐par0cle	
  evolu0on	
  and	
  pairing
Clustering	  (	  8Be,	
  12C	
  Hoyle	  state,	  ...)
Nuclear	
  physics:
Neutron-­‐rich	
  nuclei	
  and	
  limits	
  of	
  stability
Nucleosynthesis:	
  light	
  (BBN	
  fusion)	
  &	
  heavy	  elements	
  (SN,	
  neutron	
  star)Correla0ons	
  and	
  nuclear	
  response
Ties	
  to	
  other	
  ﬁelds:
fundamental	
  symmetries	
  and	
  BSM	  (ββ	
  decay,	
  superallowed	
  β	
  decay,...)
astrophysics	
  (reac0ons,	
  neutrinos,	
  gravity	
  waves,	
  ...)cold	
  atom	  physics	
  (superﬂuidity,	
  universality,	
  Eﬁmov,	
  ...)

Illustrate	
  progress	
  and	
  challenges

5Star(ng	
  point:
Two-­‐	
  plus	  three-­‐nucleon	
  interac0ons	
  (phenomenological,	
  EFT)

H =

�

Ti +

i

�

Vij +

i<j

�

Vijk

i<j<k

Vij	
  ﬁt	
  to	
  many	  NN	
  experimental	
  dataVijk	
  has	  O(5)	
  parameters,	
  typically	
  ﬁt	
  to	
  few-­‐nucleon	
  systemsOne-­‐	
  plus	  two-­‐body	  charges	
  and	
  current	
  operators:
ρ =

�
i

j =

ρi +

�
i

�

ρij

i<j

ji +

�

jij

i<j

+	
  similarly	
  for	
  weak	  interac0on;	
  few	  parameters	
  ﬁt	
  to	
  data6

Computa7onal	
  Methods
Light	
  Nuclei:	
  	
  	
  	
  Quantum	
  Monte	  Carlo
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  Conﬁgura7on	
  Interac7on
Medium	  Mass	
  Nuclei:	
  Coupled	
  ClusterHeavy	
  Nuclei:	
  Density	
  Func7onal	
  Theory
Nucleonic	
  MaQer:	
  Quantum	
  Monte	  Carlo
	
  	
  	
  	
  	
  	
  	
  Density	
  Func7onal	
  Theory7

Light	  NucleiSpectra	
  reproduced	
  with	  `realis0c’	
  NN	
  +	
  NNN	
  interac0ons-20-30Energy (MeV)

-40
-50
-60
-70
-80
-90
-100

0+

4He 6

2+
0+

He 6

1+
2+
3+
1+

Li

7/2<
5/2<
5/2<
7/2<8
1/2<
3/2<

2+
0+

He

7Li

4+
0+
2+
2+
1+
3+
1+
2+

8Li

Argonne v18
With Illinois-7
GFMC Calculations

0+
1+
4+
3+
1+
2+
4+

5/2<
1/2<
3/2<

9
+ Li

2
0+

8Be

7/2+
5/2+
7/2<
7/2<
3/2<
3/2+
5/2+
1/2<
5/2<
1/2+
3/2<

1+

3,2+
0+
2+
2+
0+

9Be 10Be

11 July 2012

3+
4+
2+
3+
2+
1+
1+
3+

10B
0+

AV18
+IL7

Expt.

0+

12C

Pieper,	
  Wiringa,	
  et	
  al.

8

Magne7c	
  moments	
  of	
  light	
  Nuclei
4
3
p

3H

2

µ (µN)

7Li

9Li

1
2H

0
-1

n

8Li 8
B

6Li

GFMC(IA)GFMC(MEC)
EXPT

9B

9C
7Be

9Be

3He

-2
Pastore,	  Wiringa,	
  Schiavilla,	
  Pieper	
  (2012)

-3

9

6

ρNN(q,Q=0) (fm )h of the configurations sampled in thenn, and remaining 1=4 of np pairs are in T; S # 1; 0
large range of values of x and X required(quasibound) states. When multibody tensor correlations
3
ed results, especially for He, require
are taken into account, 10%–15% of the T; S # 1; 0 pairs
ers of points; we used grids of up to 96
are spin flipped to T; S # 1; 1 pairs, but the number ofx and X, respectively. We also sum overBack to Back momenta np vs. pp
just pair 12.
p momentum distributions in 3 He, 4 He,
6
10
i are shown inEM
Fig.Transitions
1 as functions ofA=6,7
the
3
Hem q at fixed total pair momentum Q # 0,
5
4
10
He
6He(2+;0A0
nucleons moving back to back.
The+;0)
sta6
B(E2)
4
Li
10
to the Monte Carlo integrationaredis8
6Li(3+;0A1+;0) B(E2)Be
3
e pp pairs; they are negligibly
small for10++
striking features seen in 6all
cases
are
Li(0 ;1A1 ;0) B(M1)2
10
distribution of np pairs is much larger
7-) B(E2)
1
airs for relative momenta inLi(1/2
theA3/2
range
10(ii) for the helium and lithium
7Li(1/2isotopes-A3/2-) B(M1)010momentum distribution is absent in the7-A3/2-) B(E2)-1stead exhibits a change of Li(7/2
slope
at a
10
e of p ’ 1:5 fm$1 . The nodal
structure
7Be(1/2-A3/2-) B(M1)
0
1
2
3
4
5
8
-1
minent in Be. At small values of q theq (fm )
6He(0+)A6Li(1+) B(GT)p momentum distributions are closer toCohen-Kurath
36pair numbers, which in7Be(3/2He,-)A47He,
Li,
Li(3/2-) B(GT) FIG. 1 (color online). The np (lines) and pp (symbols) moNCSM
ctively, 2, 4, 3, and 8=3. Note that the np
mentumdistributions in various nuclei as functions of the
7Be(3/2-)A7Li(1/2-) B(GT)ution is given by the linear combination
relativeGFMC(IA)momentum q at vanishing total pair momentum Q.
7Be(3/2-) Branching ratio

132501-2
0

12
Ratio to experiment

3

GFMC(MEC)
Experiment

Light Nuclear Reactions
3He

Navratil	
  &	
  Quaglioni
PRL108,	
  042503	
  (2012)

d

p
3

4He

NCSM/RGM	
  is	
  pioneering	  ab	
  ini(o	
  calcula5ons	
  of	
  light-­‐
nuclei	  fusion	
  reac5on	
  with	  NN	
  interac5on.	
  Here,	
  3He(d,p)
4He S-factor.

Data deviate from NCSM/RGM results at
low energy due to lab. electron-screening

Navra7l,	
  Quaglioni,...

(full range to 2.0)Ab	
  ini(o	  theory	
  reduces	
  uncertainty	
  due	  to	
  conﬂic7ng	
  data FIG. 2: (Color online) The integrand of Eq. (5) (×2µ/k!w)
is shown for the p1/2 (red squares) and p3/2 (blue circles)
neutrons in 8 Li → n 7 Li. It is binned by the n-7 Li separation
rcc with bars showing Monte Carlo errors. The solid curves
are cumulative integrals of Eq. (5), starting from the origin; atlarge rcc , they are the ANCs (divided by 2 for visibility on thisscale). The dotted curve with no scale shows the distributionof Monte Carlo samples.

n+3H
important
for	
  NIF

integral is computed with relatively small statistical errors.
The computed Clj depend sensitively on the separationenergies
B implicitly through√√ B. Equation (5) contains
k = 2µB/! and η ∝ 1/ B, and it is rigorously true
when B = Eint − E for the given potential. However,there are often significant differences between this B andthe experimental separation energy Bexpt . We computed
several ANCs in light nuclei, first using the GFMC BH
for the AV18+UIX Hamiltonian and then using Bexpt .
http://physics.aps.org/synopsis-for/10.1103/PhysRevLett.107.122502
The use of Bexpt in Eq. (5) may be understood by considering small changes to the potential. When B $ |E|,
they can produce small changes in the wave-function
interior but large fractional changes in B. The short-

to
2.13

FIG. 3: (Color online) Predicted ANCs from Eq. (5), divided
by experimentally-derived values from the references given at
the right (those not appearing elsewhere are Refs. [29–31]).
For each ANC, small error bars indicate the Monte Carlo errorof Table I and larger error bars indicate its quadrature sumwith the experimental error. Results for the same computed
ANC divided by different “experimental” numbers are joinedwith dashed lines. Parentheses indicate particularly uncertain
experimental constraints.

ANCs	
  from	  light	
  nuclei
NolleQ	
  and	
  Wiringa	  (2012)Also	
  recently	
  width	  calcula7ons	
  :	
  NolleQ
range overlaps at ∼ 4 fm, where the ANC integral starts
to converge. Use of BH yields Cp2 1/2 = 0.029(2) fm−1

Scaling	
  of	
  Large	  Scale	  Conﬁgura7on	
  Interac7on	
  Calcula7ons

Vary,	
  Ng,	
  Maris,...
Iowa	
  State,	
  LBNL,	  ...

12

Anomalously	
  Long	  Life7me	
  of	
  	
  Carbon-­‐14	
  and	  the	
  importance	
  of	
  3-­‐nucleon	
  forces	
  Maris,	
  et	
  al,Phys.	
  Rev.	
  LeQ.	
  106,	
  202502	
  (2011)3-­‐nucleon	
  forces	  suppress	
  cri5cal	
  component

SRG-­‐N3LO
	
  	
  	
  	
  λ	
  =2	
  fm-­‐1ħΩ	
  =15	
  MeV
s

1

net	
  decay	
  rate	  Is	  very	  small

24Si

se
a
t C

!

Asynchronous	
  Dynamic
Load	
  Balancing	
  (ADLB)	
  LibraryLusk,	
  Pieper,	
  Butler
SciDAC	  Review	
  201014

Ab	
  ini7o	
  descrip7on	
  of	
  12C
GFMC	
  (Pieper	
  et	
  al.)The	  ADLB	
  (Asynchronous	
  Dynamic	
  Load-­‐Balancing)	
  library	   &	
   GFMC.	
   GFMC	   energy	
   93.5(6)	
   MeV;	
   expt.	
  92.16	
  MeV.	
  	
  GFMC	
  pp	
  radius	
  2.35	
  fm;	
  expt.	
  2.33	
  fm.	  

Lahce	  EFT	  (Lee,	  Epelbaum,	
  Meissner,…)
0+(2)	
  fpp

0+(1)	
  fpp

16
Epelbaum	
  et	
  al.,	
  Phys.	
  Rev.	
  LeQ.	
  106,	
  192501	
  (2011)

Medium-Mass Nuclei: Coupled-cluster method
Description of medium-mass open nuclear systems
G. Hagen et al., Phys. Rev. Lett. 109, 032502 (2012)••½+ virtual state

Strong	  coupling	
  to	
  con7nuum	
  for	
  neutron	  rich	
  calcium	
  isotopes
Level	
  ordering	
  of	
  states	  in	
  the	
  gds	
  shell	  is	
  contrary	
  to	
  naïve	
  shell	
  model	
  pictureImproved	
  Density	
  FuncGonals

Neutron	  Drops,	
  Masses,	
  Fission,…
Deriva5ve-­‐free	
  op5miza5on,	
  uncertainty	
  quan5ﬁca5on

hZp://www.deixismagazine.org/2011/03/cranking-­‐up-­‐the-­‐speed-­‐of-­‐d^/hZp://www.deixismagazine.org/2011/03/pounding-­‐out-­‐atomic-­‐nuclei/
hZp://www.mcs.anl.gov/news/detail.php?id=720

LACM, Fission: the ultimate challenge
Stability of the heaviest nuclei, r-process, advanced fuel cycle, stockpile stewardship!

!"#$%&'()*+,-#.,/01)

7/38'91-/0'):*4)

PRC 85, 024304 (2012);.00'-#A')(B,/$%-1)
B(Q20) (h2/MeV b2)

0.04 PRC 84, 054321(2011)!"#$%&'(

Fm

256

0.03
0.02
0.01
0.0

0

50

100

150

200

Quadrupole moment Q20 (b)PRC 80, 014309 (2009)

2+$'3%-/0)4'-5,%6+'1);.,<3.,=/#.,)>%=5)'?"'3%$',=@)"3'(%-#.,1)
PRC 80, 014309 (2009)

PRC 78, 014318 (2008)

19

The limits: Skyrme-DFT Benchmark 2012••

Systematic errors (due to incorrect assumptions/poor modeling)
Statistical errors (optimization and numerical errors)288~3,000

Asymptotic freedom ?

How many protons and neutrons can be bound in a nucleus?
Skyrme-­‐DFT:	
  6,900±500syst
Erler et al.,Nature 486, 509 (2012)

hQp://www.livescience.com/21214-­‐atomic-­‐nuclei-­‐varia7ons-­‐es7mate.html
hQp://www.sciencedaily.com/releases/2012/06/120627142518.htm

Neutron	
  MaZer:	  	
  from	  very	  low	  to	
  high	  density

Low-­‐Density	
  (dilute)	
  near	  free	  Fermions	  to	
  near	  Unitarity
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  range	  of	
  the	
  interac0on	
  <	
  interpar0cle	
  spacing
Stringent	
  tests	
  of	
  many-­‐body	  calcula0ons	
  with	  strong	
  correla0ons
Analy0cally	
  known	
  at	
  extremely	
  low	
  densityE	  /	
  EFG	
  rapidly	
  decreases	
  to	
  ~	
  1/2	
  with	  increasing	
  kF	
  a
Pairing	
  Gap	  very	  large,	
  up	
  to	
  ~	
  0.5	  Ef	  in	
  cold	  atoms

21

Unitary	
  Regime	
  andLow-­‐Density	  Neutron	  MajerFirst	
  Fermi	  Condensates	
  2004

Experimental Results in Cold Atoms(nearly) Free Fermions(nearly) Free Bosons‘Universality’ and the BCS-BEC transition
Polarons
Efimov States
Superfluid Fermions (s-, p-, d-wave,...
pairing)
Exotic Polarized Superfluids(FFLO,...)
PseudoGap States
Itinerant Ferromagnetism`Perfect’ Fluids
Reduced Dimensionality
More than pairing (3-,4-bodycondensates, ...)
Bose, Fermi Hubbard Models,22

5

Towards largest Nuclei: neutron-rich matter. Unitarity

Equation of state: cold atoms and0.42
low-density
neutron matter
(2)
εk

0.40

(2)

0.38

(4)

ξ

0.40

$k

εk
0.36

N = 38N = 48N = 660.34(h)

εk (4)

#

0.38

0.36

0.34

0.32

$k

0

0.2

0.1
1/3

ρ

0.3

0.4

1/3

= αN /L

38 online) The calculated ground state energy
FIG. 1.N =(color48 value of ξ versus the lattice size for (h)shown N
as=thevariousN=
66$
particle numbers and Hamiltonians.
k4for a variety of lattice sizes and find agreement with Ref.
[25]. The error bands in the figure provide least-squares
estimates for the one sigma error based upon quadraticfits to the finite-size eﬀects. The fits are of the form
E/EF G = ξ + Aρ1/3 + Bρ2/3 . For the interactions tuned
to re = 0, a fit with A fixed to zero is used. Including
a linear coeﬃcient in the fit yields a value statisticallyconsistent with zero.
The extrapolation in lattice size for the k 2 and Hubbard dispersions show opposite slope as expected fromthe opposite signs of their eﬀective ranges. The extrapolation to ρ → 0 is consistent with ξ = 0.372(0.005) inall cases. Our final error contains statistical component
and the errors associated with finite population sizes andfinite time-step errors. This value is below previous experiments, but more compatible with recent experimental
results of the Zwierlein group[8].

Effective Range
Dependence

S = 0.12(3) QMC

0.42

0.32

100× reduction
to the usual
0.1 in computer
0.2time, compared0.3
0.4
N = 66
N = 38
1/3 The improvement
1/3
FG importance function.
increased to
0.41
!
=
/L
3"N
1500× for N = 38 in a 12 lattice. For larger systems,
0.4
the discrepancy is much larger still; indeed the statisticalFGFefluctuations from the latter are such that often meaning. 1. AFMC lattice calculationsof theGas
parameter, updated 0.39
from Ref. 23).
ful results cannot
be unitaryobtainedFermi
with therunξconfigurations
Symbols are for diﬀerent
kinetic
terms as a function of particle number and lattice size. The
described
above.
0.38 for diﬀerent
lattice spacing is denoted
as 1α.weSimulations
havecalculations
been performed
with L3 lattices,
In Fig.
summarize ourof the energy
3
as a Lfunctionρ1/3 whereopen
ρ = N/N
theforparticlevalues of lattice lengthin eachofdirection;
symbolsareeven L=16,20,24; closed are
k , and0.37
is N = 38, 48 are
or 66.
We plot ξ,
Eq.ξ1,=where
wefor odd L (see text).number
All extrapolations
consistentwith
0.372(5).
have in all cases used the infinite system free-gas energy
0.36
0
0.1
�2 k 2N
EF G = 35 2mF with kF3 = 3π 2 αNasthereference.
30= ξ + S k r + ....

ξ

E/E

DMC

ξ = 0.372(5) QMC
0.375(5) Expt (MIT)

0.2

0.3

0.4

kF r eA history of results for the Bertsch parameter is given in Ref. 28). The first DMC
culation used up to 40 particles
re ≈0.3,The ground-state energy as a function
FIG. 2.kF(coloronline)Hamiltonianand
N aξ Poschl-Teller
err
A err potential withk(2)

of kF re : comparison of DMC and AFQMC results. DashedTowards largest Nuclei: neutron-rich matter
Equation of State: Cold atoms and Neutron Mattercomparison including effective range terms
-1

0
1

0.1

0.2

kF [fm ]
0.3

0.4

0.5

Neutron Matter and cold atoms
with effective range term

QMC s-wave
QMC AV4
Cold Atoms

Lee-Yang

0.9

E / EFG

0.8
0.7
0.6
0.5
0.4
0

2

4

6

- kF a810Carlson, Gandolfi, Gezerlis, PTEP (2012) in press

Laboratory tests of strongly-paired many-body problem

Superﬂuid	
  pairing	
  gap	  in	
  cold	  atoms
Spin up, down densitiesρ1

Δ / EF

0.8
0.6
0.4
0.2

ρ

0

0

0.2

0.4

0.6

( k / kF)

2

0.8

1

1.2

4

Polarization

Measurements	
  of	
  polariza0on	
  in	
  superfluida	
  trapquasiparticle dispersion relation do not signif1
T’=0.03
0.8
0.6 !(r)
0.4
0.2
0
1
T’=0.05
0.8
0.6 !(r)
0.4
0.2
0
0
0.2

icantly alter the extracted bounds on the superfluid gap.
!=0.5, tc=0.25!=0.38, tc=0.25!=0.43, tc=0.07(MIT Expt.)

� k
∆ = δ
2m

δ = 0.45(05)!=0.46, tc=0.1

0.4

r/R

0.6

0.8

Largest	  Δ/Ethef	
   pairing gap fromIn summary, we haveextracted
measurements of spin up and spin down densities in polarized Fermi gases in the unitary regime.2These
2 systems
have an extremely large gap of almost one-halfthe Fermi
Fenergy – the value extracted in this work is clearly thelargest gap measured in any Fermi system. Future more
precise experiments extending over the BCS-BEC transition region would allow an experimental determination of
the evolution of the pairing gap from the weak-coupling
regime of traditional superfluids and superconductors toJC	
  regime.
and	
  Reddy,	  PRL	
  resolve
2007 longthe strongly-interactingThis could
standing issues regarding,for example,
analyzing	  MIT	
  dtheatapairing gap
in neutron matter and the cooling of neutron stars. 25

1

FIG. 3: Polarization versus radius, theory and experiment, fordifferent values of δ and tc at T ! = 0.03 and T ! = 0.05. The

We would like to thank M. Alford, A. Gezerlis and Y.
Shin for useful comments on the manuscript. The work ofNeutron	
  MaQer
Pairing	
  Gap	  at	
  low	
  density

Transi7on	
  from	  weak	  pairing	
  (Gorkov	
  correc7on)	
  to	
  near	  unitarity	
  26

Low-moderate density EOSAll	
  methods	
  based	
  on	
  NN	
  interac0ons	
  give	  similar	
  results

27

Neutron	
  Star	  (Mass/Radius)	
  requires
neutron	
  maQer	
  at	
  higher	
  (<2-­‐3	  ρ0)	
  densi7es
Energy per Neutron (MeV)60

Fermi Gas
40

TNI
AV18+UIX

20

AV18

0
0

0.1

0.2

-3

0.3

Neutron Density (fm )TNI	
  quite	  small	  (~	  4	
  MeV)	
  at	
  satura0on	
  density
	
  	
  	
  	
  	
  	
  	
  moderate	
  at	
  2x	
  satura0on	
  density	  (<	
  1/2	  EFG)Very	
  small	
  contribu7on	
  from	
  2π	
  TNI

28

Neutron	
  MaQer	
  EOS	
  and	
  3-­‐nucleon	
  interac7ons

Gandolfi, Carlson, Reddy29 2011

Very Strong Correlation between
Symmetry Energy and its Density Dependence

Lattimer, et al, 2012

Astrophysical Constraints
Neutron Star masses

Photospheric radius expansion bursts,
Thermal Emission

Demorest, et al, Nature, 2010
Steiner, Lattimer, Brown: Astrophys. J. 722, 33 (2010); Lattimer, Astrophysical Space Science (2011).

Mass/Radius Bands for Different Symmetry Energies
3
ty:
i
l
a
us

ρ

nt
ra =
l 5
ce

(G

ρ

2.5

.9
R>2

0

2

)
M/c

M (Msolar)

Ca

ρ ce

ρ0
=l 4

ra

nt

1.97(4) Msolar

2

ρ0
=l 3

tra

ρ cen

32

1.5

1.4 Msolarρ=l 2 0raρ cent

1

33.7

0.5
0
8

35.1
Esym= 30.5 MeV (NN)

9

10

11

12

R (km)

13

14

15

16

Combined constraints on EOS and Mass/Radiusfrom Nuclear Physics and Astrophysical Observations
2.5
Esym= 33.7 MeV
Esym=32 MeV

M (Msolar)

2

Polytropes
Quark matter

1.5
1
0.5
0
8

9

10

11

12

R (km)

13

14

15

Gandolfi and Steiner (PRL 2012)

EOS critical in gravity waves, supernovae, ...

Inhomogeneous Matter:
Neutron Star Crusts / Optical Lattices
0.8
GFMC (Gandolfi, et al.)
AFDMC (Gandolfi, et al.)
DMC (Gandolfi, et al.)
AFMC (present work)

0.7(E / ETF)

2

Neutrons
0.6
effective range

0.5

0.4

0

Cold Atoms

20

N

40

Cold atoms versus neutron drops

Can be used to optimize density functionals

Further	
  Direc7ons	
  in	
  Cold	  Atoms
Vortex	  Dynamics

Mul0ple	  Components

Bulgac	
  et	
  al.,	
  Science,	
  332,	
  1288	  (2011)hZp://www.physicstoday.org/resource/1/phtoad/v64/i8/p19_s1

Exo7c	
  pairing	
  phases

Nishida,	
  PRL	  (2012)

Viscosity	
  over	  Entropy
PIMC, Nx= 8
PIMC, Nx=10
Enss et al.
kinetic theory

1.4
1.2

0.8

S/N

!/n

1

0.6
0.4
0.2
0
J.	
  Pei	  et	
  al.,	
  Phys.	
  Rev.	
  A	
  82,	
  021603(R)	
  (2010)

0

0.1

0.2

0.3
0.4
0.5
0.6
Wlazłowski,
et al, PRL (2012)
T/"FFurther	
  Direc7ons	
  in	
  Nuclear	
  Physics:	
  Neutrino	
  Physics
Nuclear	
  structure	
  is	
  cri0cal:
Double-­‐beta	
  decay
neutrino-­‐nucleus	
  scajering

Low	
  to	
  High	  energy
Small	  to	
  Large	
  A

8

Spin	
  (weak)	
  Response	
  of	
  Neutron	
  MaQer

16

Neutrino-­‐Deuteron	
  ScaQering

-2

10

-3

10

Axial

" NC

-4

10

2

0.0015-5

10 50

0.0005

0

0

100

150

200

OPE
#PT
OPE: Eq. (4.6)
sum-rule
ind. fit

0.001

50

100

! [MeV]# (cm )

-1S" (!) [MeV ]

0.002

150

10

-39

(1+2)-body
1-body

_
" NC
200

-40

10 100

200

300

400

500 600!" (MeV)

700

800

900 1000

Figure 2: (Color online) The spin response function Sσ (q = 0, ω) of neutron matter at saturation density obtained by fitting
to AFDMC sum-rules using two diﬀerent ansatz are shown as the black solid and dashed curves. The inset comparesthe2:fits(color online) Total cross sections for NC-induced processes on the deuteron, obtained with the AV18 potential andFIG.
and the two-particle response at high energy obtained by confining two neutrons in a spherical cavity of radius 7 fm (red) or 8
theinclusionof one-body (dashed line) and (one+two)-body (solid line) terms in the weak current. Also shown are the total
fm (green). The linear, low-frequency forms predicted in Ref. [22], labeled as OPE and χPT are shown for comparison. The
cross sections obtained by retaining only the axial piece of the weak current. See text for explanation.
dot-dot-dashed curve is obtained using the two-body approach in Eq. (4.6) with OPE.

q~0	
  A=∞

q	
  very	  large,	
  A=2

36

the structure function obtained in Ref. [22] are shown for the two choices of C˜σ corresponding to the OPE TABLEand χPTVII: Total cross sections in cm2 for the NC-induced processes on the deuteron at selected initial neutrino energies,
potentials discussed earlier. The form of the low-frequency response in Eq. (4.1) is valid only at ω � EFobtained. In the with the AV18 or CDB potentials and the inclusion of one-body terms (1) only and both one- and two-body terms
figure we also show the results from the two-body approach (described in Eq. (4.6)) in the Born approximation(1+2)within the weak current.

Conclusions:

Important	
  progress	
  in	
  computa0onal	
  nuclear	
  physics	
  and	
  ourunderstanding	  of	
  nuclear	
  reac0ons.
Nuclear	
  Structure	
  is	
  a	
  fascina0ng	
  subject	
  with	  deep	  connec0ons	
  to:Many-­‐Body	  Theory:	
  Condensed	
  Majer/	
  Cold	  Atoms
Astrophysics:	  	
  r-­‐process,	
  neutron	
  stars,	
  supernovae
Neutrino	
  physics	
  and	
  fundamental	
  symmetries:	
  
	
  	
  	
  	
  	
  double-­‐beta	
  decay,	
  neutrino	
  oscilla0ons,	
  ...

37

Backup38

Resolution

Hot and dense quark-gluon matter
Hadron structure

ab initio
CI
DFT
collectivemodels

Effective Field Theory

Hadron-Nuclear interface

Nuclear structure
Nuclear reactions

Nuclear astrophysics
New standard model
Applications of nuclear science

Interfaces provide crucial clues

EXAMPLE: Universal Nuclear Energy Density Functional(other, smaller, collaborations exist: NuN, TORUS, PetaApps,…)

•	
  Ab	
  ini7o	
  structure•	  Ab	
  ini7o	
  func7onals•	
  DFT	
  applica7ons•	
  DFT	
  extensions•	  Reac7ons

Focus	
  on:	
  •Predic5ve	
  power•Robust	
  extrapola5ons•Valida5on•Guidance•Funded	  for	  5	
  years	
  by	
  DOE	
  (NP/SC,	
  NNSA,	
  ASCR)•9	
  universi0es	
  and	  7	
  na0onal	
  labs•Junior	
  scien0sts:	
  11	
  students,	
  19	
  postdocs/year•	
  ~50	  researchers	
  inØ physics
Ø computer	
  scienceØ applied	
  mathema0cs•Interna0onal	
  partners
For a popular description of UNEDF, see:
• SciDAC Review Winter 2007http://www.scidacreview.org/
0704/pdf/unedf.pdf• Nucl. Phys. News 21, No. 2, 24 (2011)• Office of Science “Highlight Series”: http://
science.energy.gov/news/in-focus/2011/03-28-11-s/

UNEDF	
  in	
  2007• As	
  UNEDF	
  has	
  matured,	
  it	
  has	  been	
  sa7sfying	
  to	
  see	  the	  increased	
  coherence	
  within	
  the	
  eﬀort;	
  the	
  project	  has	
  created	
  and	
  facilitated	
  an	
  increasing	
  interplay	
  among	  the	  major	
  sec7ons	
  where	
  none	  existed	
  previously.	
  	
  Each	  of	
  the	
  main	
  physics	
  areas	  now	  includes	
  collabora7ons	
  that	  cross	  over	  into	  other	  sectors.	
  • A	
  fair	  number	
  of	
  applica7ons	
  are	  now	  running	
  on	
  the	
  biggest	
  machines	
  available,	
  a	
  
vast	
  change	
  from	  when	  UNEDF	
  started.

UNEDF	  in	
  2011LACM, Fission: the ultimate challenge
Stability of the heaviest nuclei, r-process, advanced fuel cycle, stockpile stewardship…

Op7mized	
  Func7onals

Large-­‐scale	
  DFT

PRC 85, 024304 (2012)

PRC 84, 054321(2011)

PRC 80, 014309 (2009)

Numerical	
  Techniques

PRC 78, 014318 (2008)

Collec7ve	
  dynamics

Confronta7on	
  with	  experiment;	
  predic7ons
PRC 80, 014309 (2009)

Two DOE-sponsoredworkshops on exascalecomputing for nuclear physics,
one common conclusion:

The microscopic descriptions of
nuclear fusion and fission are
our Priority Research Directions.

“The ultimate outcome of the nuclear fission
project is a treatment of many-bodydynamics that will have wide impacts innuclear physics and beyond. The
computational framework developed in thecontext of fission will be applied to thevariety of phenomena associated with thelarge amplitude collective motion in nuclei
and nuclear matter, molecules,
nanostructures and solids”“Fusion reactions in light nuclei
are critical for both basic
science and NNSA applications.
These reactions aid inunderstanding the early
universe and provide theenergy that powers the stars.
These reactions are alsoessential elements of NNSA
science..”Quest for understanding the neutron-richmatter on Earth and in the Cosmos
78 >R
10

10

B(GT +)
B(GT -)

gpp = 0

4

β + (×50)

2

Phys.	
  Rev.	
  C	
  85,	
  032801(R)	
  (2012)

Nuclearinterac7ons

Many-­‐body	  theoryβ−

4VSXSRRIYXVSR TEMVMRKMRGVIEWIW PS[IRIVK]WXVIRKXL

640

gpp = 327

8

B (GT)

6

B(GT +)
B(GT -)B (GT)

8

β−

0

5

10

β + (×50)

2

15
EQRPA

20

25

30

Nuclear	
  maQer	  equa7on	  of	
  stateNeutron	  star	  crustMicrophysics(transport,…)

0
0

5

10

15
EQRPA

20

25

30

RNB
Facilities/
NuN topicalcollaboration

Nuclear	  observables
Astronomical	
  observables

Phys.	
  Rev.	
  LeQ.	
  108,	
  081102	
  (2012)

Interna5onal	
  Impact	
  of	
  UNEDF• JUSTIPEN	
  (Japan)• FUSTIPEN	  (France)• FIDIPRO	
  (Finland)• individuals	
  worldwide• Annual collaboration meetings of UNEDF with JUSTIPEN and
FIDIPRO• Helping our Japanese colleagues to sharpen the case fornuclear theory supercomputing (K-machine)• Significant impact on low-energy nuclear theory effort
worldwide (example: INPC 2010)• Unique worldwide (no other program has such a broad scopeand such close PHY/CS/AM partnerships)

What	
  makes	
  this	  possible?
• Computa7onal	  Resources• Physicists	
  (par7cularly	
  early	
  career	  are	
  cri7cal)• Math/CS	
  research	
  and	
  people	
  (Lusk)Total	
  Resource	
  Usage	
  /	
  Request	
  /	
  Needs
2009

2010

2011

2012	
  
(alloc)2013(request)

Intrepid

12

	
  	
  9

14

19

	
  45

Jaguar

22

28

50

35

	
  65

	
  

	
  	
  

	
  	
  

	
  	
  

	
  	
  

NERSC

	
  	
  2

	
  	
  3

	
  	
  3

	
  	
  4

	
  	
  10

Jaguar

20

25

35

50

150

LLNL,	  	
  LANL,	
  ANL,	
  ORNL
TOTAL

20

25

25

30

	
  	
  40

70

90

120

140

300

INCITEIn	
  million	
  core-­‐hours	
  (approximate)Nuclear	
  Reac5ons

Microscopic	
  Theoryof	
  Fission

2012

2012

2020 2020

2030 2028 2020

Desired	
  TrajectoryFlat	
  TrajectoryFlat-­‐Flat	  Trajectory

2020 2020

2030 2028 2020

Neutrinos	
  as	
  Nuclear
Physics	
  Laboratories

20122020 2020

2030 2028 2020

Physics	
  of	
  Extreme	  Neutron-­‐Rich
Nuclei	  and	
  Neutron	  Stars

2012

Desired	
  TrajectoryFlat	
  TrajectoryFlat-­‐Flat	  Trajectory

2020 2020

2030 2028 2020

Junior Scientists in UNEDF
POST-DOCTORAL ASSOCIATES (2010)
Christopher Calderon, LBNL (staff, Numerica co.)Joaquin Drut (Professor, UNC)
Stefano Gandolfi, LANL (staff, LANL)
Kai Hebeler, OSU (TRIUMF)
Heiko Hergert, MSU (OSU)
2010: Early
Jason Holt, UTK/ORNL
Eric Jurgenson, LLNL (staff, LLNL) Career Award
Markus Kortelainen, UTK (U. Jyväskylä)
Plamen Krastev, UCSD (research, Harvard)
Pieter Maris, ISU (Research Prof. ISU)
Eric McDonald, MSU (staff scientist, MSU)
Gustavo Nobre, LLNL (BNL, NNDC)
2011: Faculty
Junchen Pei, UTK (Prof., Pekin U.) UNC/Chapel
Nicolas Schunck UTK (staff, LLNL) Hill
Roman Senkov, CMU
Ionel Stetcu, UW (staff, LANL)
Jun Terasaki, UNC (staff, U. Tsukuba)
Stefan Wild, ANL (staff, ANL)
2012: Faculty
GuelphRelevant instruction (workshops, courses)
is crucial for the future of the field

Effect of UNEDF on workforce

Year-1: 9 students, 17 postdocs;
Year-2: 12 and 12;
Year-3: 10 and 18;
Year-4: 11 and 19

2010:
Math/CS
Staff ANL

2010:Staff
LLNL

2012:
HarvardResearch
ComputingMathema5cs	
  and	  Computer	
  Science	
  in	
  
Two	
  Nuclear	
  Physics	
  SciDAC	
  Projects
UNEDF:	
  	
  2007-­‐2011
NUCLEI:	
  	
  2012-­‐2016	
  (hopefully)Outline• The	
  UNEDF	  project	  just	  ending– goals– structure– Math/CS	
  areas• Some	  Speciﬁc	
  accomplishments• The	
  NUCLEI	
  project	  just	  star7ng	
  (no	
  logo	  yet)• Lessons	
  learned

Universal	
  Low-­‐Energy	  Density	
  Func5onal	
  	
  (UNEDF)• Sponsors:	
  	
  NP,	  ASCR,	
  and	
  NNSA• Math/CS	
  goals	
  of	
  this	  SciDAC-­‐2	
  project:– Improve	
  algorithms	
  and	  so^ware	
  in	
  support	
  of	
  moving	
  codes	  (exis5ng,	
  for	
  the	
  most	  part)	
  to	
  petascale	
  compu5ng	
  facili5es,	
  thus	  enabling	  new	  scien5ﬁc	
  results– Achieve	  results	
  in	
  mathema5cs	
  and	
  computer	
  science	  applicable	
  outside	  the	
  project• Structure:– Not	  a	  Math/CS	
  team	  and	  a	
  physics	
  team– Rather,	
  a	
  collec5on	
  of	
  speciﬁc	
  projects	
  requiring	
  math/CS	
  and	
  physics	
  both,	
  5ed	
  together	
  by	
  an	
  overarching	
  physics	
  objec5ve

Math/CS	
  Areas•••••Op0miza0on
Load	
  Balancing
Eigenvalues
Frameworks
Programming	
  Models• Each	  one	  part	  of	
  a	
  speciﬁc	
  physics	
  subproject	
  of	  UNEDFComputer	
  Science	
  in	
  UNEDF	  •Several	  partnerships;	
  model	  collabora7ons•Collabora7ons	
  established	
  before	
  proposal	
  was	
  submiQed
Green's	
  Func7on	  Monte	  CarloPhysics	
  challenge:	
  ab-­‐ini7o	
  descrip7on	
  of	
  12C,	
  14C,	
  16OComputa7onal	
  challenges:	
  800,000	
  core	  hours	
  each	  on	
  Argonne‘s	
  IBM	  BlueGene/P	
  for	
  12C.	
  14C	  is	
  60	  7mes	  harder,	
  and	  	
  16O	  is	
  more	  than	
  1,000	
  7mes	  harder.
Exploi7ng	
  BG/Q	
  will	  require	  a	
  redesign	
  of	
  the	
  ADLB	
  Load	  Balancing	
  Library.
Coupled-­‐cluster	
  Code	  Suite	  Physics	
  challenge:	
  ab-­‐ini7o	
  descrip7on	
  of	
  structure	
  and	
  reac7on	
  proper7es	
  of	
  medium-­‐mass	
  and	
  heavy	  nuclei.
Computa7onal	
  challenges:	
  itera7vely	
  solve	
  coupled,	
  non-­‐linear	
  equa7ons	
  scaling	
  roughly	  as	
  O(N4).	
  Current	
  calcula7ons	
  include	
  N~1,000	
  (oxygen	
  and	
  calcium	
  isotopes)	
  with	  calcula7ons	
  planned	
  to	
  N~7,000	
  (nickel	
  and	
  7n	
  isotopes).	
  	
  To	
  address	
  scaling	
  issues	
  and	
  facilitate	
  feature	
  addi7ons,	
  a	
  generalized	
  tensor	
  contrac7on	
  engine	  (TCE)	
  and	
  global	
  arrays	
  will	  be	
  implemented.	
  Improved	
  I/O,	
  checkpoint	
  and	
  fault	
  tolerance	
  tools	
  will	  be	
  required,	
  as	
  well	  as	
  adap7ve	
  mesh	  technologies	
  for	  collision	
  modeling.
No-­‐Core	
  Shell	
  ModelPhysics	
  challenge:	
  ab-­‐ini7o	
  descrip7on	
  of	
  light	
  nuclei	  Computa7onal	
  challenges:	
  typically	
  solve	
  for	  the	
  lowest	
  10-­‐20	
  eigenvalues	
  and	
  eigenvectors	
  of	
  a	
  large	  sparse	
  symmetric	
  matrix	
  with	  dimensions	
  up	
  to	
  1010.	
  A	
  single	
  case	  can	
  run	
  for	
  about	
  5	
  hours	
  of	
  wallclock	
  7me	
  using	
  the	
  full	  Jaguar	  PF	
  machine	  (~240,000	
  cores)	
  including	
  post	  processing.	
  Future	
  applica7ons	
  will	  involve	
  signiﬁcantly	
  increased	
  matrix	
  dimensions.

Nuclear	
  Density	
  Func5onal	
  Theory
Physics	
  challenge:	
  global	
  descrip5on	
  of	
  nuclear	
  proper5es	
  and	
  decays	
  (including	
  ﬁssion)	
  throughout	
  the	
  nuclear	
  landscape	
  with	  the	
  focus	
  on	
  medium-­‐mass	
  and	
  heavy	  nuclei.
Computa5onal	
  challenges:	
  huge	  systems	  (over	
  107)	
  of	
  integro-­‐diﬀeren5al	
  equa5ons	
  per	  conﬁgura5on	
  	
  solved	
  itera5vely.	
  This	  is	
  usually	
  done	  by	  	
  projec5on	
  into	
  a	
  pre-­‐determined	
  basis	
  with	  subsequent	
  computa5ons	
  employing	
  mostly	  dense	
  matrices.	
  Various	
  boundary	
  condi5ons	
  assumed	
  leading	  to	  discrete,	
  resonance,	
  and	
  scaZering	  	
  problems.	
  A	
  realis5c	
  descrip5on	
  of	
  ﬁssion	
  requires	
  up	
  to	
  109	  	
  conﬁgura5ons.	
  Requirements:	
  scalable	
  IO;	
  fault	  tolerance;	
  on-­‐the-­‐ﬂy	
  simula5on	
  steering;	
  high-­‐dimensional	
  array	
  manipula5ons	
  via	
  tensor	
  contrac5ons;	
  novel	  use	  of	
  visualiza5on	
  tools	
  for	
  analysis.	
  The	
  use	  of	
  adap5ve	
  mul5-­‐resolu5on	
  techniques	
  such	  as	
  MADNESS	
  framework	
  will	  be	
  required	  to	
  solve	
  extreme	
  problems.	
  MADNESS	
  parallel	
  run5me	
  ensures	
  eﬃcient	
  execu5on	
  on	
  leadership	
  computers.	
  	
  Nuclear	  Density	
  Func5onal	  Op5miza5ons
Physics	
  challenge:	
  Simula5on-­‐based,	
  deriva5ve-­‐free	
  op5miza5on	
  involving	
  10-­‐20	
  parameters;	
  sensi5vity	
  analysis	
  for	
  uncertainty	
  and	
  correla5on	
  es5ma5ons.
Computa5onal	
  challenges:	
  Eﬃcient	
  parameter	
  es5ma5on	
  and	
  mul5level	
  op5miza5on	
  algorithms	
  for	
  large	  scale	  (many	
  core)	
  DFT	
  calcula5ons.	
  Es5ma5ng	
  deriva5ves	
  of	
  noisy	
  simula5ons.
Time-­‐dependent	  Density	
  Func5onal	
  Theory
Physics	
  challenge:	
  Dissipa5ve	
  nuclear	
  processes	  -­‐	
  nuclear	
  ﬁssion	
  and	
  fusion,	
  nuclear	
  reac5ons,	
  and	
  5me-­‐dependent	
  	
  phenomena	
  in	
  	
  nuclei	
  and	  	
  related	
  systems	
  (cold	
  gases,	
  neutron	
  stars)
Computa5onal	
  challenges:	
  Determina5on	
  of	
  nuclear	
  ground	
  states	
  requires	
  full	  diagonaliza5on	
  of	
  large	
  matrices	
  (O(106)	
  and	
  higher)	
  -­‐	
  one	  diagonaliza5on	
  requires	
  the	
  whole	  	
  JaguarPF	
  for	  several	
  hours	
  today;	
  eﬃcient	
  I/O	
  for	
  large	  data	
  volumes	
  for	
  check	
  point	  and	  restart;	
  in	
  situ	  analysis	
  for	
  reduc5on	
  of	
  large	
  data	  and	
  novel	  use	  of	
  visualiza5on	
  tools	
  for	
  analysis.	
  A	
  real	  5me	
  stochas5c	
  formula5on	
  of	
  the	
  current	
  approach	
  will	  require	
  new	
  communica5on	
  algorithms	
  for	
  reduc5ons	
  and	
  overlaps,	
  fault	
  tolerance,	
  in	
  situ	  data	  reduc5ons,	
  likely	
  higher	
  precision	
  numerics,	
  and	
  improved	
  I/O.

Deriva5ve-­‐free	
  Op5miza5on	
  for	
  Energy	
  Density	
  Func5onal	
  Calibra5onNew	
  algorithm	
  for	
  tuning	
  large-­‐scale	
  nuclear	
  structure	
  simula5ons	
  turns	
  days	
  into	  hours§ Previous optimizations required too many evaluations to obtain
desirable features exhibited by UNEDF0, UNEDF1, …
§ Derivative-free sensitivity analysis procedure developed for
UNEDF exposes correlations and constraining data in 1 minuteusing 20k cores• “Nuclear Energy Density Optimization” Kortelainen et al., PhysRevC ‘10.
• “N.E.D.O.: Large Deformations” Kortelainen et al., PhysRevC ‘12.
• “Occupa(on	
  Number-­‐based	
  Energy	
  Func(onal	
  for	
  Nuclear	
  Masses” Bertolli et al.,PhysRevC ‘12.

Theoretical and Computed Mass Difference§ Energy density functional (EDF) predictions rely on largescale computer simulations that must be calibrated toexperimental data§ TAO 2.0’s POUNDERS developed for UNEDF to exploit themathematical structure of this calibration problem§ Substantial computational savings over alternativealgorithms enables fitting of complex EDFs

Op0mal	
  Deriva0ves	
  of	
  Noisy	
  Numerical	
  Simula0ons	
  Computational NoiseIn all computations of DOE
interest containing§ adaptivity,§ discretizations§ iterative methods§ petaflops,§ roundoff errorsvIncludes deterministic computations
Our optimal forward difference parameter

Noise Impacts in UNEDF & Beyond§ Uncertainty in computed outputs§ Unstable derivative estimates forsensitivity analysis§ Can be unrelated to/overwhelm
truncation error§ Blurs relationship between
tolerance values and stability

Tools & Techniques§ ECNoise provides reliable estimates of stochastic anddeterministic noise in few simulations§ Nonintrusive stability bounds for extreme scale simulations,can instruct precision levels/tolerances for subroutines§ Optimal difference parameters calculated without computationallyexpensive parameter sweeps
Optimal step was obtained with only two simulation
evaluations. Classical approach (circles): result of a sweep
across 100 difference parameters, each point requiring a new
simulation.
“Estimating Computational Noise,” Moré & Wild, SIAM Sci. Comp., ’11“Estimating Derivatives of Noisy Simulations,” Moré & Wild, ACM TOMS‘12

Making a simple programming model scalable:
The Asnychronous Dynamic Load Balancing Library
Objec(ves	
  Impact§ Enable	
  Green’s	
  Func5on	
  Monte	  Carlo	
  calcula5ons	
  for	
  12C	
  on	
  
full	
  BG/P	
  as	
  part	  of	
  UNEDF	  project§ Demonstrate	
  capabili5es	
  of	
  simple	
  programming	  models	  at	
  petascale	
  and	  beyond§ Show	
  path	  forward	
  with	  hybrid	
  programming	  models	
  in	
  library	
  implementa5on§ Simplify	
  programming	  model§ Scale	
  to	
  leadership	
  class	
  machines
Improved	
  Eﬃciency	
  (compute	
  7me/wall	
  7me)	
  with	  more	  nodes

Deployment	
  of	
  Detec7on	  Network

Progress§ Initial load balancing was of
CPU cycles§ Next it became necessary tobalance memory utilization as
well§ Finally ADLB acquired thecapability to balance message
flow§ “More Scalability, Less Pain” by
E. Lusk, S.C. Pieper and R.
Butler published in SciDAC
Review 17, 30 (2010)

MFDn:	
  Total-­‐J	
  Progressv	  M-­‐scheme	
  approach:	
  works	
  directly	
  on	
  the	
  Hamiltonian,	  extracts	  all	
  low	
  energy	
  statesv	
  J-­‐scheme	
  approach:	
  alterna7ve	
  to	
  ﬁnd	  a	
  large	
  number	
  of	
  low	
  energy	
  states	
  for	
  a	
  	
  	
  	
  
	
  	
  	
  	
  prescribed	
  total	
  angular	
  momentum	
  (J)	
  valuev	
  Targeted	
  applicaGons:	
  inves7ga7on	
  of	
  nuclear	
  level	  densi7es,	
  
	
  	
  	
  	
  evalua7on	
  of	
  scaQering	
  amplitudesv	
  Total-­‐J	
  code:	
  implementa7on	
  of	
  the	
  J-­‐scheme	
  approach	
  in	
  Fortran,	
  MPI
v	
  CS	
  Challenges	
  in	
  Total-­‐J:	
  Three	  dis7nct	
  phases,	
  each	  with	  very	  diﬀerent	
  compu7ng	
  
	
  	
  	
  	
  and	
  storage	
  characteris7cs

Phase	  1:	  ConstrucGon	
  of	
  
the	
  J-­‐basis

Implemented	
  a	
  mul7-­‐level	
  greedy	  load	  balancing	  algorithm

Phase	  2:	  Invariant	  Subspace	  ProjecGon

In-­‐core	
  implementa7on	
  to	
  reduce	  I/O	
  overheads	
  in	
  the	  out-­‐of-­‐core	  version

6Li,	
  N
max=14,	
  J=3

dimension	
  of	
  H	
  ~	
  1.7x108
10x	
  speed-­‐up,	
  default	
  striping
4x	
  speed-­‐up,	
  op7mal	
  striping

ApplicaGon:
Predic5ng	
  the	
  Nuclear	
  Level	  Density	
  of	
  6Li
(PRELIMINARY)M-­‐scheme:	
  Performance	
  Improvements
ver1:	
  SC’08	
  paper	  (beginning	  of	
  SciDAC2)

ver6:	
  current	
  release	
  
(end	
  of	
  SciDAC2)

g
in
l
a
c
s
	
  
al
e
id
beWer

Weak	
  Scaling

Strong	  Scaling	
  for	
  10B,	
  Nmax	
  =	
  8

One	
  Demonstra0on	
  of	
  NP	
  –	
  ASCR	  Coupling:
Over	
  20	
  joint	
  publica0ons	
  resul0ng	
  from	  UNEDF• "Real-­‐Time	  Dynamics	
  of	
  Quan5zed	
  Vor5ces	
  in	
  a	
  Unitary	  Fermi	  Superﬂuid,"	
  A.	
  Bulgac,	
  Y.-­‐L.	
  Luo,	
  P.	
  Magierski,	
  K.J.	
  Roche,	  and	  Y.	
  Yu,	
  Science	  332,	
  1288	  (2011).
• "Origin	
  of	
  the	
  anomalous	
  long	  life5me	
  of	
  14C,"	
  P.	
  Maris,	
  J.	
  P.	
  Vary,	  P.	
  Navra5l,	
  W.	
  E.	
  Ormand,	
  H.	
  Nam,	
  and	  D.	
  J.	
  Dean,	
  Phys.	
  Rev.	
  LeZ.	
  106,	
  202502	
  (2011).
• "More	  Scalability,	
  Less	  Pain,"	
  E.	
  Lusk,	  S.C.	
  Pieper	
  and	  R.	
  Butler,	
  SciDAC	
  Review	
  17,	  30	  (2010).	
  • "Nuclear	
  Energy	
  Density	  Op5miza5on,"	
  M.	
  Kortelainen,	
  T.	
  Lesinski,	
  J.	
  More,	
  W.	
  Nazarewicz,	
  J.	
  Sarich,	
  N.	
  Schunck,	
  M.	
  V.	
  Stoitsov,	
  and	  S.	
  Wild,	
  Phys.	
  Rev.	
  C	
  82,	
  024313	
  (2010).
• "One-­‐quasipar5cle	
  States	
  in	
  the	
  Nuclear	
  Energy	
  Density	
  Func5onal	
  Theory,"	
  N.	
  Schunck,	
  J.	
  Dobaczewski,	
  J.	
  McDonnell,	
  J.	
  More,,	
  W.	
  Nazarewicz,	
  J.	
  Sarich,	
  and	
  M.V.	
  Stoitsov,	
  Phys.	
  Rev.	
  C	
  81,	
  024316	
  (2010).
• "Scaling	
  of	
  ab-­‐ini5o	
  nuclear	
  physics	
  calcula5ons	
  on	
  mul5core	
  computer	
  architectures,"	
  P.	
  Maris,	
  M.	
  Sosonkina,	
  J.	
  P.	
  Vary,	
  E.	
  G.	
  Ng	
  
and	
  C.	
  Yang,	
  Interna5onal	
  Conference	
  on	
  Computer	
  Science,	
  ICCS	
  2010,	  Procedia	
  Computer	
  Science	
  1,	
  97	
  (2010).
• "Hamiltonian	
  light-­‐front	
  ﬁeld	  theory	
  in	
  a	
  basis	
  func5on	
  approach,"	
  J.	
  P.	
  Vary,	
  H.	
  Honkanen,	
  Jun	
  Li,	  P.	
  Maris,	
  S.	
  J.	
  Brodsky,	
  A.	
  Harindranath,	
  G.	
  F.	
  de	
  Teramond,	
  P.	
  Sternberg,	
  E.	
  G.	
  Ng,	  C.	
  Yang,	
  Phys.	
  Rev.	
  C	
  81,	
  035205	
  (2010).
• "Ab	
  ini5o	
  nuclear	
  structure:	
  The	  Large	  sparse	
  matrix	
  eigenvalue	
  problem,"	
  J.P.	
  Vary,	  P.	
  Maris,	
  E.	
  Ng,	  C.	
  Yang,	
  and	  M.	
  Sosonkina,	
  J.	
  Phys.	
  Conf.	
  Ser.	
  180,	
  012083	
  (2009).
• "Fast	  Mul5resolu5on	
  Methods	
  for	
  Density	
  Func5onal	
  Theory	
  in	
  Nuclear	
  Physics,"	
  G.	
  I.	
  Fann,	
  J.	
  Pei,	
  R.	
  J.	
  Harrison,	
  J.	
  Jia,	
  J.	
  Hill,	  M.	
  Ou,	  W.	
  Nazarewicz,	
  W.	
  A.	
  Shelton,	
  and	  N.	
  Schunck,	
  Journal	
  of	
  Physics:	
  Conference	  Series	  180,	
  012080	
  (2009).
• "Solu5on	
  of	
  the	  Skyrme-­‐Hartree-­‐Fock-­‐Bogolyubov	
  equa5ons	
  in	
  the	  Cartesian	  deformed	
  harmonic-­‐oscillator	
  basis.	
  (VI)	
  HFODD	  (v2.38j):	
  a	
  new	  version	
  of	
  the	
  program,"	
  J.	
  Dobaczewski,	
  W.	
  Satula,	
  B.G.	
  Carlsson,	
  J.	
  Engel,	
  P.	
  Olbratowski,	
  P.	
  Powalowski,	
  M.	
  Sadziak,	
  J.	
  Sarich,	
  N.	
  Schunck,	
  A.	
  Staszczak,	
  M.	
  Stoitsov,	
  M.	
  Zalewski,	
  H.	
  Zdunczuk,	  Comp.	
  Phys.	
  Comm.	
  180,	
  2361	
  (2009).
• "Towards	
  The	
  Universal	
  Nuclear	
  Energy	
  Density	
  Func5onal,"	
  M.	
  Stoitsov,	
  J.	
  More,	
  W.	
  Nazarewicz,	
  J.	
  C.	
  Pei,	
  J.	
  Sarich,	
  N.	
  Schunck,	
  A.	
  Staszczak,	
  and	  S.	
  Wild,	  Journal	
  of	
  Physics:	
  Conference	  Series	  180,	
  012082	
  (2009).
• "Deformed	
  Coordinate-­‐Space	
  Hartree-­‐Fock-­‐Bogoliubov	
  Approach	
  to	
  Weakly	
  Bound	
  Nuclei	
  and	  Large	  Deforma5ons,"	
  J.C.	
  Pei,	
  M.V.	
  Stoitsov,	
  G.I.	
  Fann,	
  W.	
  Nazarewicz,	
  N.	
  Schunck	
  and	
  F.R.	
  Xu,	
  Phys.	
  Rev.	
  C	
  78,	
  064306	
  (2008).
• "Time-­‐Dependent	  Density	
  Func5onal	
  Theory	
  Applied	
  to	
  Superﬂuid	
  Nuclei,"	
  A.	
  Bulgac	
  and	
  K.J.	
  Roche,	
  J.	
  Phys.:	
  Conf.	
  Ser.	
  125,	
  012064	
  (2008).
• ...

A	
  SciDAC-­‐3	
  Project:
Nuclear	
  Computa5onal	
  Low	
  Energy	
  Ini5a5ve	  (NUCLEI)• Mostly	
  same	  cast	  as	
  UNEDF• Targe0ng	
  next	  genera0on	
  architectures	
  on	
  path	  to	
  exascale• Examples	
  of	
  speciﬁc	
  plans	
  in	
  Math/CS…

Optimization Plans in NUCLEINew Optimization Capabilities§ State-of-the art mathematical/numerical
optimizations of next-generation EDFs
(with LLNL, ORNL, Tennessee)§ Optimization of basis states and nonperturbative coupling constants arisingin chiral Hamiltonians (with ISU)§ Enable nucleus lifetime computations
with collective action minimizations (with
LLNL, others)

Coupling NUCLEI Subgroups§ Incorporate new observables from various
NUCLEI subgroups§ giant resonance data,§ binding energy of neutron droplets ina trap,§ …

Deploy code optimization tools§ Exploit additional parallelism at thesimulation-optimization interface§ Extend POUNDERS to address missing
states and available sensitivity
information§ Incorporate uncertainties and QUEST
technologies§§Introduce performance, energy, andresilience tools developed by the SUPERSciDAC Institute for use in NUCLEI codes
Deliver representative
NUCLEI computational
kernels to SUPERFuture	
  Plans	
  for	  the	  Asynchronous	
  Dynamic	
  Load	  Balancing	
  Library• The	
  Exascale	
  target	
  for	
  GFMC	  (Green’s	
  Func0on	
  Monte	  Carlo)	
  
code	
  is	
  16O.
• The	
  overall	  structure	
  of	
  GFMC	
  can	
  remain	
  the	
  same	
  if	
  – Eﬀec5ve	
  use	  of	
  many-­‐core	
  nodes	
  can	  be	
  made• may	
  need	  to	
  go	
  beyond	
  current	
  OpenMP	
  approach– ADLB	
  can	
  support	
  mul5-­‐node	
  parallelism	
  on	
  single	
  work	  units• This	
  will	  require	
  major	
  changes	
  to	
  ADLB	
  that	  take	  advantage	
  of	
  
new	
  features	
  in	
  MPI-­‐3.

MFDn:	
  Future	
  Work• 	
  Version	
  14	
  (under	
  development)• Eﬃcient	
  construc0on	
  of	
  the	
  Hamiltonian,	
  faster	  SpMV• 3-­‐body	  interac0on	
  support	
  is	
  underway	
  • 	
  MFDn	
  on	
  diﬀerent	
  architectures• Inves0ga0ng	
  the	
  use	  of	
  GPU	
  (Titan)	
  and	
  BG/Q	
  (Mira)	
  platorms• Out-­‐of-­‐core	
  implementa0on	
  on	
  SSD-­‐equipped	
  clusters• 	  Compression	
  of	
  the	
  Hamiltonian	  to	
  enable	
  the	
  study	
  of	
  	
  
heavier	
  nucleiSummary	
  (for	  Math/CS)• The	
  structure	
  of	
  the	
  UNEDF	  project	
  was	  the	
  key	  to	
  its	  success– Math/CS//Physics	
  partnerships	
  formed	  locally	
  at	
  proposal-­‐wri5ng	  stage,	
  and	
  developed	
  over	  the	
  5-­‐year	
  5meframe	
  of	
  the	
  project• Signiﬁcant	  science	
  resulted• Future	
  looks	
  bright– Many	  hard	  steps	  now	  behind	
  us• Collabora5ons	
  quite	  deep	
  at	
  this	  point• Codes	  have	  been	
  signiﬁcantly	
  improved	
  for	
  scalability– Many	  interes5ng	
  steps	
  ahead	  of	
  us• Exascale	
  provides	
  new	
  set	  of	
  challenges

NUclear	
  ComputaGonal	
  Low-­‐Energy	
  IniGaGve	
  	
  fusion

TJNAF
NIF

Validated	
  NuclearInterac5ons
Chiral	  EFTAb-­‐iniGo

Structure	
  and	
  Reac5ons:
Light	
  and	
  Medium	
  Nuclei
Stellar	
  burningAb-­‐iniGo
RGM
CI

OpGmizaGon
Model	
  validaGon
Uncertainty	
  QuanGﬁcaGon

Neutron	  dropsEOS
CorrelaGons

Load	
  balancing
Eigensolvers
Nonlinear	
  solvers
Model	
  validaGon
Uncertainty	
  QuanGﬁcaGon

Neutrinos	
  and	  Fundamental	
  Symmetries

SNS
Majorana

LENP	
  faciliGes

FRIB

Structure	
  and	
  Reac5ons:
Heavy	
  Nuclei
DFT
TDDFT

Load	
  balancing
OpGmizaGon
Model	
  validaGon
Uncertainty	
  QuanGﬁcaGon
Eigensolvers
Nonlinear	
  solvers
MulGresoluGon	
  analysis

Fission

Neutron	  Stars

LANL
LLNL