# Presentation: 861103

## DWF on BG/L and QCD Thermodynamics

**P. Vranas**
**IBM Watson Research Lab**
**LLNL EOS 2/2006**

- P. Vranas, IBM Watson Research Lab

## Lattice gauge theory

**:**

- P. Vranas, IBM Watson Research Lab

## Symmetries and limits

-  The Lorentz symmetry is the symmetry of continuous space-time:
-     It is recovered fast as a      0.
-  Gauge symmetry is a local internal symmetry related to color:
-     It is exact.
-  Flavor symmetries are global internal symmetries:
-    Some are exact.
-    Some are not exact and are recovered ***very slowly*** in the a      0 limit.
-  If we want to decrease a to a/2 we must double the lattice sites per
-    direction in order to keep the physical volume fixed:
-    This does not just cost a factor of 2**4** but for algorithmic reasons
-    it costs a factor of 2**6 **to 2**10**.

- P. Vranas, IBM Watson Research Lab

## Lattice fermions

**:**

- P. Vranas, IBM Watson Research Lab

## The fly in the ointment: fermion doubling

**The fly in the ointment: fermion doubling**

- P. Vranas, IBM Watson Research Lab

## Staggered fermions

-  The Dirac operator can be “spin diagonalized”:
-     Thin-out the degrees of freedom by 1⁄4  => 16/4 = 4 flavors.
-  4-flavor, 4-spin components => the 16 corners of a lattice hypercube.
-  At finite lattice spacing:

- U(4)**L**x U** **(4)**R ** symmetry explicitly broken to U(1)**o**x U** **(1)**e **
-  Broken flavor symmetry (non-degenerate pions). At finite temperature one pion is light but the rest have masses > critical temperature.
-  To go from 4 flavors to 2 flavors one takes               . The action is not local and the operators do not correspond to the action.
-  Instanton – zero mode properties?

-  The remaining chiral symmetry is exact. One Goldstone pion
-    with mass => 0 with m=> 0. No fine tuning.
-  The degrees of freedom are thinned-out.
-  Improved staggered actions.
-  Experience by staggered groups.

- P. Vranas, IBM Watson Research Lab

## Domain wall fermions (DWF)

-  Continuum Dirac operator:

-  States describing massless propagation must satisfy:

- D.B. Kaplan PLB 288 (1992) 342

- P. Vranas, IBM Watson Research Lab

## Defect localization

-  Because               is non-normalizable we have only one + chirality massless mode “bound” exponentially on the wall. Same on the lattice.

**o**

**o**

**- m****0**

**+ m****0**

***s***

***m******0***

**o**

**o**

***s***

- P. Vranas, IBM Watson Research Lab

## How do we add gauge fields?

-  5-dimensional gauge fields do not work **???**
-  4-dimensional gauge fields. No 5th component. Constant along 5th dir.
-  The “s” space can be thought of as an internal flavor space.

- P. Vranas, IBM Watson Research Lab

## Overlap formalism

-  Four-D gauge fields. Think of 5th dimension as internal flavor space.

**R. Narayanan, H. Neuberger, PLB 302 (1993) 62**

-   ***Vector theory:***

-   ***Chiral theory:***

**x**

**x**

- P. Vranas, IBM Watson Research Lab

## Index and topology changing

-  When a configuration changes from one sector to another H**+** develops a zero eigenvalue:
-  T**+** has a unit eigenvalue.
-  The walls do not “decouple” even at infinite L**s**.
-  Chirality cannot be restored>

-  Topology changing configurations are of zero measure.
-  But on the lattice instantons of size 1 appear/disappear copiously at strong coupling. At weaker coupling the phenomenon is diminished.
-  Configurations in their vicinity have slow decoupling.
- L**s** must be large in order to have small chiral symmetry breaking.

- P. Vranas, IBM Watson Research Lab

## DWF as a tool for lattice QCD

**DWF as a tool for lattice QCD**

-  Restoration of Lorentz symmetry : lim(a=0)
-  Restoration of chiral symmetry     : lim(L**s**= infinity)
-  The two limits are decoupled  !!!
-  Computing is only linear in L**s **!!!
-  Wilson or staggered need 2**6** - 2**10** more computing for a <= a/2.

-  In free theory:

**m****f**

**m****0**

**m****0**

**<=    5****th**** direction   =>**

**P. Vranas, PRD 57 (1998) 1415**

**V.Furman, Y. Shamir,**
**NPB 439 (1995) 54**

- P. Vranas, IBM Watson Research Lab

## Index theorem on the lattice

-  Can the continuum Atiyah-Singer index theorem be carried over to the lattice in a probabilistic sense? SU(2) overlap index:

-  P(Q): from topological charge = squares, from index = diamonds

**R. Narayanan, P. Vranas, NPB 506 (1997), 373****.**

- P. Vranas, IBM Watson Research Lab

## Zero modes on the lattice

-  On an 8**4** lattice configuration constructed to have one instanton:

-  Staggered

-  DWF

**Ls=4**

**Ls=6**

**Ls=8**

**Ls=10**

**P. Vranas and Columbia U., PRD 59 (1999) 054508**

- P. Vranas, IBM Watson Research Lab

## Zero modes in quenched QCD

-  Since there is no fermionic determinant to suppress the zero-modes, the chiral condensate should diverge for small m**f** as 1/ m**f.**

**T > T****c**

**T = 0**

**P. Vranas and RBC, PRD69 (2004) 074502**

**P. Vranas and Columbia U., hep-lat/9812011**

- P. Vranas, IBM Watson Research Lab

## Chiral symmetry restoration

-  How much is the exponential rate of L**s **for chiral symmetry restoration?
-  How does it depend on the lattice spacing (coupling)?
-  First application of DWF was on the Schwinger model:

**P. Vranas, PRD57 (1998) 1415**

- P. Vranas, IBM Watson Research Lab

## Eye diagrams

-  Spectral flow of the Hamiltonian on quenched QCD configurations.
-  At weaker coupling  (                  )  there is a gap.

**R. Edwards, U. Heller, R. Narayanan, NPB 535 (1998) 403.**

- P. Vranas, IBM Watson Research Lab

## Victims of our own devices

**:**

- P. Vranas, IBM Watson Research Lab

-  What we can do depends largely on the speed of our computers.

- P. Vranas, IBM Watson Research Lab

## DWF vs. Staggered 

-  Both methods have their limitations but we are now at the Teraflops regime.
-  DWF have the chiral structure correct by construction but they are expensive to simulate for light pions. As a result one may have to use smaller volumes.
-  Staggered are less expensive to simulate and can reach larger volumes. But staggered may not get the chiral structure correctly at the presently accessible lattice spacings.
-  Both methods must be used to compare intermediate and final results. The two methods can complement each other and provide realistic estimates on the systematic errors.

- P. Vranas, IBM Watson Research Lab

## BlueGene/L

**:**

- P. Vranas, IBM Watson Research Lab

## Slide 22

- P. Vranas, IBM Watson Research Lab

## A Three Dimensional Torus

- P. Vranas, IBM Watson Research Lab

## BlueGene/L

**BlueGene/L**

- 2.8/5.6 GF/s
- 4 MB

- 2 processors

- 2 chips, 1x2x1

- 5.6/11.2 GF/s
- 1.0 GB

- (32 chips  4x4x2)
- 16 compute, 0-2 IO cards

- 90/180 GF/s
- 16 GB

- 32 Node Cards

- 2.8/5.6 TF/s
- 512 GB

- 64 Racks, 64x32x32

- 180/360 TF/s
- 32 TB

- Cabled 8x8x16

- Rack

- System

- Node Card

- Compute Card

- Chip

- P. Vranas, IBM Watson Research Lab

## Slide 25

- P. Vranas, IBM Watson Research Lab

## Dual Node Compute Card

**9 x 512 Mb DRAM; 16B interface; no external termination**

**Heatsinks designed for 15W**

**206 mm (8.125”) wide,  54mm high (2.125”), 14 layers, single sided, ground referenced **

**Metral 4000 high speed differential connector (180 pins)**

- P. Vranas, IBM Watson Research Lab

## 32- way (4x4x2) node card

**Custom dual voltage, dc-dc converters; I2C control**

**IO Gb Ethernet connectors  through tailstock**

**Latching and retention**

**Midplane (450 pins) torus, tree, barrier, clock, Ethernet service port **

**16 compute cards**

**2 optional IO cards **

**Ethernet-JTAG FPGA**

- P. Vranas, IBM Watson Research Lab

## Slide 28

- P. Vranas, IBM Watson Research Lab

## 64 racks at LLNL 

- P. Vranas, IBM Watson Research Lab

## Torus hardwareDynamical virtual cut-through with bubble escape and priority channels

**A hardware implementation of multicasting along a line**

**Deterministic and adaptive routing**

- P. Vranas, IBM Watson Research Lab

## BlueGene next generations

**:**

- P. Vranas, IBM Watson Research Lab

##  P

- P. Vranas, IBM Watson Research Lab

##  Q

- P. Vranas, IBM Watson Research Lab

## BlueGene/L and QCD

**:**

- P. Vranas, IBM Watson Research Lab

## The 1 sustained-Teraflops landmark

**June 2004**

**The 1 sustained-Teraflops landmark**

- P. Vranas, IBM Watson Research Lab

## QCD on the hardware

- Virtual node mode:
-     CPU0, CPU1 act as independent “virtual nodes”
-     Each one does both computations and communications
-     The 4th direction is along the two CPUs
-     The two CPU’s communicate via common memory buffers
-     Highly optimized kernel code is in inline assembly.
-     Extremely careful balance between computations, memory
-          accesses and network communications

**CPU0**

**CPU1**

- P. Vranas, IBM Watson Research Lab

## Wilson kernel node performance (strong scaling)

- Spin-projection and even/odd preconditioning
- Numbers are for single chip with self-wrapped links
- Full inverter (with torus global sum)

| %of peak |     24 |   4x 23 |     44 |   8 x 43 |  82 x 42 | 16 x 43 |
| --- | --- | --- | --- | --- | --- | --- |
| D no comms | 31.5 | 28.2 | 25.9 | 27.1 | 27.1 | 27.8 |
| D | 12.6 | 15.4 | 15.6 | 19.5 | 19.7 | 20.3 |
| Inverter | 13.1 | 15.3 | 15.4 | 18.7 | 18.8 | 19.0 |

- P. Vranas, IBM Watson Research Lab

## Slide 38

- P. Vranas, IBM Watson Research Lab

## Full QCD physics system

- The physics code (besides the Wilson Dslash) is the Columbia C++ physics system (cps).
- The full system ported very easily and worked immediately.
- The BG/L additions/modifications to the system have been kept isolated.

**Acknowledgement**

-    **We would like to thank the QCDOC collaboration for useful discussions and for providing us with the Columbia physics system software.**

- P. Vranas, IBM Watson Research Lab

## BU workshop 1/2006

- http://super.bu.edu/~brower/qcd-bgl

- P. Vranas, IBM Watson Research Lab

## Some interesting machine configurations

**1024**

**512**
**512**

**Nodes:  8 x   8 x 16 x   2**
**Local :   2 x  2  x   2 x   8**

**= ****16 x 16 x 32 x 16**

**Zero temperature dynamical **

**Nodes:   8 x  8 x   8 x  2**
**Local :   2 x   2 x   2 x  4**

**= ****16 x 16 x 16 x  8**

**Thermo – hot start**

**Nodes:   8 x  8 x   8 x  2**
**Local :   2 x   2 x   2 x  4**

**= ****16 x 16 x 16 x  8**

**Thermo – cold start**

- P. Vranas, IBM Watson Research Lab

## QCD and IBM

**:**

- P. Vranas, IBM Watson Research Lab

## Market, engineering and science significance

** ****20+ BG/L rack sales ****enabled by the QCD IBM Research effort.**
** ****QCD exposed BG/L in a very demanding computations, communications and memory access environment. This suggested the main architectural directions for the next generation.**
** ****QCD has served as a hardware diagnostic (fault isolation, run-to-run reproducibility) and performance tool.**
** ****Unprecedented sustained performances have enabled other HPC applications. More to come.**
** ****Showed that QCD can be efficiently embedded in the new multi-core environment of modern microelectronic chips in a fully scalable way.**
** ****First ever to cross the 1 Teraflops sustained speed landmark for QCD marking the next generation of lattice QCD simulations.**
** ****Simulations the QCD phase transition between stable nuclear matter and the quark gluon plasma with cutting edge scientific algorithms and methods.**
** ****Provides theoretical physics with an unprecedented **
**   ****tool for explorations ranging from QCD to **
**   ****Supersymmetric theories and to the strong interaction **
**   ****regime of string theory.**

- P. Vranas, IBM Watson Research Lab

## QCD thermodynamics with DWF as of 2004

**:**

- P. Vranas, IBM Watson Research Lab

- Iwasaki improved gauge action:

**P. Vranas, NPB Prc 83 (2000) 414**

-  It works

- P. Vranas, IBM Watson Research Lab

-  It is very sensitive: anomalous symmetry breaking above T**c**

**P. Vranas, NPB Prc 83 (2000) 414**

- P. Vranas, IBM Watson Research Lab

## The bleeding edge

**:**

- P. Vranas, IBM Watson Research Lab

## The proposed parameters

- Finite temperature lattices: V=16x16x16, Nt=8.
- Zero temperature lattices:  V=16x16x16, Nt=32.
- DWF, Ls=32.
- Iwasaki gauge action with c1 = -0.331.

- P. Vranas, IBM Watson Research Lab

## How long will it take

**?**

- P. Vranas, IBM Watson Research Lab

## Estimating Tc calculation to a few %: ...

- 163x8

- 1⁄2 rack (512 nodes) at 10% of peak

- # of  points	  10
- # of config	300
- hot/cold		    2
- (2+1)		    2

- 	         12,000
- X 		  45 min

- 9,000 hours

- 163x32

- # of propagators	   50
- # of traj. per prop	   20
- (2+1)		    2

- 	          2,000
- X 		    3 hr

- 6,000 hours

-                    15,000 hours on 1⁄2 rack = 312 rack days
- + 2 points on 32x16x16x8 (finite vol) =   33 rack days
-                                                               350 rack days

**1 rack year**

- P. Vranas, IBM Watson Research Lab

## Estimating pressure calculation to about 10%: ...

- 163x8

- 1⁄2 rack (512 nodes) at 10% of peak

- # of  points	    15
- # of config	2,500

- 	            37,500
- X 		     45 min

- 28,125 hours

- 163x16

- 	              15,000
- X 		       1.5 hr

- 22,500 hours

-                    50,625 hours on 1⁄2 rack = 1054 rack days

**2.9 rack years**

- # of  points	    15
- # of config	1,000

- P. Vranas, IBM Watson Research Lab

## Glue

**:**

- P. Vranas, IBM Watson Research Lab

## This time over (2005+) 

-  The lattice and the big supercomputers are here and stronger than ever
-   ... for heavy ion collisions
-   ... for nuclear physics
-   ... for untangling the QCD interactions from the LHC data
-   ... for the physics beyond the standard model
-   ... maybe there are more  little perfect and isolated worlds such as QCD
-   ... maybe the extra dimensions of DWF are really there
-   ... for the things we have not thought of yet...

- P. Vranas, IBM Watson Research Lab

## So

- ?

- P. Vranas, IBM Watson Research Lab

## It can be done.
The time is now.
The place is here.

**The time is now.**
**The place is here.**

- P. Vranas, IBM Watson Research Lab

## What would you do

**?**

- P. Vranas, IBM Watson Research Lab

## ... if they come to you with 1 Petaflop for a year ? 

- Think fast.

- P. Vranas, IBM Watson Research Lab

## QCD, the movie

**:**

- P. Vranas, IBM Watson Research Lab

##  QCD thermal phase transition  a clip from a BG/L lattice simulation. 

**  ****This clip is from a simulation of QCD on a 1⁄2 a rack of a BG/L machine (2.8 Teraflops). It took about about 2 days.**
**  ****It shows 2-flavor dynamical QCD on a 16x16x16x4 lattice with the DWF 5th dimension set to 24 sites.**
**  ****The pion mass is about 400 MeV.**
**  ****The color of each lattice point is the value of the Polyakov loop which can fluctuate between -3 and 3. Think of it as a spin system.**
**  ****The graph shows the volume average of the Polyakov line. This value is directly related to the single quark free energy. In the confined phase there are no free quarks and the value is low (not zero because of screening), in the quark-gluon plasma phase quarks can exist alone and the value is large.**

**G. Bhanot, D. Chen, A. Gara, P. Heidelberger, J. Sexton, P. Vranas, B. Walkup**** **

- P. Vranas, IBM Watson Research Lab