# Presentation: 437679

## Future Directions for U.S. LGT Software & Algorithm  

- Richard  C. Brower
- BNL
- Nov 30, 2005

- Code distribution see  http://www.usqcd.org/usqcd-software

**Notes:**
MAKE SLIDES MIRROR WRITE UP!

## K. Wilson (1989 Capri): “One lesson is that lattice gauge theory could also require a 108 increase in computer power AND spectacular algorithmic advances before useful interactions with experiment ...”

- ab initio Chemistry
- 1930+50 = 1980
- 0.1 flops  10 Mflops
- Gaussian Basis functions

- ab initio QCD
- 1980 + 50  = 2030?*
- 10 Mflops  1000 Tflops
- Clever Collective Variable?

- vs

****Much sooner but need **less than** $10/Gflops!*

## Software Golden Rule: must advance Physics Goals

- _Increased_ Performance
- 2.    _Reduced_ Labor to Port to new Architectures
- 3.    _Stimulate_ discovery of new  Algorithms & Apps

*These benefits have just begun to be realized in 1**st** 5 years!*

## OUTLINE

- CPS/MILC/Chroma application codes are doing physics on QCDOC @ BNL & Clusters @ FNAL and JLab.
-  SciDAC-2.0 New directions
- Foundations: Multi-core QMP/QLA/QDP et al
- Level 4  QLIB & Data Analysis Libraries
- Uniform Environments
- Fault Tolerance for Petascale  systems
-  Multi-scale Algorithms for the Petaflops Era
-  Some Fun speculations

## SciDAC -1.0 Participants  in Software  Project 

- SciDAC -1.0 Participants  in Software  Project

- * _Software Coordinating Committee_

- UK              Peter Boyle
-   (JLab)        Balint Joo

## Lattice QCD – API Design Concept

- (Perfect) Load Balancing:  *Uniform periodic lattices &  identical sublattices per processor.*
- (Complete) Latency hiding:* overlap computation            /communications*
- Data Parallel: *operations on Small 3x3 Complex Matrices per link.*
- Critical kernel :  *Dirac  Solver is ~90%  of Flops.*

- _Lattice Dirac operator:_

## SciDAC 1.0 Foundations Libraries

*Level 1 Code (in assembly if necessary)*
*QMP (Message Passing) version 2.0*
*   **GigE,  in native QCDOC  & over MPI. *
*QLA (Linear Algebra)*
* **C routines (24,000 generated by Perl with test code).*
* **Optimized (subset) for P4 Clusters in **SSE **assembly.*
* **QCDOC optimized using **BAGEL **-- assembly generator for  PowerPC, IBM power, UltraSparc, Alpha... by P. Boyle UK.*

## SciDAC-1.0 QCD API 

- QDP      (QCD Data Parallel)
- Lattice Wide Operations, Data shifts,
- Hides message passing and layout

- _Level 2_

- QMP (QCD Message Passing)

- QLA        (QCD Linear Algebra)

- _Level 1_

-      QIO
- Binary DataFiles / XML Metadata

- SciDAC-1.0 QCD API

## SciDAC-1.0 Programming Model 

*Level 2: QDP (Data Parallel) API *
*C version for MILC applications code*
*C++ version for new code base Chroma*
*QIO  File management: *
*  **Read/Write  for XML/binary files compliant with ILDG*
*Level 3 QOP interface** *
*   **Dirac inverters for  QCDOC and P4 clusters*

## Example of QDP++ Expression

**Typical for Dirac Operator:**

**QDP/C++ code:**

- Portable Expression Template Engine (PETE)
- Temporaries eliminated, expressions optimized

## Level 3 Performance on QCDOCy 

- y  Warning:  O(40%) is for inverters at optimal sub-volumes.

## Case History: SciDAC improvements to MILC Asqtad  

- Lattice  403 x 96
- (2K rack sub-volume 5x102x6   Level 3 CG =33%)
-                                                                             incremental            cumulative        1. CG inverter MILC to Edram MILC                   1.5                       1.5        2. CG inverter Edram MILC to QDP/C                1.2                       1.8        3. CG inverter QDP/C to Level 3                         1.2                       2.1        4. C code FF force improvements                       1.2                       2.4        5. Hoped for improvementsy                                1.4                       3.5
- Lattice   483 x 144
- (4K rack subvolume63x18  Level 3 CG = 35%)
-                                                                                incremental           cumulative        1. CG inverter MILC to Edram MILC                    1.4                       1.4        2. CG inverter Edram MILC to QDP/C                 1.2                       1.7        3. CG inverter QDP/C to Level 3                          1.1                       1.8        4. C code FF force improvements                        1.2                       2.0        5. Hoped for improvementsy                                 1.5                       3.0

- y Note because of Chulwoo’s fast level 3 inverter, now 70% of the time is in other routines.

##  Asqtad on Infiniband Cluster 

## Level 3 Domain Wall  CG Inverter y 

- Level 3 Domain Wall  CG Inverter y

-  y Ls = 16, 4g is P4 2.8MHz, 800MHz FSB

- (6)102 40

- 6 163

- (32 nodes)

- 42 6 16

- (4) 52 20

## Domain decomposition  & multi-time step    

-  Machine specific software
-  Threaded version of the software for multi-core chips
-  New versions of QMP (BlueGene/L, Infiniband)
-  Additional level-3 routines
-  Support for commercial machines (when available to USQCD)
-  Software to monitor & control for QCDOC and large clusters.
-                           (with Vanderbilt and Illinois Institute of Technology)
-      Support for physics application codes
-   Documentation & Support for existing QCDAPI
-   Optimization of low level libraries QLA/QDP
-   Software for database-driven physics analysis (like root?)
-   Level 4 LFT Library
-   Performance analysis (collaboration with Reed’s group)
-  Uniform computing environment
-  Common runtime environment
-  Data management (collaboration with ILDG)
-  Accounting tools
-  Grid tools (PPDG & ILDG Middleware group)
-  Multi-scale algorithms (collaboration TOPS ASICS)
- Multi-grid
- Domain decomposition  & multi-time step
- Integration of new algorithms into API and Level 4

## SciDAC-2.0 QCD API 

- QDP      (QCD Data Parallel)
- Lattice Wide Operations, Data shifts,
- Hides message passing and layout

- _Level 2_

- QMP (QCD Message Passing)

- QLA        (QCD Linear Algebra)

- _Level 1_

- SciDAC-2.0 QCD API

-      QIO
- Binary DataFiles / XML Metadata

-      Qdat
- Data fitting, work flow, graphics,..
- (like root?)

## Algorithms for the Year 2015

- End of H. Clinton’s 2nd term
- Cost is $1 per Gigaflops
- Lattices sizes are up to 128x 643 & at “physical quark mass”   [ scaling:Time » (1/m6)(1/a7) ]
-  All Algorithms are Multi-scale!
- The non-SUSY limit is still a challenge for die hard QCD/String theorists.

## 1. Length/Mass Scales in QCD

-  Quarks Masses: (197 fm Mev)
-  2, 8, 100, 1200, 4200, 175,000 Mev
- String Length:
-   1000 Mev  ( » .2 fm)
- Chiral  limit:  m =140 Mev ( »1.4 fm)
- Nuclear: scattering length/effective range
-   asinglet = - 23.714 fm  ( » 8 Mev)    &    r = 2.73
-   atriplet  =     5.425 fm  ( » 36 Mev)   &   r = 1.749 fm
- Deuteron Binding =  50 Mev. (» 4 fm)
- Finite T, finite  etc

*Log(m**q**)*

*Flavor: u,d,s,c,b,t*

##  Confinement length vs Pion Compton length

*l***

*l***

## Moving the Berlin Wall for Wilson Fermions (red)2 scales multiple time step: Hasenbusch (blue) or Schwarz

* **Time for 1000 configurations: **»** 1/m**4****   equals Schwarz algorithm*

*(Urbach, Jansen, Shindler, Wegner, hep-lat/0506011)*

## Q: How to take a Trace? A: Pseudo Fermion Monte Carlo 

*Can  do “standard” Monte Carlo with low eigenvalue  subtraction on H = ****5** D*
*Or “perfect” Monte Carlo – Gaussian ****x*

## Problem is Large Variance

_*Gaussian Noise:*_

_*Z*__*2*__* Noise:*_

## Multi-grid Trace Projecty

- Many ways to speed up the inverse (1/N1/2) and reduce the variance (2)
- MG Speed up Inverse
- Amortize Pre-conditioner with multiple RHS.
- MG variance reduction at long distances.
- Low eigenvalue projection at long distances
- Unbiased subtraction at short distance.
- Dilution at short distances

-  y Brower, Clark, Osborn, Rebbi in collaboration with Keyes (TOPS)and  Brannick(LLNL)

## New  Algebraic Multi-grid

*New non-trivial smooth aggregation adaptive algebraic Multi-grid !*
- Idea is to try naïve iteration
-  Start with x = random vector & iterate A x = 0
-  Get  global “slow mode” in near null space.
- Use cut this vector into blocks (aggregates)
- Smooth to construct Projector. Precondition with this and see if it is still slow.
- If so find another near null  in orthogonal space if not  go to the next level.
- Use Multi-grid at every stage to adaptively find the next better MG operator!

- James Brannick(LLNL)

## Slide 25

## Test on 1 flavor 2-d Schwinger Model with  Wilson Fermions

*Test on 1 flavor 2-d Schwinger Model with  Wilson Fermions*

## Numerical results for 2-d Schwinger Model with Wilson Fermions 

- Numerical results for 2-d Schwinger Model with Wilson Fermions

## Dynamical Gluey in 5-d ?  
( Improved isolation of Left and Right  domain walls by “localization”?) 

SUSY broken only by DW  boundaries ?

 Hadronization in  5-d ? AdS5/CFT works pretty well. Why? 

- ( Improved isolation of Left and Right  domain walls by “localization”?)
- SUSY broken only by DW  boundaries ?
-  Hadronization in  5-d ? AdS5/CFT works pretty well. Why?

- Taking the 5th Dimension Seriously?

- y D-Theory: Field Quantization by Dimensional Reduction of Discrete Variables  hep-lat/0309182  (R.Brower, S.Chandrasekharan, S.Riederer, U.-J.Wiese  Here Ls  1 is the continuum limit.

## Note Kaplan’s Fermions have5-d Vector Current  4-d Vector/Axial Current 

*Vector:*

*Axial:*

## Confinement takes place in 5-d

|  |  |  |  |  |
| --- | --- | --- | --- | --- |
|  |  |  |  |  |
|  |  |  |  |  |
|  |  |  |  |  |

- s = 1

- s = 2

- s = M

- s = L_s

**q****L**

**q****R**

**Q****L**

**Q****R**

**q****L**

**q****R**

**Q****R**

**Q****L**

- LEFT

- RIGHT

## “QCD and a Holographic Model of Hadrons”  Erlich, Katz, Son, Stephanov, hep-ph/05011

*The 5-d flavor Vector  Field A has even/odd  in s pieces for vector/axial hadrons,*
*with Goldstone modes at the boundaries! (Brower, Guralnik and Tan)*

## FINI