# Presentation: 876500

## Lattice QCD and the SciDAC-2 LQCD Computing Project

- Lattice QCD Workflow Workshop
- Fermilab, December 18, 2006
- Don Holmgren, djholm@fnal.gov

## Outline

- Lattice QCD Computing
- Introduction
- Characteristics
- Machines
- Job Types and Requirements
- SciDAC-2 LQCD Computing Project
- What and Who
- Subprojects
- Workflow

## What is QCD?

- Quantum ChromoDynamics is the theory of the strong force
- the strong force describes the binding of quarks by gluons to make particles such as neutrons and protons
- The strong force is one of the four fundamental forces in the Standard Model of Physics– the others are:
- Gravity
- Electromagnetism
- The Weak force

## What is Lattice QCD?

- Lattice QCD is the numerical simulation of QCD
- The QCD action, which expresses the strong interaction between quarks mediated by gluons:where the Dirac operator (“*dslash*”) is given by
- Lattice QCD uses discretized space and time
- A very simple discretized form of the Dirac operator iswhere *a* is the lattice spacing

## A quark, (x), depends upon (x + a) and the local gluon fields U  
(x) is complex 3x1 vector, and the U  are complex 3x3 matrices.  Interactions are computed via matrix algebra
On a supercomputer, the space-time lattice is distributed across all of the nodes

***(x)** *is complex 3x1 vector, and the *U****  *are complex 3x3 matrices.  Interactions are computed via matrix algebra
- On a supercomputer, the space-time lattice is distributed across all of the nodes

## Computing Constraints

- Lattice QCD codes require:
- Excellent single and double precision floating point performance
- Majority of Flops are consumed by small complex matrix-vector multiplies [SU(3) algebra]
- High memory bandwidth (principal bottleneck)
- Low latency, high bandwidth communications
- Typically implemented with MPI or similar message passing APIs
- On clusters: Infiniband, Myrinet, gigE mesh

## Computing Constraints

- The dominant computation is the repeated inversion of the Dirac operator
- Equivalent to inverting large 4-D and 5-D sparse matrices
- Conjugate gradient method is used
- The current generation of calculations requires on the order of Tflop/s-yrs to produce the intermediate results (“vacuum gauge configurations”) that are used for further analysis
- ~ 50% of Flops are spent on configuration generation, and ~ 50% on analysis using those configurations

## Near Term Requirements

- Lattice QCD codes typically sustain 30% of the Tflop/s reported by the Top500 Linpack benchmark, typically 20% of peak performance.
- Planned configuration generation campaigns in the next few years:

## Lattice QCD Codes

- You may have heard of the following codes:
- MILC
- Written by the MIMD Lattice Computation Collaboration
- C-based, runs on essentially every machine (MPI, shmem,...)
- _[http://www.physics.indiana.edu/~sg/milc.html](http://www.physics.indiana.edu/~sg/milc.html)_
- Chroma
- _[http://www.usqcd.org/usqcd-docs/chroma/](http://www.usqcd.org/usqcd-docs/chroma/)_
- C++, runs on any MPI (or QMP) machine
- CPS (Columbia Physics System)
- C++, written for for the QCDSP but ported to MPI (or QMP) machines
- _http://phys.columbia.edu/~cqft/physics_sfw/physics_sfw.htm_

## LQCD Machines

- Dedicated Machines (USA):
- QCDOC (“QCD” On a Chip) – Brookhaven
- GigE and Infiniband Clusters – JLab
- Myrinet and Infiniband Clusters – Fermilab
- Shared Facilities (USA):
- Cray XT3 – PSC and ORNL
- BG/L – UCSD, MIT, BU
- Clusters – NCSA, UCSD, PSC, ...

## Fermilab LQCD Clusters

| Cluster | Processor | Nodes | MILC
performance |
| --- | --- | --- | --- |
| qcd | 2.8 GHz P4E,
Intel E7210 chipset,
1 GB main memory,
Myrinet | 127 | 1017 MFlops/node

0.1 TFlops |
| pion | 3.2 GHz Pentium 640,
Intel E7221 chipset,
1 GB main memory,
Infiniband SDR | 518 | 1594 MFlops/node

0.8 TFlops |
| kaon | 2.0 GHz Dual Opteron,
nVidia CK804 chipset,
4 GB main memory,
Infiniband DDR | 600 | 3832 MFlops/node

2.2 TFlops |

## Job Types and Requirements

- Vacuum Gauge Configuration Generation
- Simulations of the QCD vacuum
- Creates *ensembles* of *gauge* *configurations*; each *ensemble* is characterized by lattice spacing, quark masses, and other physics parameters
*Ensembles* consist of simulation time steps drawn from a sequence (Markov chain) of calculations
- Calculations require a large machine (capability computing) delivering O(Tflop/sec) to a single MPI job
- An *ensemble* of configurations typically has O(1000) time steps
*Ensembles *are used in multiple *analysis calculations *and are shared with multiple physics groups worldwide

## Sample Configuration Generation Stream

- Currently running at Fermilab:
- 48^3 x 144 configuration generation (MILC asqtad)
- Job characteristics:
- MPI job uses 1024 processes (256 dual-core dual Opteron nodes)
- Each time step requires ~ 3.5 hours of computation
- Configurations are 4.8 Gbytes
- Output of each job (1 time step) is input to next job
- Every 5th time step goes to archival storage, to be used for subsequent physics analysis jobs
- Very low I/O requirements (9.6 Gbytes every 3.5 hours)

## Job Types and Requirements

- Analysis computing
*Gauge configurations* are used to generate *valence quark propagators*
- Multiple *propagators* are calculated from each *configuration* using several different physics codes
- Each *propagator generation *job is independent of the others
- Unlike *configuration generation* can use many simultaneous job streams
- Jobs require 16-128 nodes, typically 4-12 hours in length
*Propagators* are larger than *configurations* (factor of 3 or greater)
- Moderate I/O requirements (10’s of Gbytes per few hours); lots of non-archival storage required (10’s of Tbytes)

## Job Types and Requirements

- “Tie Ups”
*Two point* and *three point* *correlation* calculations using *propagators* generated from *configurations*
- Typically small jobs (4-16 nodes)
- Heavy I/O requirement - 10’s of Gbytes for jobs lasting O(hour)

## SciDAC-2 Computing Project

- Scientific Discovery through Advanced Computing
- _[http://www.scidac.gov/physics/quarks.html](http://www.scidac.gov/physics/quarks.html)_
- Five year project, sponsored by DOE Offices of High Energy Physics, Nuclear Physics, and Advanced Scientific Computing Research
- $2.2M/year
- Renewal of previous 5 year project:http://www.scidac.gov/HENP/HENP_QCD.html

## Slide 17

## SciDAC-2 LQCD Participants

**Principal Investigator:** Robert Sugar, UCSB
**Participating Institutions and Co-Investigators:**Boston University - Richard Brower and Claudio RebbiBrookhaven National Laboratory - Michael CreutzDePaul University - Massimo DiPierroFermi National Accelerator Laboratory - Paul MackenzieIllinois Institute of Technology - Xian-He SunIndiana University - Steven GottliebMassachusetts Institute of Technology - John NegeleThomas Jefferson National Accelerator Facility - David Richards       and William (Chip) WatsonUniversity of Arizona - Doug ToussaintUniversity of California, Santa Barbara - Robert Sugar (PI)University of North Carolina - Daniel ReedUniversity of Utah - Carleton DeTarVanderbilt University - Theodore Bapty

## SciDAC-2 LQCD Subprojects

- Machine-Specific Software
- Optimizations for multi-core processors
- Native implementations of message passing library (QMP) for Infiniband and BlueGene/L
- Opteron linear algebra optimizations (XT3, clusters)
- Intel SSE3 optimizations
- Optimizations for BG/L, QCDOC, new architectures
- “Level-3” codes (highly optimized physics kernels)

## SciDAC-2 LQCD Subprojects

- Infrastructure for Application Code
- Integration and optimization of QCD API
- Documentation and regression testing
- User support (training, workshops)
- QCD Physics Toolbox
- Shared algorithms and building blocks
- Graphics and visualization
***Workflow***
- Performance analysis
- Multigrid algorithms

## Slide 21

## SciDAC-2 LQCD Subprojects

- Uniform Computing Environment
- Common runtime environment
- Data management
- Support for GRID and ILDG (International Lattice Data GRID)
***Reliability – monitoring and control of large systems***
- Accounting tools