# Presentation: 674402

## Parallel Multi-Reference Configuration Interaction on JAZZ

- Ron Shepard (CHM)Mike Minkoff (MCS)Mike Dvorak (MCS)

## The COLUMBUS Program System

- Molecular Electronic Structure
- Collection of individual programs that communicate through external files
- 1: Atomic-Orbital Integral Generation2: Orbital Optimization (MCSCF, SCF)3: Integral Transformation4: MR-SDCI5: CI Density6: Properties (energy gradient, geometry    optimization)

## Real Symmetric Eigenvalue Problem

- Use the iterative Davidson Method for the lowest (or lowest few) eigenpairs
-  *Direct CI*: **H** is not explicitly constructed, **w**=**Hv** are constructed in “operator” form
- Matrix dimensions are 104 to 109
- All floating point calculations are 64-bit

## Davidson Method

- Generate an initial vector **x**1
- MAINLOOP: DO *n*=1, NITER
-     Compute and save **w***n* = **H** **x***n*
-     Compute the *n**th* row and column of **G **= **X***T***HX **= **W***T***X**
-     Compute the subspace Ritz pair:  (**G** – ****1**) **c** = 0
-     Compute the residual vector **r** = **W c** – ** **X c**
**    **Check for convergence using |**r**|, **c**, **, etc.
-     IF (converged) THEN
-         EXIT MAINLOOP
-     ELSE
-         Generate a new expansion vector **x***n*+1 from **r**, **, **v**=**Xc**, etc.
-     ENDIF
- ENDDO MAINLOOP

## Matrix Elements

*H**mn* = <*m*| *H**op* |*n*>
- |*n*> = |**(**r**1) **1 **(**r**2)**2 ... **(**r**n)**n |      with ***j**=***, **
* *
* *

## ...Matrix Elements

*h**pq* and *g**pqrs* are computed and stored as arrays (with index symmetry)
- <*m*|*E**pq*|*n*> and <*m*|*e**pqrs*|*n*> are *coupling coefficients*; these are sparse and are recomputed as needed

## Matrix-Vector Products

- The challenge is to bring together the different factors in order to compute **w** efficiently

- • **w** = **H x**

## Coupling Coefficient Evaluation

- Graphical Unitary Group Approach (GUGA)
- Define a directed graph with nodes and arcs: *Shavitt Graph*
- Nodes correspond to spin-coupled states consisting of a subset of the total number of orbitals
- Arcs correspond to the (up to) four allowed spin couplings when an orbital is added to the graph

## ...Coupling Coefficient Evaluation

- w,x,y,z

-  graph head

- graph tail

- Internal orbitals

- External orbitals

## ...Coupling Coefficient Evaluation

## Integral Types

-  0: *g**pqrs*
-  1: *g**pqra*
-  2: *g**pqab*,     *g**pa,qb*
-  3: *g**pabc*
-  4: *g**abcd*

## Original Program (1980)

- Need to optimize wave functions for *N**csf*=105 to 106
- Available memory was typically 105 words
- Must segment the vectors, **v** and **w**, and partition the matrix **H** into subblocks, then work with one subblock at a time.

## ...First Parallel Program (1990)

- Networked workstations using TCGMSG
- Each matrix subblock corresponds to a compute task
- Different tasks require different resources (pay attention to load balancing)
- Same vector segmentation for all *g**pqrs* types
*g**pqrs*, <*m*| *e**pqrs* |*n*>, **w**, and **v** were stored on external shared files (file contention bottlenecks)

## Current Parallel Program

- Eliminate shared file I/O by distributing data across the nodes with the GA Library
- Parallel efficiency depends on the vector segmentation and corresponding **H** subblocking
- Apply different vector segmentation for different *g**pqrs* types
- Tasks are timed each Davidson iteration, then sorted into decreasing order and reassigned for the next iteration in order to optimize load balancing
- Manual tuning of the segmentation is required for optimal performance
- Capable of optimizing expansions up to *N**csf*=109

## COLUMBUS-PetaflopsApplication

- Mike Dvorak, Mike Minkoff
- MCS Division
- Ron Shepard
- Chemistry Division
- Argonne National Lab

## Notes on software engineering

- PCIUDG parallel code
- Fortran 77/90
- Compiled with Intel/Myrinet on Jazz
- 70k lines in PCIUDG
- 14 files containing ~205 subroutines
- Versioning system
- Currently distributed in a tar file
- Created a LCRC CVS repository for personal code mods

## Notes on Software Engineering (cont)

- Homegrown preprocessing system
- Uses “*mdc*if  parallel” statements to comment/uncomment parts of the code
- Could/should be replaced with CPP directives
- Global Arrays library
- Provides global address space for matrix computation
- Used mainly for chemistry codes but applicable for other applications
- Ran with most current version --> no perf gain
- Installed on Softenv on Jazz (version 3.2.6)

## Gprof Output

- 270 subroutines called
***loopcalc***** subroutine using ~20% of simulation time**
**Added user defined MPE states to 50 loopcalc calls**
**Challenge due to large number of subroutines in file**
**2 GB file size severe limiter on number of procs**
**Broken logging**
**Show actual output**

## Jumpshot/MPE Instrumentation

- Live Demo of a 20 proc run

## Using FPMPI

- Relinked code with FPMPI
- Tell you total number of MPE calls made
- Output file size smalled (compared to other tools i.e. Jumpshot)
- Produces a histogram of message sizes
- Not installed in Softenv on Jazz yet
- ~riley/fpmpi-2.0
- Problem for runs
- Double Zeta C2H4 without optimizing the load balance

## Total Number of MPI calls

## Max/Avg MPI Complete Time

## Avg/Max Time MPI Barrier

## COLUMBUS Performance Results

## COLUMBUS Performance Data

- R. Shepard, M. Dvorak, M. Minkoff

## Timing of Steps (Sec.)

| Time
Basis Set | Integral Time | Orbital 
Opt. Time | CI Time |
| --- | --- | --- | --- |
| QZ | 388 | 11806 |  382,221 |
| TZ |  26 |     104 |    31,415 |
| DZ |    1 |       34 |      3,281 |

## Walks Vs. Basis Set (Millions)

| Walk Type 

Basis Set | Z | Y | X | W | Matrix
Dim. |
| --- | --- | --- | --- | --- | --- |
| cc-pVQZ | .08 | 15 | 536 | 305 | 858 |
| cc-pVTZ | .08 |  7 | 120 |  69 | 198 |
| cc-pVDZ | .08 |  2 |  13 |   8 | 24 |

## Timing of CI Iteration

## Basic Model of PerformanceTime = C1+C2*N+C3/N

## Constrained Linear TermC2 > 0