# Presentation: 698193

## The Distributed Data Interface in GAMESS

- Brett M. Bode, Michael W. Schmidt,
- Graham D. Fletcher, and Mark S. Gordon
- Ames Laboratory-USDOE,
- Iowa State University

- 10/7/99

## What is GAMESS?

- _G_eneral _A_tomic and _M_olecular _E_lectronic _S_tructure _S_ystem

- First principles - fully quantum mechanical
- Created from other programs in ~1980
- Developed by Dr. Mark Gordon’s research group since 1982 with Dr. Michael Schmidt as the principle developer.
- Parallelization begin in 1991
- Emphasis on Distributed memory systems
- Currently includes methods for treating        1-atom to several hundred atoms

## Partial list of capabilities

- C = Uses disk storage
- D = Minimal disk usage
- P = Parallel execution

## First Generation ParallelCode

- Parallel communications were performed using either:
- TCGMSG
- Vendor supplied MPI
- Parallel version was usually a slightly modified version of the sequential code

## IBM-SUR cluster

- 22 IBM RS/6000 43P-260:
- Dual 200MHz Power3 CPUs
- 4 Mb of Level 2 cache
- 1 GByte of RAM
- 18 GBytes fast local disks
- Jumbo Frames Gig Ethernet
- Integrated Fast-Ethernet
- Fast Ethernet Switch to all
- 3x9 port Gigabit Switches

## Gigabit Performance on theIBM 43P-260 Cluster

## Test Molecule

- Ti(C5H5)2 C2H4SiHCl3
- Basis Set
- 6-31G(d,p) on C and H.
- SBKJC ECP on Si, Ti, and Cl extended with 1 d-type polarization function on Si and Cl.
- 345 total basis functions

## Parallel SCF

- Very good scaling dependant on the size of the molecule.
- Large systems show nearly linear scaling through 256 nodes

## Successes and Limitations

- SCF methods scale very well
- Most methods run in parallel
- Good use is made of aggregate CPU and disk resources.

- MP2 and MCSCF methods scale to only a few (8-32) nodes
- The aggregate memory is not utilized so jobs are still limited by the memory size of one node.

## Second Generation Methods

- New methods should take advantage of the aggregate memory of a parallel system
- Implies a higher communication demands
- Many to many messaging profile
- Methods should scale to hundreds of nodes (at least)
- Demanding local storage needs

## The Distributed Data Interface (DDI)

- DDI provides the core functions needed to treat a portion of the memory on each node as part of a global shared array.

## DDI

- Runs on top of:
- MPI (MPI-2 preferred)
- TCP/IP sockets
- Lightweight - Provides only the functionality needed by GAMESS

- Is not intended as a general purpose library.
- Does optimize for mixed SMP and distributed memory systems

## New MP2 implementation

- Uses DDI to utilize the aggregate memory of the parallel machine at the expense of communications
- Trades some symmetry in the MP2 equations for better parallel scalability
- Requires more memory than the sequential version
- Is slower than the sequential version on 1 CPU

## MP2 Scalability

## Conclusions

- DDI provides a scalable way of taking advantage of the global memory of a parallel system
- The new MP2 code demonstrates code written specifically for parallel execution without replacing the sequential version.

## Future Work

- DDI needs further work to enhance the features and increase robustness, or possibly needs to be replaced with a more general library such as the GA tools from PNNL.
- The global shared memory approach is being applied to many other parts of GAMESS to increase scalability.

## Thanks!

- David Halstead
- Guy Helmer
- For $:
- IBM Corp. for an SUR grant (of 15 Workstations)
- DOE MICS program (interconnects and 7 workstations)
- Air Force OSR (long term dev. Funding)
- DOD CHSSI program (improved parallelization)