Acceleration of computational quantum chemistry by AVX + CUDA on
heterogeneous computer architectures
Yuki Furukawa, Ryota Koga : X-Ability Co.,Ltd.
Koji Yasuda : Nagoya University
BackgroundResults & Discussion

1. Biologically important macromolecules, such as proteins, arewithin the scope of electronic-structure calculation by using FMO
because of the improvement of computational resources.
2. AVX is effective on Intel SandyBridge Architecture. This is a firststep for the era of MIC(Many Integrated Cores) computation.
3. GPGPU computation by CUDA is successfully applied for manykinds of problems in high performance computation.

Purpose of this Research

Benchmark Environment :(1) Intel Core i5 2500 @ 3.30GHz(4core), 6GB, CUDA 3.2,(2) Intel Core i7-3930K @3.2GHz (6 core) , GTX580 x 2, 32GB, CUDA 4.0(3) Intel Core i7-3930K @3.2GHz (6 core), GTX580 x 4, 32GB, CUDA 4.0
using Intel Compiler XE 12.0 + MKL 10.3

Table 1 : Ab-initio calculation time of Valinomycine byGAMESS 2010 R3 (RHF/3-21G, mainly Direct SCF) (1)SoftwareOriginal

Based on Yasuda [2008], We develop a software library (XAAVX/CUDA-QM) to accelerate famous QM softwares (GAMESS-USetc...) using both NVIDIA GPU and Intel SandyBridge CPU. We
show the excellent performance of XA-AVX/CUDA-QM together withFMO. This implemetation is a preliminary preparation for the era ofMIC/GPGPU computation.

Time(sec) Energy[a.u.]
476.24-3750.9205018139

Original + XA-AVX-QM

1. Total energy and the gradient of under FMO approximation.
2. Most time-consuming steps (the evaluation of Coulomb, HartreeFock exchange, and exchange-correlation matrices…) are
executed on GPUs. Remaining parts are calculated on CPUs.
3. Use single precision arithmetic as much as possible for the sakeof effective speedup, without degradation of accuracy.
4. Take special care for task scheduling and host-GPU
communication time to keep as many CPUs and GPUs busy.
5. Modular design applicable to many quantum chemistry softwares

Table 2-1 : FMO2 monomer energies ofeach residue of Insulin(2HIU) by GAMESS2010 R3 (FMO-RHF/6-31G, a.u.) (2) + (3)ΔE(AVX+CUDA)

1(GLY)-442.916776617

2(VAL)-308.573606501

1. The value of dipole moment isexactly the same.
0.000000005
0.000000009 2. Total Monomer Energy3(GLU)-457.046042916

0.000000011

E(Original) : -21635.4488652520

4(GLN)-437.255799945

0.000000002

E(AVX+CUDA) : -21635.4488649211

5(CYS)-1254.889732125

0.000000004

ΔE

6(CYS)
7(THR)
8(SER)
9(ILE)-1254.930631919-344.327645726-305.393476307-347.549172133

0.000000003
3. Calculation Time (sec) :0.000000007
Original:
3279.944
0.000000004
GPGPU : 790.527
0.000000010

FMO-ESP(CUDA : XA-CUDA-QM)
FMO-ERI(AVX : XA-AVX-QM)

The two-electron integrals and the J-matrix in terms of primitive
Hermite Gaussian basis are evaluated by GPU (Direct J Engine).
pJ ab   (ab | cd ) Dcd   Eab  ( p | q ) Dq
cdpq

Schwarz upper bounds are used to reduce the number of ERIs to becalculated, but we don’t use integral symmetry ( e.g. (p|q)=(q|p) ).
The McMurchie-Davidson algorism is used to evaluate Hermite ERIs.
The communication time between CPU and GPU to send / receivep, q, Dq , J p was only about 10 % of GPU computation time.
 log | ( p | q) Dq |The upper bound of each ERI is
first evaluated and ERIs which aresmall in magnitude are calculated
on GPU with single-precision. The
rests (10% of total count of ERIs)
are calculated on GPU with double
precision.
Fig 1. Valinomycin(x 108 , converged D)

Environmental Electrostatic
Potential (ESP):
J-matrix algorithm is reused.
This is the mainly timeconsuming step of FMO.

: 3.31×10-7 a.u.

Table 2-2 : Calculation time of FMO2 environmental
electrostatic potential of Insulin(2HIU) (sec) (2) + (3)Algorithms
Electrostatic potential (Coulomb or J-matrix) :-3750.9205017267

We achieved 3.18x acceleration of Valinomycine calculation
using AVX PRISM algorithm.

Residue E(CPU)

Coding Policy

149.69Time(Original) Time(AVX+CUDA)
2571.490
170.897708.454619.630

We achieved 4.15x acceleration of 44 residue protein calculation
for total time by AVX+CUDA than the original GAMESS, and 15.0x
acceleration for ESP alone. The calculation of ESP consumes
predominant time (over 75%) so that FMO2 accelerates so much.
The direct SCF method is unfavorable for GPGPU because
fragments are so small that we can store all ERIs in a host memory.
AVX was found to accelerate ERI slightly in this case. The totalenergy of FMO and the dipole moments were essentially the same,
which means the accuracy of computation is enough.

Summary
Ab initio calculation by AVX becomes 3.18 time faster than
GAMESS original one by multi-core CPUs. FMO calculation ofenvironmental electrostatic potential by mainly CUDA becomes 15.0
times faster than multi-core CPUs calculation. AVX is found to be
useful to accelerate the host-side ERI evaluation for FMO3 or higher.

Future Themes,sp,p,dGPU(DP)GPU(SP) unnecessary

f or more
CPUunnecessary

PRISM host code was re-written to use AVX as much as possible.
Four shell quartets are packed in an AVX vector and the 1- and 2-electrontransformations in PRISM are fully vectorized. Some parts (evaluation ofBoys function and summation of ERIs to Fock) are not vectorized yet.

1. Acceleration of ERI by AVX+CUDA including f or higher basis formetal protein
2. Parallel evaluation of target using RI-MP2 algorithm by GPGPU
3. Combination with molecular dynamics method
4. AVX vectorization of Boys function and summation of ERIs to Fock
5. Implementation on MIC architecture such as Knights Corner