Quantum Espresso 6.7 with GPU support
Webpagehttps://www.quantum-espresso.org/
https://gitlab.com/QEF/q-e-gpu/-/releases

Version6.7-gpu

Build Environment
PGI 20.4
MKL 2019.0.5 (intel 2019 update 5)
CUDA 10.1

Files Required
q-e-gpu-qe-gpu-6.7.tar.gz
openmpi-4.0.2.tar.bz2(PBS Pro files under /local/apl/lx/pbs14)

Build Procedure#!/bin/sh
VERSION=6.7
FULLVER=${VERSION}
BASEDIR=/home/users/${USER}/Software/QE/${VERSION}-gpu
TARBALL=${BASEDIR}/q-e-gpu-qe-gpu-${FULLVER}.tar.gz
INSTDIR=/local/apl/lx/espresso67-gpu# openmpi
WORKDIR=/work/users/${USER}OMPIVER=4.0.2
OMPITARBALL=/home/users/${USER}/Software/OpenMPI/${OMPIVER}/openmpi-${OMPIVER}.tar.bz2
OMPIROOT=${INSTDIR}/openmpi-${OMPIVER}PBSROOT=/local/apl/lx/pbs14
PARALLEL=12# ----------------------------------------------------------------------umask 0022
module purge
module load pgi/20.4
module load mkl/2019.0.5
export LANG=C
export LC_ALL=C
ulimit -s unlimited# build openmpi first
cd ${WORKDIR}
if [ -d openmpi-${OMPIVER} ]; then
mv openmpi-${OMPIVER} openmpi-erase
rm -rf openmpi-erase &fitar jxf ${OMPITARBALL}
cd openmpi-${OMPIVER}
mkdir rccs && cd rccs

CC=pgcc CXX=pgc++ FC=pgfortran \../configure --prefix=${OMPIROOT} \--with-tm=${PBSROOT} \--enable-mpi-cxx \--enable-mpi1-compatibility \--with-psm2
make -j ${PARALLEL} && make install && make check# openmpi setting
export OMPI_MCA_btl_openib_allow_ib=1
export CPATH="${OMPIROOT}/include:${CPATH}"
export LIBRARY_PATH="${OMPIROOT}/lib:${LIBRARY_PATH}"
export LD_LIBRARY_PATH="${OMPIROOT}/lib:${LD_LIBRARY_PATH}"
export PATH="${OMPIROOT}/bin:${PATH}"# qe build
cd ${INSTDIR}if [ -d q-e-gpu-qe-gpu-${FULLVER} ]; then
mv q-e-gpu-qe-gpu-${FULLVER} q-e-gpu-qe-gpu-erase
rm -rf q-e-gpu-qe-gpu-erase &fitar zxf ${TARBALL}
cd q-e-gpu-qe-gpu-${FULLVER}
mv * .[a-zA-Z]* ../cd ../ && rmdir q-e-gpu-qe-gpu-${FULLVER}
export MPIF90=mpif90
export MPIFC=mpif90
export MPIF77=mpif90
export MPICC=mpicc
export MPICXX=mpicxx# --with-cuda should point cuda bundled with pgi... but i couldn't do it...
FC=pgfortran F90=pgfortran F77=pgfortran CC=pgcc CXX=pgc++ \./configure --enable-openmp \--enable-parallel \--with-cuda=/local/apl/lx/cuda-10.1 \--with-cuda-cc=60 \--with-cuda-runtime=10.1# force to add cc70 support
sed -i -e "s/cc60/cc60,cc70/" make.inc
make -j${PARALLEL} pw cp
cd test-suite
make run-tests-pw-serial
make run-tests-cp-serial
make clean
make run-tests-pw-parallel
make run-tests-cp-parallel
cd ..

Notes
There were errors on atomic_cmpset* tests of OpenMPI. However, all the QE tests have passed successfully.
Many functions of pw.x seem to be supported by GPU.
(GPU version of cp does not seem to be very useful for now.)
Limited parts such as fft can be performed by GPUs.
Conjugate gradient (cg) is not yet supported.
OpenMP is enabled in this build.
It works both on P100 and V100.
On MPI parallel, single GPU would be assigned to single process.
Processes can share single GPU, but it may not be very effective. (Two processes on single GPU might beadvantageous in some situation.)
Please use mpirun in /local/apl/lx/espresso67-gpu/openmpi-4.0.2 when you perform MPI parallel runs. (Pleasecheck sample job script under espresso67-gpu/samples/.)
Some of calcul;ations might not be supported by GPU. Please check official document or the output file (timinginformation at the end).
Very small calculation may not be significantly accelerated by GPU.
Large calculation may fail due to memory allocation error (probably of GPU memory). This error may be avoided byusing multiple GPUs with MPI parallel run.