Why Large Scale DFT Calculations?

FET with CNT
IBM3Dim Si NAND Memory
Tohoku Univ

Si-Nanowire FETTokyoIT, IBM，,,,

Nano-Scale Structure (Nano-Shape) Modifies Wavefunctions and then Material Properties

Computics: Fusion of Materials Science and Computer Science Saturation of performance of a single
compute processor
Multi-core massively parallel architecture withhardware accelerators
Collaboration between Computational Physics/Chemistry& Computer Science Developments of Mathematical Methodology andAlgorithms Suitable to Current Hard wares

1

http://computics-material.jp/index.html

Advantages of Real-Space DFT (RSDFT) in Parallel Computing

Real space
CPU space

Electron-state space

On multi-core parallel machines:- Huge number of 3D mesh points divided intomoderate-size cells- Each cell treated by a single node or core
- Also, electron states (orbitals) are groupedand each group is treated by a single node ora core : Hybrid parallelization- MPI for CPU and OpenMP for core2

Almost free from FFT, reducing
communication burden⇒ high efficiency
 Flexible boundary condition towave-functions⇒ targets expanded
including charged objects


ISWGPDS2021

High Performance Computing (HPC)：Maximum Use of BLAS3 or Hit the Cache!
BLAS: Basic Linear Algebra Subprograms
Solving Kohn-Sham Equation 1 2rr−∇+vn(;())
kkeff
 2
 ϕi (rk ) = ε i ϕi (rk )Getting SCF by iterative computations

Linear Algebraic Calculation
1. Inner product of vectors
NA = ∑ a jbj
j =1

1. Conjugate gradient methodO(N 2)
2. Ortho-normalization by Gram-SchmidtmethodO(N 3)
3. Density and Potential update
O(N 2)

BLAS Level 1

Computing = 2N, Data = 2N + 1

2. Matrix times vector
NAi = ∑ aij b j
j =1

BLAS Level 2

Computing = 2N2, Data = N2 + 2N

3. Matrix times matrix
NAik = ∑ aij b jk BLAS Level 3
j =1

O(N 3) operations, e.g., Gram-Schmidt
should be accelerated!!

3

Computing = 2N3, Data = 3N2High ratio of (computing / data ) in BLAS3 has anadvantage in large-scale computations!!
ISWGPDS2021

Gram-Schmidt Ortho-normalization:
Maximum use of BLAS Level 3 O(N3) Computation
ϕ1 = ψ 1ϕ=
ψ 2 − ϕ1 ϕ1 | ψ 22ϕ3 =ψ 3 − ϕ1 ϕ1 | ψ 3 − ϕ2 ϕ2 | ψ 3On K computer,
96.6 % efficiency
using DGEMM tothe peak performanceϕ4 =ψ 4 − ϕ1 ϕ1 | ψ 4 − ϕ2 ϕ2 | ψ 4 − ϕ3 ϕ3 | ψ 4
ϕ5 =ψ 5 − ϕ1 ϕ1 | ψ 5 − ϕ2 ϕ2 | ψ 5 − ϕ3 ϕ3 | ψ 5 − ϕ4 ϕ4 | ψ 5
ϕ6 =ψ 6 − ϕ1 ϕ1 | ψ 6 − ϕ2 ϕ2 | ψ 6 − ϕ3 ϕ3 | ψ 6 − ϕ4 ϕ4 | ψ 6 − ϕ5 ϕ5 | ψ 6..............................
Most calculations can be performed asMatrix x Matrix operations!

 †ϕ1 ϕ3 , ϕ4 , ϕ5 , ϕ6 ,....] [ψ 3 ,ψ 4 ,ψ 5 ,ψ 6 ,....] − [ϕ1 , ϕ2 ]  †  [ψ 3 ,ψ 4 ,ψ 5 ,ψ 6 ,....][= ϕ2 4ISWGPDS2021

RSDFT on K Computerat Kobesourcechanneldrain

 Benchmark Data for Si nanowires– 107,292-atom LDA/GGA electronic-structure
calculations using 82,944 nodes•

5.48 PFLOPS = 51.7% efficiency to the peak performance•Single iteration in SCF = 2,900 sec

Side view

Gordon Bell Prize 2011

 LDA/GGA Run for 10,000-atom Systems

Cross secction– Getting SCFields = one day using ～102 nodes with 34% - 70% efficiency– Good strong scaling (78% - 88%) for 4,608 – 15,552-atom systems using 768 – 12,288 nodes

 HSE Calculations for 1000-atom systems– Anti-ferromagnetic spin polarization along step edges of nanofacets in SiC
Spin density by HSE

5Sawada, Iwata, Oshiyama, PRB 93, 235421 (2016)

ISWGPDS2021