CCS-EPCC Workshop
Dc. 7-8, 2017

Ab-initio density functional simulation for nano-optics
K. YABANA
Division of Quantum Condensed Matter Physics
Center for Computational Sciences
University of Tsukuba

http://salmon-tddft.jp
We are developing a computational code, SALMON

Scalable Ab-initio Light-Matter simulator for Optics and Nanoscience
It is capable of describing light-matter interactions
starting with ab-initio electron dynamics calculations.

Two aspects of light-matter simulation
Light propagation usually describedusing Maxwell’s equations
Macroscopic electromagnetism
Light wave of pulsed laser

Electron dynamics is inducedby light electric field, describedby Schrödinger equation
Quantum mechanics
Atoms, molecules, solidshttp://salmon-tddft.jp
We are developing a computational code, SALMON

Scalable Ab-initio Light-Matter simulator for Optics and Nanoscience
It is capable of describing light-matter interactions
starting with ab-initio electron dynamics calculations.

Two aspects of light-matter simulation
Light propagation usually describedusing Maxwell’s equations
Macroscopic electromagnetism
Light wave of pulsed laser

Electron dynamics is inducedby light electric field, describedby Schrödinger equation
Quantum mechanics
Atoms, molecules, solids

Popular tool in computational materials science:Density functional theory (DFT)𝜀" 𝜙" 𝑟 =& '𝑝 + 𝑉",- 𝑟'(+ 𝑉. 𝑟 + 𝑉/0 𝑟

𝜙" 𝑟

Theory for ground state. Not applicable to photoexcitation
W. Kohn, 1998 Nobel prize in chemistry

Extension to electron dynamics: Time-dependent DFT (TDDFT)𝑖ℏ3𝜓 𝑟, 𝑡34 "=& '𝑝 + 𝑉",- 𝑟'(+ 𝑉. 𝑟, 𝑡 + 𝑉/0 𝑟, 𝑡 + 𝑉8/4 𝑟, 𝑡

𝜓" 𝑟, 𝑡Electronic excited states, electron dynamics under external field, …

Real-time response: Optical response of Ethylene (C2H4) molecule
Dipole moment
Electron dynamics in ethylene

p-p* oscillation(Applying an impulse at t=0)

time

Photoabsorption cross section

Real-time and real-space method
YXZ

Real-space grid representationin 3D Cartesian coordinate- High-order finite differenceTime evolution calculation by explicit method(Taylor expansion of 4th order)
N1 æ hKS (t )Dt öé hKS (t )Dt ù()y i (t + Dt ) = expêyt»ç÷y i (t ),åú ii!
k!
i!
ëûèøk =0N =46

Calculation of large systems using massively parallel supercomputers
Surface plasmon resonance of Au clusters

4 nm

40 6

Oscillator strength (1 / eV)

Au1414Au560

35 4 146
30 2141454

25 02.0 2.5 3.0
2.0922

3.0

20560

15308

10
50
1.5

2.0

2.5

3.0

3.5

4.0

Energy (eV)15,554 electrons

0.08

strength (1 / eV)

Au54

0.07

0.020.2

0.06

0.010.12.1 JPCA,
K. Iida, M. Noda, K. Ishimura, K. Nobusada,118,
2.32.5 11317 (2014)

0.05

2.1

0.041462.32.5(Group of Institute for Molecular Sciences,Okazaki)
54

7

Electric Field (a.u.)

Crystalline silicon under intense laser pulse

0

10

20
30Time (fs)40

⎡ 1 ⎛ ! ! e ! ⎞2! e2!
! ⎤⎤ ! !
∂ ! !
⎡i! unk ( r, t ) = ⎢⎜ p + k + A (t ) ⎟ + ∫ dr ' ! ! n ( r ', t ) + µ xc ⎣n ( r, t )⎦⎥ unk ( r, t )⎠∂tcr −r'⎢⎣ 2m ⎝⎥⎦(direct bandgap 2.4 eV in LDA)
E = 27.5 V/nm!ω = 1.55 eV
TFWHM =7 fs

Electrondensity8http://salmon-tddft.jp
We are developing a computational code, SALMON

Scalable Ab-initio Light-Matter simulator for Optics and Nanoscience
It is capable of describing light-matter interactions
starting with ab-initio electron dynamics calculations.

Two aspects of light-matter simulation
Light propagation usually describedusing Maxwell’s equations
Macroscopic electromagnetism
Light wave of pulsed laser

Electron dynamics is inducedby light electric field, describedby Schrödinger equation
Quantum mechanics
Atoms, molecules, solids

Usually, two simulations are separately used in optical sciences
Macroscopic Electromagnetism (EM)Quantum Mechanics (QM)

Light propagation description by Maxwell
equations. Materials’ properties comes into
through constitutive relations (dielectricconstant).

First-principles calculations for dielectricfunction.
Perturbation theory in quantum mechanics.

D = εE
Constitutive relation
connects two theories

However, at the frontiers of optical science, unified simulation is required.

Ordinary electromagnetism
D(~r, t) =Z t

dt0 ✏(tt0 )E(~r, t0 )Linearity and locality
In forefront optical science, a theoretical and computational approach
unifying Electromagnetism and Quantum Mechanics is required.
Extreme nonlinear optics

Nano-optics
Oscillator strength (1 / eV)

40 635 4 146
30 2141454

25 02.0 2.5 3.0
2.0922

3.0

20560

15308

1050
1.5

2.0

2.5

3.0

3.5

4.0

Near field induces strong
nonlinear interactions

4.0

Energy (eV)High harmonic generationfrom atoms and solids

Nonthermal laser processing
using femtosecond laser pulses

Renormalized strength (1 / eV)

0.08
0.020.2

0.07
0.06

0.010.12.1
2.1

0.05

2.3

2.3

2.5

2.5

54146308560922
1414

0.04
0.03
0.02
0.01
0.00

1.5

2.0

2.5

3.0

3.5

Energy (eV)Figure 2: (upper panel) Oscillator strengths of Aun (n = 54, 146, 308, 560, 922, aand (lower panel) their renormalized strengths with respect to the number of atomin the upper and lower panels show magnifications of the curves for Au54 and Au1
Au560 , Au922 , and Au1414 , respectively.

SALMON

Light propagation description: Coupling with Maxwell equations‘Multi-scale’ coupling
Macroscopic gridfor Maxwell equationµm scale

Microscopic gridfor RT-TDDFT electron dynamics

ｚ1 ¶2¶24p()()AtA
t=J Z (t )ZZc 2 ¶t 2
¶Z 2c

less than nm scale!
!
!
!
∂⎡
i! uZ,nk! ( r, t ) = h ⎣ AZ (t ), nZ ( r, t )⎤⎦ uZ,nk! ( r, t )
∂t!
1 ! !
J Z (t ) = ∫ jZ ( r, t )V V

At each macroscopic grid point,
we solve real-time electron dynamics in parallel
K. Yabana et.al, Phys. Rev. B85, 045134 (2012).

Ab-initio simulation for light-wave propagation in Si

I=1010W/cm2
A/cLaser frequency：1.55eV：
lower than direct bandgap
2.4eV(LDA）

VacuumSi[mm]Z=0 [mm]

Z=0.8 [mm]Z=1.6 [mm]14Towards Laser processing of transparent materials
At which intensity of light, glass changes from transparent to opaque material ?

15

Laser pulse propagation through SiO2 10µm thin film

Compare pulses of different intensitiesThin film of SiO2
10µm

Exp: Attosecond streaking measurements
by Max-Planck Inst. Quantum 16Optics

Maxwell + TDDFT multiscale simulation : 10 mm SiO2
Laser electric field, red (strong), blue (weak)!ω = 1.55eV

λ =800nm
I=7 ×1013 W/cm 2

Energy transfer from laser pulse to electrons [eV/atom]

80,000 cores, 20 hours
at K-Computer, Kobe

17

0.1 eV/atom

Energy deposited to electrons in SiO2
increases rapidly at a certain laser intensity.
>> Damage threshold

I=5 ×1013 W/cm 20Multiphoton/Tunneling excitation

0.1 eV/atom

I=7 ×1013 W/cm 2

Conduction bandbandgap
10 eV

10mm

1.55eV*7
0.1 eV/atom

I=1×1014 W/cm 2Valence band18

Energy deposition from laser pulse to SiO2 at mid point (5µm)

Experiment

TheoryA. Sommer et.al, Nature 534, 86 (2016).
(EXP: Max Planck Institute for Quantum Optics)
19

3D Maxwell + 3D RT-TDDFT simulation: a computational challenge
Our university started to operate
supercomputer ‘Oakforest-PACS’ from 2016.11.
1st in Japan
9th in the world (Nov, 2017)
8208 nodes x 68 cores (Intel Xeon Phi 7250)
25 PFLOPS

2017.3 We had 3-day machine time to use full nodes.

(a) Macroscopic system(b) Microscopic system (Silicon)

Strong laser pulses on Silicon nano-structures: Two test cases• Silicon triangular prisms
EM Energy

Excitation Energy

40nm

600nm

E(t)

Si420nm

Optical hot spot at the gap• Silicon nanosphere
EM Energy

Excitation Energy~250 nm

SiE(t)

Focus of incident light

We anticipated: dielectric response for weak, and metallic response for strong laser pulses.

Laser pulse (1012W/cm2, 1.55eV, 5fs) on silicon-nanosphere~250 nmSi- 3D Maxwell + 3D TDDFT multiscale calculation- Si sphere is expressed by 32,752 points(32,752 electron dynamics calculation in parallel)- 2.5 hourse by Oakforest-PACS 8188 nodes

Energy densityof EM fields
𝐸2 + 𝐵2

Electronic
excitation energy(two photonabsorption)
22

Multiscale Maxwell-TDDFT calculation
Macroscopic system (EM field)

Microscopic system (Electron dynamics)x ⇠ 0.05 nmx ⇠ 10 nm

FDTD calculation is inexpensive
and carried out redundantly in nodes.

1 node calculates 4 electron dynamics.
(32,752 electron dynamics by 8,188 nodes)
Each electron dynamics calculation uses
- 16x16x16 spatial grids- 8x8x8 k-points- 16 orbitals

Total number of variables : 1.1 x 1012
Number of time step : 20,000Performance in various processors
In-House collaboration with Computer Science group(Prof. Boku, Ph.D student Hirokawa)

Max: 3Tflops

Max: 128Gflops

24Weak scaling
4 PFLOPS
Hamiltonian performance
4096

4502048

300
250
200
150
100
50

Graphite

1024
512

Higher is
BetterPerformance [TFLOPS]

350

Lower isFaster

Dynamics time / Iteration [msec]

400

256
12864

Silicon0Graphite

Silicon

32
128

512
2048
# of compute node

8192

1285122048# of compute node8192

Strong scaling
Silicon1000

Lower is Faster

Dynamics time / Iteration [msec]

Graphite

100

Saturation is seen for graphite1051210242048# of compute node40968192

Summary
We are developing SALMON- ab-initio calculation for light-matter interaction- large-scale computing for Maxwell + TDDFT multiscale simulation- in-house collaboration between application and computer researchers- good scaling and performance using many-core processors- provide numerical experiment platform for forefront optical science- to be usable by experimental and company researchers

SALMON-TDDFT Code-Project
Scalable Ab-initio Light-Matter simulator for Optics and Nanoscience
Open-source, Real-time TDDFT (+Maxwell)
http://salmon-tddft.jp/Acknowledgement
Financial supports

Collaborators
Univ. Tsukuba
Mitsuharu Uemoto
Yuta Hirokawa
Taisuke Boku
Univ. Tokyo
Yasushi Shinohara
Max-Planck Institute for
Structure and Dynamics
of MatterShunsuke Sato

Univ. Washington
George F. Bertsch
Max Planck Institute forQuantum Optics
Annkatrin Sommer
Martin Schultze
Ferenc Krausz

Supercomputers28