Parallelization of Atomic Image
Reconstruction from X-ray Fluorescence
Holograms with XcalableMP
Atsushi Kubota1
Tomohiro Matsushita2 Naohisa Happo1
1Hiroshima City University
2Japan Synchrotron Radiation Research Institute (JASRI)

Outline• What is “X-ray Fluorescence Holography” ?
• Parallelization of 3D atomic image
reconstruction• Evaluation on PC cluster• Conclusion

2

JSPS Grant-in-Aid for Scientific Research on Innovative Areas“3D Active-Site Science”
2014-2019

Conventional techniques such as X-ray diffraction→ Global structure analysis
Our Approach

From http://www.3d-activesite.jp
In many functional materials, the local structure of the “dopant,”“hetero-interface,” and “nanomaterial,” i.e., the “Active-Sites”
in the parent material play a vital role in the functional expressionof the material
3D Atomic Resolution Imaging
・Fluorescent X-rays and Photoelectron Holography
・Surface/Interface Holography
・Nano-structure Imaging
3

Experiment：Inverse Mode of
X-ray Fluorescence Holography
Detector

Incident
X-ray

X-ray fluorescence

φSampleθ

Intensity of X-ray fluorescence is measured
while rotating samples
4

Synchrotron radiation facilities

Spring-8
http://www.spring8.or.jp

Photon Factory,
High energy accelerator
http://www2.kek.jp/imss/pf/

5Principle of Inverse Mode

Intencity of X-ray
Fluorescense
Ｉλ

Incident X-ray

ReferenceWave

Detect X-ray
Fluorescence Hologram

d cos ααEmit X-ray
Fluorescence

A(0,0,0)B(x,y,z)dObject
Wave
Scatter theIncident X-ray
6

Analysis Procedure of
Atomic Image Reconstruction from
X-ray Fluorescence Holograms
Holograms obtained
by Experiment
Removal of
background waves
Completion of
sphere data

Long time required for
Reconstruction on computer

Reconstruction of
atomic images(same as discrete
Fourier transform)

Performance improvementneeded7

A Hologram after
removel of background waves

8Holograms after
completion of sphere data

18500 eV

23500 eV9

An example 3D reconstructed image

10

An example of 2D reconstructed image(z=2.0)11

Principle of Inverse Mode

Intencity of X-ray
Fluorescense
Ｉλ

Incident X-ray

Reference
Wave

Detect X-ray
Fluorescence Hologram

d cos ααEmit X-ray
Fluorescence

A(0,0,0)B(x,y,z)dObject
Wave
Scatter theIncident X-ray
12

3D Atomic Image Reconstruction
by Discrete Fourier Transform
Detection of X-ray Fluorescence Hologram
Position on the sphere l distance apart from sample：： λPosition of atom A：OPosition of atom B：
Caluculation of image χ atParallelizable for X, Y, Z
Sum for θ,φ,λSuperimpose multiple holograms obtained by several input X-rays
with different wave lengths λ → Reduction of ghost image 13

Parallelization of3D Atomic Image Reconstruction• Hard to apply FFT to Reconstruction(DFT)
Input data on the polar coordinate system
Output image on the rectangular coordinate system• 2D atomic image reconstruction by OpenMP• 3D atomic image reconstruction by hybrid
parallelization of XcalableMP and OpenMP14

Programming Languages / Libraries for
Parallel Processing

HighLevel

Distributed Memory

HPFPGAS

Shared Memory

OpenMP(XcalableMP,GoTBB
X10,Chapel,Fortress CAF,UPC) OpenACC

OpenCL,CUDA

Low
Level

MPI, PVM

pthreads

socketshmem
15

Parallelization of 2D Atomic Image
Reconstruction by OpenMP• Loop interchange– Loop nest of x,y,θ,φ,λ to λ,x,y,θ,φ– Expect• Improvement of cache hit ratio• SIMDization(such as Intel SSE) by increased number of iterations ofthe inner most loop

• Table look up of trigonometric function– Some calls of trigonometric functions are replaced withreferences of arrays calculated before entering loop
– Some trigonometric calls remain in the kernel loop nest

• Parallelize x loop by OpenMP16

Table Look-up of Trigonometric Function Calls/* Trigonometric function calls in the loop*//* Save trigonometric function values */for(th=0;th<NTH;th++) {for(phi=0;phi<NPHI;phi++) {… cos(phi)… sin(phi)}}for(phi=0;phi<NPHI;phi++) {ctab[phi]=cos(phi);stab[phi]=sin(phi);}/* Table loop-up in the loop */for(th=0;th<NTH;th++) {for(phi=0;phi<NPHI;phi++) {… ctab[phi]… stab[th]}}17

Parallelization of 3D Atomic Image
Reconstruction by XcalableMP• The 6 nests of loop λ,z,x,y,θ,φ– Z dimension is added to 2D reconstruction• Table loop-up of trigonometric function calls– same as 2D reconstruction• Parallelize z loop by XcalableMP18

Data Declarations in XcalableMP/* Input hologram in0 is duplicated in the all nodes */double in0[NTH][NPHI];/* Output image out1 is distributed among nodes */double out1[NZ][NX][NY];#pragma xmp nodes p(8)#pragma xmp template t(NZ)#pragma xmp distribute t(block) onto p#pragma xmp align out1[z][*][*] with t(z)/* Output image out2 is duplicatged in the all nodes */double out2[NZ][NX][NY];19

Parallelized Nest of Loops of3D Atomic Image Reconstructionfor (lambda=0;lambda<NL;lambda++) {#pragma xmp loop on t(z)for (z=0;z<NZ;z++) {#pragma omp parallel forfor (x=0;x<NX;x++) {for (y=0;y<NY;y++) {for (th=0;th<NTH;th++) {for (phi=0;phi<NPHI;phi++) {… ctab[phi]… stab[th]out1[z][x][y] = …
}}}}}}

Parallelize Z loop
among nodes
Parallelize X loop on
each node20

File I/O and Aggregation of Data/* Read input hologram in0 from file on the all nodes */fread(in_file, in0,…);…
/* Parallelized Kernel DFT Loop */…
/* Aggregate distributed data out1 to out2 */#pragma xmp gmove
out2[:][:][:] = out1[:][:][:];/* Write atomic image out2 only on node 1 */Aggregation to avoid writing
data to single fileby multiple nodes

if(xmp_node_num()==1) {fwrite(out_file, out2,…);}
21

Performance Evaluation: I/O Data
• Input data file

• Sample：PZT• Incident angle1 θ=1°~ 179°, 1°resolution• Incident angle2 φ=0°~ 359°, 1°resolution• Record measured intencity of X-ray fluorescence• Energy of incident X-ray• 18,500 eV – 23,500 eV• 250 eV step, 21 levels of energy• Output data file– 2D atomic image: 192×192– 3D atomic image: 192×192×192– -9.6Å ~ 9.6Å, 0.1Å resolution22

PC cluster used for reconstruction
PC cluster
The number of nodes

8

The number of sockets / node2

CPUXeon X5660 2.8GHz

The number of cores (threads)

6 cores (6threads, HT off)

The total number of cores

96

Cache size

L3:12MBInter-node network

4Gbps(InfiniBand DDR)

XcalableMP Compiler

1.2.2

Compiler, Optimization

Intel Compiler 18.0.1 –O3 xHOSTOSLinux (CentOS 6.3)
23

Results of 2D Atomic Image
Reconstruction
Sample: PZT
1200

Execution Time(sec.)

1000

972.473914.301

800600Xeon X5660

40020075.8140
Original

1 CoreTable look-up andLoop interchange

12Cores12.8 xSpeed up24

Results of 3D Atomic Image
Reconstruction
64Speedup

32
16Ideal

8Z8Z201

42112481632

64

The number of threads
Parallelize loops with XMP and OpenMP

25

Break Down of Execution time
3D Image (NZ,NX,NY)=(192,192,192)(#Nodes×FileTotal#Threads/ Execution
File Input Aggregation Output
threads
Node)Time (sec.) Speedup(sec.)(sec.)(sec.)
8 (8×1)
21,683.076
8.000
0.781
0.176
9.772
48 (8×6)
3,623.352
47.874
0.721
0.163
9.731
96 (8×12)
1,840.942
94.226
0.767
0.174
9.701

From 6 hours
To 30 min.

Estimated
2days withsingle thread

3D Image (NZ,NX,NY)=(8,192,192)(#Nodes×File#Threads/ Execution
File Input Aggregation Output
Total threadsNode)Time (sec.) Speedup(sec.)(sec.)(sec.)
1 (1×1)
7,214.325
1.000
0.745
0.007
0.301
8 (8×1)
923.830
7.809
0.473
0.009
0.303
12 (1×12)
603.962
11.945
0.459
0.005
0.310
48 (8×6)
151.574
47.596
0.458
0.008
0.303
96 (8×12)
76.57694.211
0.455
0.009
0.310

From 2 hours
To 1min.17sec.
26

Compare XcalableMP with MPIParallelization

Execution Time(sec.)

Modified/Inserted
Lines of CodeXcalableMP

1,840.942

32

MPI

1,817.042

53

3D Image (NZ,NX,NY)=(192,192,192)
96 threads execution
Higher productivity by XcalableMP than MPI
w/o sacrificing performance

In general…

Productivity

Performance

XcalableMP(High Level)○△

MPI(Low Level)

×○27

Conclusion
Hybrid parallelization of 3D atomic image
reconstruction from X-ray fluorescence hologramswith XcalableMP and OpenMP– PC Cluster (8nodes, 96cores in total)• Grid size of 3D Atomic Image (8,192,192)– 94x speedup by 96 threads• Grid size of 3D Atomic Image (192,192,192)– 30 min. (1,841 sec.) with 96 threads– Estimated 2days with single core

– Inter-node parallelization with XcalableMP– Intra-node parallelization with OpenMP– High productivity and performance by XcalableMP28

Future work
Execution of 3D Atomic Image Reconstruction
on supercomputers such as GPU cluster (Ito)at Kyushu University29