Development of 5.04 Mpixel CITIUS detector for high-resolution
ptychography
Charge Integration Type Imaging Unit with high-Speed
extended-Dynamic-Range
This variant is in the development stage

Takaki Hatsui, Kyosuke Ozaki, Yoshiaki Honjo, Haruki Nishino, Kazuo
Kobayashi, Koji Motomura, Kyo Nakajima, Yasumasa Joti

RIKEN SPring-8 Center

18th June, 2024

Coherence 20241

Improve Efficiency by Advanced Detector Technology: CITIUS

Current Limits and CITIUS capability

EIGER, EIGER2From Dectris web site

40%

10 Mcps

3 Mcps[1] Y. Imai and T. Hatsui, J. Synchrotron Rad. 31(2) (2024) 295.
18th June, 2024

Coherence 2024

T. Hatsui

2

CITIUS 5.04 M
Major Specifications

Camera Head (draft)

Saturation Count rate

1 Gcps/pixel @ 8 keV
19 Tcps/cm2

Sensitivity> 90% and uniform

Sensor modules

in a vacuum

Proximity electronics

in airFrame rate

17.4 kHzNoise

40 e-rms (0.018 phs@8 keV)

Data bandwidth

2.8 Tbps

Data depth

32 bit floating

18th June, 2024
Pixel rate

Delivery

Coherence 2024

87.7 Gpixel/s
Dec. 2025

Pixel Size: 72.6 µm□

18 sensor modules(280 kpixels for each )T. Hatsui

3Photon Counting?

EIGER, EIGER2Current State-of-the-art

From Dectris web site

Existing Detectors⁃ PILATUS、Eiger⁃ Medipix-based detectors (Lambda etc.)⁃ HyPix series (RIGAKU) ……
Limited Count Rate
⁃ 1-10 Mcps/pixel(10^3-10^4 photons/pixel @ 10 kframes/s)⁃ reduced further by x80-390 for quasi-single
bunch modes of SPring-8 [1]40%

10 Mcps

3 Mcps[1] Y. Imai and T. Hatsui, J. Synchrotron Rad. 31(2) (2024) 295.
18th June, 2024

Coherence 2024

T. Hatsui

4Count Rate (1/2)
Edge of thevacuum pipe

Photon Energy: 10keV

The most intense
pixel : 590 Mcps

Slit to CITIUS: 15m

Slit Size: 20×20μm
SinglephotonEdge of thevacuum pipe

integrating-type pixel
count rate is independent from bunch patterns
18th June, 2024

Coherence 2024

Y. Honjo, RIKEN

5

Count Rate (2/2)
945 Mcps/pixel
18 Tcps/cm2

Saturated Count Rate (Mcps/pixel)

Photon Energy = 10 keV

1.18 Gcps/pixel @8keV
22.4 Tcps/cm2

1000

X

100
101x1000
1 Mcps/pixel

Si thickness: 650 µm
18th June, 2024

Coherence 2024

T. Hatsui

6

CITIUS 5.04 M
Major Specifications

Camera Head (draft)

Saturation Count rate

1 Gcps/pixel @ 8 keV
19 Tcps/cm2

Sensitivity> 90% and uniform

Sensor modules

in a vacuum

Proximity electronics

in airFrame rate

17.4 kHzNoise

40 e-rms (0.018 phs@8 keV)

Data bandwidth

2.8 Tbps

Data depth

32 bit floating

18th June, 2024
Pixel rate

Delivery

Coherence 2024

87.7 Gpixel/s
Dec. 2025

18 sensor modules(280 kpixels for each )T. Hatsui

7Sensitivity: Photon Counting Detector
Corner Responsepixel

Q.E. map by scanning a pencil beam
Eiger, 8 keV @SPring-8 BL40XU
18th June, 2024

Coherence 2024

Threshold = 0.5 photons

0.25

0.25

000.250.25

00

Medipix solved this by summingat the penalty of the count rate

Integrating-type pixel: free from the corner sensitivity loss
T. Hatsui

8Sensitivity
No aluminum filter
High QE down to 1 keV
Low energy limit
determined by the noisefor the typical case with 40 e-rms
1.5 keV with S/N =5High energy limit
650 µm thick silicon

18th June, 2024

Coherence 2024

T. Hatsui

9Applications to quasi-elastic scattering
Demands high sensitivity @ 14.4 keV
System: CITIUS 840k
Operated at 17.4 kframes/s
5.1 PB/day (35 PB/beamtime)

Makina Saito et.al.,
Phys. Rev. Lett. 132, 256901 (2024)

Summation to 8.7 kframes
On-the-fly calibration, summation andcompression
compression ratio > 100018th June, 2024

Coherence 2024

T. Hatsui

10Parallax
Data at 37 keVdeconvolution software under development
Courtesy of Jaemyung Kim & Y. Hayashi (RIKEN)
18th June, 2024

Coherence 2024

T. Hatsui11

3D XRD to visualize the metal grains

Courtesy of Jaemyung Kim & Y. Hayashi (RIKEN)

111

100

18th June, 2024
2024

Coherence

12110

CITIUS 5.04 M
Major Specifications

Camera Head (draft)

Saturation Count rate

1 Gcps/pixel @ 8 keV
19 Tcps/cm2

Sensitivity> 90% and uniform

Sensor modules

in a vacuum

Proximity electronics

in airFrame rate

17.4 kHzNoise

40 e-rms (0.018 phs@8 keV)

Data bandwidth

2.8 Tbps

Data depth

32 bit floating

18th June, 2024
Pixel rate

Delivery

Coherence 2024

87.7 Gpixel/sDec. 2025

18 sensor modules(280 kpixels for each )T. Hatsui

13Flange-mounted in-vacuum CITIUS
feasibility study for ptychography

in collab. with Y. Takahashi Groupof Tohoku Univ.

840k

18th June, 2024
2024

Coherence

T. Hatsui

14Sample： Ta Test Chart (NTT-AT) 200 nm thick (phase shift 0.41 rad@6.5keV)
Virtual Source Size： 30 µm×150 µm
Source Intensity： 2.6 x 10^10 photons/s
Exposure Time： 1 second
Max. Intensity: 250 Mphotons/s/pixel

Y. Takahashi et.al., J. Synchrotron Rad. Vol. 30(5) (2023) 989.

Low-Q Region

without attenuators
18th June, 2024

Coherence 2024

15Sample： Ta Test Chart (NTT-AT) 200 nm thick (phase shift 0.41 rad@6.5keV)

Pixel Size:
20.94 nm10.46 nmY. Takahashi et.al., J. Synchrotron Rad. Vol. 30(5) (2023) 989.
Resolution Evaluation by phaseretrieval transfer function

5.23 nmPhase (rad)0.2

Exposure Time： 1 s/scanning point
Num. of Scanning points： 17 x 17-0.4

Full-period spatial resolution ofabout 10 nm demonstrated

Nov. 13rd 2023

50 nm feature Resolved

500 nm16CITIUS 5.04 M
Major SpecificationsCamera Head (draft)

Saturation Count rate

1 Gcps/pixel @ 8 keV
19 Tcps/cm2

Sensitivity> 90% and uniform

Sensor modules

in a vacuum

Proximity electronics

in airFrame rate

17.4 kHzNoise

40 e-rms (0.018 phs@8 keV)

Data bandwidth

2.8 Tbps

Data depth

32 bit floating

18th June,Pixel
2024 rate

Delivery

Coherence 2024

87.7 Gpixel/s
Dec. 2025

18 sensor modules(280 kpixels for each )T. Hatsui17

BL29XU EH3 in July 2021
XPCS @ 17.4 kfps

Y. Honjo, K. Ozaki, H. Nishino
T. Hoshino

CITIUSPixel Number: 280 kpixels
Frame rate: 17.4 kframes/s
Frame Cycle: 57.5 μs
Duty ratio: 94 %40 Mphotons/s/pixel at 8 keV.

Sample:Silica (100 nmφ, 28.8 wt%)
in MEK + PEI + MeOH (66.8, 3.9, 0.5 wt%)
in 0.5mm capillary
Slit:
Size: 20 μm □,Position: 1 m upstream of the sample
Incident X-ray beam:
Photon energy: 8 keV,
Photon flux: 2E9 photons/s
Geometry:
Camera length: 3.2 m
Image Processing:
Dark image subtracted
18th June, 2024
2024

Coherence

T. Hatsui, RIKEN
18CITIUS 280k
Bragg CDI at ESRF-EBS

M. Grimes et.al., J. Appl. Cryst. 56(4)2023
https://doi.org/10.1107/S1600576723004314

Count rate

ResolutionAcquisition
TimeCITIUS

30 Mcps/pixel

20 ± 6 nm

23 s

MAXIPIX

1 Mcps/pixel

22 ± 9 nm

200 s

A detailed analysis showed CITIUS with 1
ms exposure gives similar quality of datawith MARPIX with 1 s exposure.

CITIUS 5.04 M
Major Specifications

Camera Head (draft)

Saturation Count rate

1 Gcps/pixel @ 8 keV
19 Tcps/cm2

Sensitivity> 90% and uniform

Sensor modules

in a vacuum

Proximity electronics

in airFrame rate

17.4 kHzNoise

40 e-rms (0.018 phs@8 keV)

Data bandwidth

2.8 Tbps

Data depth

32 bit floating

18th June,Pixel
2024 rate

Delivery

Coherence 2024

87.7 Gpixel/s
Dec. 2025

18 sensor modules(280 kpixels for each )T. Hatsui

20NoiseTi.

Fe.

Ni. Fe Cu.

Cu→ Spectro-imaging

K. Ozaki, Y. Honjo, et.al.,
18th June, 2024

Coherence 2024

T. Hatsui21

X-ray Beam Monitor for SPring-8-II
Accelerator feedback with electronmonitors

K. Ozaki, T. Kudo, S. Takahashi, M. Sano, T. Itoga, et.al.,⁃ not enough

X-ray Intensity Image

X-ray beam position monitor is required.

Undulator beam is hidden by
the bending X-rays

Spectro18th June, 2024

Coherence 2024

T. Hatsui, RIKEN
22

X-ray Beam Monitor

T. Kudo et.al., J. Synchrotron Rad. (2022). 29, 670-676

K. Ozaki, T. Kudo, S. Takahashi, M. Sano, T. Itoga, et.al.,

18th June, 2024

Coherence 2024

T. Hatsui, RIKEN
23

CITIUS 5.04 M
Major Specifications

Camera Head (draft)

Saturation Count rate

1 Gcps/pixel @ 8 keV
19 Tcps/cm2Sensitivity> 90% and uniform

Sensor modules

in a vacuum

Proximity electronics

in airFrame rate

17.4 kHz

Noise40 e-rms (0.018 phs@8 keV)

Data bandwidth

2.8 Tbps

Data depth

32 bit floating

18th June,Pixel
2024 rate

Delivery

Coherence 2024

87.7 Gpixel/s
Dec. 2025

18 sensor modules(280 kpixels for each )T. Hatsui

24

CITIUS 20.2M for SACLASALCA: XFEL facility with 60 Hz
Major Specifications
Frame rate: 5 kHz
DAQ: 960 Hz in the 16-sampling mode
(1 pulse image is taken by 16 images)
DAQ bandwidth: 620 Gbps @ 32 bit/pixelFirst Beam Test: July 2024

18th June, 2024

Coherence 2024

T. Hatsui25

Mo 40 kV,500 µA, total exposure time 338 ms (6400 frames)

flat field image
18th June, 2024 dark subtracted
Coherence 2024

transmitted image
flat field corrected T. Hatsui

26

Enlarged view of the transmitted image
flat field correctedNoise and Energy Resolution

25 e-rms〜250 eV FWHM

2024/8/7

28

Edge Computing for CITIUS 20.2M
FPGA-based Edge Computing

DAQ bandwidth: 620 Gbps
On-the-fly calibration and compression
developed with R-CCS Fugaku teamcf) Fugaku: a supercomputer with 488 PFLOPS
18th June, 2024

Coherence 2024

x4.5 will give 2.8 Tbps computation
T. Hatsui29

Edge Server Configurations
CITIUS 20.2M
for SACLA
2 Sensor per 1 FPGA card
x6 computation resource
CITIUS 5.04M
for ptychography

1 Sensor per 3 FPGA cards
already validated for CITIUS 840k [1-3]

18th June, 2024

Coherence 2024

[1] Makina Saito et.al., Phys. Rev. Lett. 132, 256901 (2024)[2] H. Nishino, Nucl. Inst. and Meth., A, Vol. 1057, (2023) article id. 168710.
[3] Y. Takahashi, J. Synchrotron Rad. Vol. 30(5) (2023) 989.
T. Hatsui
30Computation for 5.04 Mpixel CITIUS
FugakuSACLA Data Center

Theoretical peak performance: 0.360 Pflops
5k CPU cores (intel CPUs)
Nvidia A100 x16
Out-bound: 100 Gigabit network
Storage capacity: 20PB
File system: Lustre (~30GB/sec)

18th June, 2024

Coherence 2024

x2000

Theoretical peak performance: 488 Pflops
Arm-based CPUs
Total memory capacity: 4.85 PiBInterconnect: Tofu Interconnect D (28 Gbps x 2 lane x 10 port)

Collaboration with Argonne National LabT. Hatsui

31Summary
CITIUS 5.04M for ptychography is under development• major specifications
Saturation Count rate

1 Gcps/pixel @ 8 keV
19 Tcps/cm 2Sensitivity> 90% and uniform

Sensor modules

in a vacuum

Proximity electronics

in airFrame rate

17.4 kHzNoise

40 e-rms (0.018 phs@8 keV)

Data bandwidth

2.8 Tbps

Data depth

32 bit floating

Pixel rate

87.7 Gpixel/s

DeliveryDec. 2025

18th June, 2024

Coherence 2024

demonstrated

620 Gbps demonstrated for CITIUS 20.2M
x4.5 needed
x6 resource increase by configuration change

T. Hatsui32

Acknowledgment
RIKEN and JASRI Team
⁃ T. Kudo, T. Sugimoto, M. Yamaga, T. Kameshima, Y. Inagaki, K. Fujiwara, T. Nakagawa,Y. Oyaki, M. Kimoto, M. Nakamachi, M. Yabashi, T. Ishikawa

RIKEN R-CCS⁃ S. Matsuoka, K. Sato, K. Sano, F, Shoji and their division members

Private Companies⁃ Sony Semiconductor Solutions⁃ GLORY System Create Ltd⁃ Nihon Gijyutu Center⁃ Meisei Electric Co. Ltd.
⁃ JEPICO Corporation⁃ Tokyo Electron Device Limited
Dec. 13rd 2023

Thank you for your attention.

T. Hatsui33

SPARE SLIDES

18th June, 2024

Coherence 2024

T. Hatsui34

Bottleneck delivering CITIUS:
2020–2023 Global chip shortage
Sensor modules: Sony produced and delivered on time.
After the reliability review, we got approval for mass production (MP),In total, 249 MP devices have been produced.
Proximity board, data-processing cards:
delayed by about one year, but now solved and under production at 100 units/year

18th March, 2024

IFDEPS

T. Hatsui35