New opportunities in Photon Science withhigh-speed X-ray imaging detector CITIUS,
and associated data challenge
Takaki Hatsui
RIKEN SPring-8 Center

R-CCS workshop Feb. 17th, 20201

Outline Photon science in 2020s
 Detector Technology: New Trends
 CITIUS detector

 Data Challenge Summary

R-CCS workshop Feb. 17th, 2020

T. Hatsui, RIKEN

2

Where ?

SPring-8XXKobe

80 km from R-CCS（C）OpenStreetMap
contributors

R-CCS workshop Feb. 17th, 2020

T. Hatsui, RIKEN

3

Photon Science: X-ray Science based on Accelerators
SPring-8 Site

X-ray Free-electron Laser
SACLA (since 2012)
3 beamlines

Synchrotron Radiation Source
SPring-8 (since 1997)
57 Beamlines
Circumference 1.5 km

Quasi continuous chaotic x-rays< 1 PB/year

Femto-second Laser shining at 60 Hz
1.6 PB/year

R-CCS workshop
Feb. 17th, 2020

Today, we focus on synchrotron
radiation (SPring-8).

T. Hatsui, RIKEN

4

SPring-8 Science Highlights (from MEXT web site) Macromolecule structure
In July 2008, Dr. Sam-Yong Park of
Yokohama City University clarified thestructure of an important section of aprotein related to influenza virus,
which will lead to the development ofnew drugs.

 Polymer science
Development of high performance,fuel-efficient tires was enabled byutilizing SPring-8, J-PARC, and K
computer

R-CCS workshop Feb. 17th, 2020

More can be found at SPring-8 web site
http://www.spring8.or.jp/en/news_publications/research_highlights/
5How to generate X-rays?
Müuller Heavy Target Tube ca 1890-1910

Museum Dedicated to Surgical Science, Chicago
R-CCS workshop
Feb. 17th, 2020

T. Hatsui, RIKEN

6

X-ray Tube, Synchrotron Radiationelectronca. 1990
Atomic Nuclei(Nmagnet)1(Nmagnet)2ca. 1960s
Late 70's/early 70's
1982: Photon Factory
1994: ESRF
1996: APS
1997: SPring-8
2013: Petra III

107
108
1012
1017
1019-21

Moore’s Law 2x in 18 months
Synchrotron Radiation brightness 3x in 18 months
Shpyrko O G 2004 Experimental x-ray studies of liquid surfaces PhD Thesis
Harvard University
R-CCS workshop Feb. 17th, 2020

7

X-ray Tube, Synchrotron Radiationelectronca. 1990
Atomic Nuclei(Nmagnet)1(Nmagnet)2
Multi-bend
achromatelectron lattice
R-CCS workshop Feb. 17th, 2020ca. 1960s
Late 70's/early 70's
1982: Photon Factory
1994: ESRF
1996: APS
1997: SPring-8
2007: Petra III
2020: ESRF EBS
2024: APS-U
202X: Petra IV
202X: SPring-8-II

10710810121017
1019-211022-23

SPring-8-II Conceptual Design Report (Nov., 2014)8Outline Photon science in 2020s
 Detector Technology: New Trends
 CITIUS detector

 Data Challenge Summary

R-CCS workshop Feb. 17th, 2020

T. Hatsui, RIKEN

9X-ray Material Interactions
X-ray Imaging detector needs
FluorescenceIF ∝ A
X-ray photon energy detection

SampleX-raysR-CCS workshop Feb. 17th, 2020

Elastic scattering/diffraction
Ies ∝ q-4 (uniform material)
Ies ∝ Eph-2
Angle detection: ～sub 100 µm pixel
High Dynamic Range Imager
Absorption
A= - log (I/Io)
Focused Image: high spatial
resolution sub 1 µm
High dynamic range imager
Inelastic scattering (Compton,nuclear resonance,,,)X-ray photon energy detection
Vector detection
arrival time detection

10

Introduction: What is needed toward SPring-8-II
6 GeVDLSRs

 Faster Shorter exposure while keeping S/N

 Higher DR Higher S/N while keeping the exposure time

• Achieve thesesimultaneouslywhile keeping thebeauty of theexisting detectors.

 High resolution
 Higher pixel count Higher photon energy（>30 keV) Expanding applicable range

R-CCS workshop Feb. 17th, 2020

T. Hatsui, RIKEN

11

CITIUS detector(Charge Integration Type Imaging Unit withhigh-Speed extended-Dynamic-Range Detector)

Main Project Target
for scattering/diffraction experiments

R-CCS workshop Feb. 17th, 2020

12

CITIUS Detector
Development 2013-2020
2.28 Mpixel Camera Head (Mock-up)
Pixel Size
Pixel
numberSensor
Module72.6 µm
0.28 Mpixel/sensor

Peak Signal 1800 phs.@12 keV
Noise0.02 phs.@12 keV

Frame Rate 17.4 kHzSaturation
Count rate 30 (600) Mcps
Detector
SystemPixel Number

Max. 20 Mpixel

Imagine area Max. 32 x 32 cm2Tiling Sensor Modules: Parallel in nature
R-CCS workshop Feb. 17th, 2020

T. Hatsui, RIKEN

13

CITIUS Prototype Camera

K. Ozaki, Y. Honjo,T. Kudo, T. Hatsui

Prototype sensor• Produced in 2018
• 70 kpixels• Capacitive and resistive load
to mimic final 0.28 Mpixels• Data output of 0.28 Mpixels
with dummy pixels

R-CCS workshop Feb. 17th, 202014

High Dynamic Range

100 msIntensity

PhotonIntensity

Conventional Detector Output
Saturation

Intensity

Ideal Output

Intensity

time

Next Gen. Detector
High frame rateOutput→ ideal operation

R-CCS workshop Feb. 17th, 2020

CITIUS detector• 17.4 kfps
• 1800 photons/frame
15

Count Rate
Conventional
Photon Counting detector
EIGER, EIGER2From Dectris web site

40%

10 Mcps

3 McpsR-CCS workshop Feb. 17th, 2020

K. Ozaki, Y. Honjo, T. Hatsui

Next Generation
Charge Integrating detector
CITIUS
Measurement with Proto camera

𝑰 − 𝑶 /𝑰< 0.2-0.5%Target
30 Mcps➡ Equivalent to600 Mcps
in extended mode16

Transmission Image@17.4 kfps

K. Ozaki, Y. Honjo,
T. Kudo, T. Hatsui

Micro Focus X-ray source (Cu target)
CITIUS Proto Camera、17.4 kfps、55 μs exposure/frameSingle Frame @ 17.4 kfps

R-CCS workshop Feb. 17th, 2020

Accumulated Image

17

Energy Resolution demonstration
Data processing optimized for Energy Resolution
400～450 eV FWHM
Cu Kα
Ni Kα

Co Kα
Fe Kα
Cr Kα

Element information can be extracted

R-CCS workshop Feb. 17th, 202018

Spectr- Imaging：Charge∝ Photon Energy
9±1 keV5±1 keV11±1 keV

13±1 keV

Analysis of
15,000 frames
Transmitted Spectrum (～1 second)

01020

30

Output ofAnalysis～100 frames
40

Energy [keV]R-CCS workshop Feb. 17th, 202019

Sub-pixel Imaging: Charge Spread ➡ Sub pixel resolution
Demonstration by SOPHIAS-L

30 µm□ Physical Pixel

R-CCS workshop Feb. 17th, 2020

T. Kudo, T. Matsumoto et.al.,

6 µm□ Virtual Pixel

20

Outline Photon science in 2020s
 Detector Technology: New Trends
 CITIUS detector

 Data Challenge Summary

R-CCS workshop Feb. 17th, 2020

T. Hatsui, RIKEN

21

Detector Output:

Conventional Photon Counting vs. Next Gen.Charge Integrating
Conventional Photon Counting detector

Process partly in digital

Processed in silicon

0.65 0.15

100.15 0.05

00Next Gen. Charge Integrating
0.65 0.15
0.15 0.05

R-CCS workshop Feb. 17th, 2020

Smaller dataat the expenseof loss ofinformation

Process all in digital

Larger data
More information(photon energy, sub-pixel etc.)

22Data Output Rate from the Largest Detector SystemCitius 20 Mpixels＠17.4 kfps

SACLA Computer room

Calibrated data stream
32 bits/pixel, 20 Mpixels, 17,400 frames/s
0.348 Tpixels/s, 11.1 Tbps, 1.4 TB/s
SPring-8 Operation senario
4000 hours/year
30 % data acquisition duty ratio

6.0 EB/year
R-CCS workshop Feb. 17th, 2020

“Raw Data” Storage
not practical

Store Processed Databy edge computing(eg.) “Accumulated” data23

Implementation Example : SOPHIAS 1.9 Mpixel 30 fps
All implemented on CPU, 8 core parallel
Realtime calc. at 30 fps(almost)Pixel-wise Calc.

4x4 pixels Calc.

Input Data: 2 bits flag + 14 bits/pixel
Output Data: 32 bit floating

Pattern matching for 4x4 matrix

T. Matsumoto, T. Kudo, K. Ozaki

Data Reallocation(Event building)
Data from 24 ADCs

Charge at neighboring pixels are summed up

Single Frame data
R-CCS workshop Feb. 17th, 2020

T. Hatsui, RIKEN

24

Performance Estimation

Detector
SOPHIAS
CitiusPixel(Mpixels)

CPU threads

Calc.
Frame Rate (fps)Detector
Frame Rate (fps)Comment

1.9
0.28
0.28

88300

100
679
25,44630 Measurement (*)
17,400 calc.
17,400 calc.

20.16

21,60025,446

17,400 calc.

*) Processor : Intel Xeon E5-1640 v4,
3.6GHz(6 core, 12 thread), 15 MB cache

Edge calculations: significant cost.
Reduces the data size significantly without losing the information.
On-the-fly calculation will be a MUST in photon science.
R-CCS workshop Feb. 17th, 202025

Data Processing and Transfer
An example meeting objective Performance Requirements
CameraHead

Data Archive &Analysis

CITIUS DAQDFB card
DFB card
DFB card
DFB card
DFB card
Disk Cache

Data Bandwidth(Gbps)

9.6 Mpixels 17.4 kfps

2340

R-CCS workshop Feb. 17th, 2020

Edge Computing
17.4 → 0.5 kfps83 PByte/year/system～ 400 PByte/year/5 systems

Duty Ratio of 30%
1544626

Data Processing and Transfer
An example meeting min. Performance Requirements
CameraHead

Data Archive &Analysis

CITIUS DAQDFB card
DFB card
DFB card
DFB card
DFB card
Disk Cache

Data Bandwidth(Gbps)

9.6 Mpixels 5.8 kfps

Edge Computing
5.8 → 0.1 kfps
780

R-CCS workshop Feb. 17th, 2020

16 PByte/year/system～ 80 PByte/year/5 systems

Duty Ratio of 30%
31

9.2

Critical HPC needs for
Data Analysis, Storing andArchiving
27

Data Visualization and Storage

M. Yamaga
Y. JotiEdge Computing near detectors

Cache Storage
4 days ～ 300 TB
On-site Computing Resource

Off-site Computing Res.
CloudThree-way meeting
Feb. 10th, 202028

Data Generation Estimatesat other facilities
 ESRF-EBS of Europe (ESRF)
 Conservative estimate
 2021: 20 PB/year 2022: 30 PB/year 2023: 45 PB/year APS-U of U.S. (Argonne National Laboratory)

 Now around 5-10 PB/year Over 100 PB/year after upgrade in 2024
 50 Pflops at peak on-demand computing reserved for APS
 Priority Area

 Data management and workflow tools Real-time data analysis capabilities On-demand utilization of computing environments
 Data storage and archival

R-CCS workshop
Feb. 17th, 202029

Summary Photon science X-ray delivered to samples will be

2020: ESRF EBS in Europe
2024: APS-U in U.S.
202X: Petra IV in Germany
202X: SPring-8-II in Japanx100-x1000 brighter in coming years
 CITIUS
 Next generation Detector with Integration-type pixel & High-frame
rate detector
 Frame rate 17.4 kfps (standard) Schedule: commissioning starts from fall 2020.
 Jump from conventional counting detectors.
 Detector Technology, and data infrastructure Data rate may reach 6.0 EB/year/system With min. operation scenario, 80 PB/year/5 systems is estimated.
 Proper edge computing + HPC is mandatory
 Compression, Information extraction…

Photon science will soon be an area of data science
R-CCS workshop Feb. 17th, 202030

Acknowledgement
 Members of RIKEN and JASRI

K. Nakajima, T. Tosue, K. Ozaki, Y. Honjo, K. Kobayashi, T. Kudo, Y. Joti,T. Sugimoto, T. Kameshima, Y. Inagaki, K. Fujiwara, M. Yamaga, T. Abe,T. Nakagawa, Y. Oyaki, M. Kimoto, M. Nakamachi
 Glory Mechatronics Ltd.
 Nihon Gijyutsu Center Meisei Electric Co. Ltd.
 Yuji Matsuda (former RIKEN RSC, now at MIST Co,.Ltd）
 Nobukazu Teranishi, Takeo Watanabe, Hiroo Kinoshita (U. of Hyogo）
 Sony Semiconductor Solutions Cooperation
 Tokyo Electron Device Limited
 M. Yabashi, T. Ishikawa（RIKEN）
 Y. Imai, O. Sekizawa, N. Yasuda, K. Sugimoto, T. Uruga（JASRI)
 K. Sato, K. Sano, T. Ueno, M. Kondo, J. Guo, S. Matsuoka (RIKEN R-CCS) MEXT: Government subsidies for large-scale facilities METI/NEDO Fukugo-Keisoku Project（複合計測）
 Feasibility Study Program T. Amemiya（JASRI）、A. Fujiwara（Kwansei Gakuin University）


R-CCS workshop Feb. 17th, 2020

Thank you for your attention
T. Hatsui, RIKEN

31