The Fault Diagnosis of Event
Timing System in SuperKEKB
D. Wang, K. Furukawa, M. Satoh, H. Kaji, H. Sugimura,
T. Kudou, S. Kusano, Y. Iitsuka
KEKTuesday, 08 October 2019, ICALEPCS

1

Outline• Event Timing System in SuperKEKB• Event Fault Diagnosis System• Data Acquisition• Data Processing• Fault Diagnosis• Beam Mode Replacement• Redundant Beam Mode

• Conclusion and Outlook
Tuesday, 08 October 2019, ICALEPCS

2

Event Timing System in SKEKBSuperKEKB HER/LER

PF-AR

Photon Factory (PF)Damping Ring (DR)
LINAC
Tuesday, 08 October 2019, ICALEPCS

3Event Timing System in SKEKBRF clockreference

Mains voltage
reference

8 bitscode0

Event delay

0xAA

1

Event delay

0xBB

2

Event delay

0xCC

2047

Event delay

Trigger Source

Event
Generator
fiber

SoftwareSequencer

fanoutEvent code
EventReceiver

Event
Receiver……

Event
Receiver
EPICS
Channel Access

Kicker

SeptumPulsedMagnetBasic structure of Event Timing System
Tuesday, 08 October 2019, ICALEPCS

IOC
EVG

commitHardwareSequencer……

32 bits timestamp

RAMaddress

MRF
Event Link

IOC
EVR

0xEE
OPIIOCEVRIOC
EVR

4Event Timing System in SKEKB• Simultaneous injection to 4 rings !
• High precision: jitter less than 30 ps for MR, 300/700
ps for PF/PF-AR !
• Every pulse synchronized with 50 Hz AC power
supply !

• MRF’s series Event Generator and Receivers• VME-EVG-230: 4• VME-EVR-230-RF: 50• PXI-EVR-300 : 20

• 50 Hz beam mode repetition rate
• 114.24 MHz event rate
• 12 kinds of beam mode
• About 120 event codes are defined• 11 or 12 events every pulse• EPICS R3.14.12 with mrfioc2 (device support)
Tuesday, 08 October 2019, ICALEPCS

5

Event Timing System in SKEKB• Common frequency between 2856 MHzand 508.9 MHz is 10.38 MHz (96 ns, 49buckets duration)• Chance of injection timing turns up once per
96ns (49 buckets).

• Requiring two bucket selection timings(injection and extraction at DR, two EVGs
are needed)• The number of combination is 5120 × 23
(least common multiple of DR and MR)• Can not coincide with AC50 every pulse

Bucket Selection

MR 508.9 MHz

96 ns (49 buckets)
DR 508.9 MHz

5120
buckets
230
buckets
LINAC 2856 MHz

DR + MR : 5120 × 23 × 96 = 11.34 ms

MR only : 5120 × 96= 493 µs

16/18 sequences injection, details are talked in
H. Kaji et al. “ BUCKET SELECTION SYSTEM FOR SuperKEKB”, PASJ 2015 THP100
H. Kaji et al. “INSTALLATION AND COMMISSIONING OF NEW EVENT TIMINGSYSTEM FOR SuperKEKB”, PASJ 2015 FROL15

Tuesday, 08 October 2019, ICALEPCS

6Event Timing System in SKEKBKEKB rev/49/23/2

KEKB rev/49/23/2
Set EVGAC Power

20 ms

Upper-EVG 16/18 sequence
20 ms

20 ms

20 ms

Time toDigitalConverter

EVG

EVGEVR
Software Sequencer
Hardware Sequencer

Tuesday, 08 October 2019, ICALEPCSAC50BSdelayC, pre A

Set EVG

Set EVGPF Trigger(PF-AR)

TDC Start

TDC

DelayPF Trigger

AC50
TDC Stop

EVR

Set EVG

TTL

Set EVG

NIM

Next beammode event(pre event*)

AC50

Current beammode event

50 Hz
generator

AC50

EVG
AC50

• *Pre event is necessary
because of the kicker
charging time

Real beam
trigger time

CA

B

C

A, pre B

B, pre C

C, pre AA, pre B

C, pre AA, pre B

B, pre C

C, pre AA, pre B

7

Outline• Event Timing System in SuperKEKB• Event Fault Diagnosis System• Data Acquisition• Data Processing• Fault Diagnosis• Beam Mode Replacement• Redundant Beam Mode

• Conclusion and Outlook
Tuesday, 08 October 2019, ICALEPCS

8

Data Acquisition• Requirements:••

Event code and nanosecond-level timestamp is needed
Two EVRs in one VME to monitor two lower-EVGs.

• Obstacles using Channel Access• Some data lost due to high CPU usage•

EPICS period Scan task delay in VxWorks IOC is very high

• New approach Implementation:•••

An EPICS thread with a large size ring buffer is createdFetch data and timestamp from EVR FIFO memoryTransmit to NFS server using binary format

CACAClientServer
ModifiedWaveform Record IOCSTATS
Mrfioc2 Device
Support

Network File
System protocol

NFS

Channel Access

CA Client(Python, Java)

Hard Disk

Modified
Mrfioc2 Device
SupportHard Disk

EVR

EVR
Tuesday, 08 October 2019, ICALEPCS

9Data Processing• Using Python script check beam modeinterval and event code
• Abnormal data is extracted• Results are sent by email• All actions automatically run every day

Optional, manually checkdetailed data
Tuesday, 08 October 2019, ICALEPCS

10Outline• Event Timing System in SuperKEKB• Event Fault Diagnosis System• Data Acquisition• Data Processing• Fault Diagnosis• Beam Mode Replacement• Redundant Beam Mode

• Conclusion and Outlook
Tuesday, 08 October 2019, ICALEPCS

11

Fault Diagnosis
Excerpt of the abnormal

Beam mode repeat

Abnormal PF beam mode interval
Tuesday, 08 October 2019, ICALEPCS

12

Beam Mode Replacement
Upper-EVG 16/18 sequence
KEKB rev/49/23/2

Current beammode event

AC50

Next beammode event
DelaySet EVG• Bucket Selection delay increases• ‘Set EVG’ signal comes late than pre event• Current beam mode is replaced by the next beammode• Happened 15 times during 2-weeks operation in June

Set EVG• AC power fluctuation• Alternating during 16/18 sequences injection

AC50• AC50 comes very late

BS delayC, pre A

C, pre AA, pre B

B, pre C

B, pre C

C, pre A

Tuesday, 08 October 2019, ICALEPCS

C, pre A

C, pre AA, pre B

A, pre B

Lower-EVG
Trigger

SoftwareSequencer
HardwareSequencer

C, pre A

CAA, pre B

B, pre C

C, pre A

B, pre C

13Redundant Beam Mode

20 msCCA

ABB

15 ms

Tuesday, 08 October 2019, ICALEPCSCCC, pre AA, pre B
C, pre A?

B, pre CA, pre B

B,pre CSet EVG

Set EVG

Delay

BS delay
Lower-EVG
Trigger

SoftwareSequencer
HardwareSequencer

AC50

Next beammode event

20 ms

B

AC50

Current beammode event

Set EVGPF Trigger

20 msUpper-EVG 16/18 sequence
KEKB rev/49/23/2

AC50• Abnormal value of the calculation of bucket
selection of positron in DR and MR• A positron beam mode comes about 5 msearlier than normal• Sequencer trigger source changed• PF Trigger signal is later than ‘Set EVG’
• Happened 10 times during 2-weeks operationin June

BS delayC, pre A
B, pre C14

Outline• Event Timing System in SuperKEKB• Event Fault Diagnosis System• Data Acquisition• Data Processing• Fault Diagnosis• Beam Mode Replacement• Redundant Beam Mode

• Conclusion and Outlook
Tuesday, 08 October 2019, ICALEPCS

15

Conclusion and Outlook• Solution of beam mode replacement:• Upgrade the 50 Hz generator module• Decrease reference interval of AC50 (Already done)• Solution of redundant beam mode:• Temporarily separate the positron beam mode and PF mode
• Figure out the reason of abnormal bucket selection delay• Much more data required to diagnose the bucket selectionprogram• Near future: On-line alarm system• Furthermost, a fault prediction system based on time series
forecasting models or deep learning

Tuesday, 08 October 2019, ICALEPCS16

• Di. Wang
• sdcswd@post.kek.jp• KEK Linac

Tuesday, 08 October 2019, ICALEPCS

Thank You17