THE FAULT DIAGNOSIS OF EVENT TIMING SYSTEM IN SuperKEKB∗
D. Wang1,†, K. Furukawa, M. Satoh, H. Kaji, H. Sugimura, KEK, Japan
Y. Iitsuka, East Japan Institute of Technology Co., Ltd, JapanT. Kudou, S. Kusano, Mitsubishi Electric System & Service Co., Ltd, Japan
1 also at SOKENDAI, Japan
Abstract
The new MRF event timing system is one of the mostimportant components to maintain the reliable and stable
operation of the SuperKEKB project. This system is utilized to distribute high precision level timing signals andaccompanying control instructions to synchronize different
subsystems and machines. Event generator (EVG) generates
signals of different beam modes every 50-Hz pulse whichcontains several event codes while Event receivers (EVR)
receives them and output signals to dedicated devices allover the installation. To certain these events are consistentduring the distribution, an event fault diagnosis system isessentially needed. An EVR based event timing diagnostic
system is thus developed by modifying the driver support
module to provide a log system of persistent event data as
well as comparing the received event codes with the beam
injector pattern, detecting the event timing interval fault
and notifying the results by email every day. Then, we areable to locate the fault, analyze the data, fix bugs or replace
hardware and resume accelerator operation quickly.

INTRODUCTION
The SuperKEKB is an electron/positron collider upgradedfrom the KEKB project since 2010 at KEK, whose scientifictarget is to update the world highest luminosity record anddiscover new particle physics by Belle II experiment. The
7 GeV electrons and 4 GeV positrons are injected into different main rings (MRs) called high energy ring (HER) andlow energy ring (LER), respectively. During the phase-2
operation in 2018, a newly constructed 1.1 GeV Damping
Ring (DR) at the middle of injector linac (LINAC) aiming
to lower the positron emittance was deployed. One of the
major challenges in the phase-3 operation is the upgrade oftiming system of SuperKEKB [1]. The 700-meter LINAC
delivers trigger signals to five rings, SuperKEKB HER/LER,
two light source ring PF and PF-AR and a positron DR. The
DR timing signal must synchronize with the MR signal.
For switching the beam modes swiftly, a large quantities
data like bucket selection delay time, beam mode event codesand pulse number is required to update rapidly. Two sets oftiming signals should be conveyed because the condition ofbeam line before DR and after DR is dissimilar. Therefore
we make use of three EVGs and some other modules toconstruct our timing delivery system. With the growth of
the system complexity, it is important to diagnose the timing
system and ensure the correctness of the signal delivery.
∗ Work supported China Scholarship Council† sdcswd@post.kek.jp

Hence a fault diagnosis system is developed to assist us tomonitor the timing event information.
In this paper, some efforts to improve the reliability andstability of timing system will be introduced.

TIMING SYSTEM ARCHITECTURE
At an accelerator facility, the function of timing system isto provide synchronized trigger signal to all relevant components and devices which locates at LINAC, beam transmission line, injection and extraction system and beam monitor
system. The precision of synchronization depends on theoperation requirement as well as the size of the installation [2]. In our case, a jitter of less than 30 ps is required inSuperKEKB MR and 300/700 ps in PF/PF-AR.

MRF Event Timing
We introduced the event timing system produced by the
MRF company to meet our requirements [3]. The products
we choose for our main timing system are “Event Generator(VME-EVG-230)” and “Event Receiver (VME-EVR-230RF)”. At SuperKEKB, the event clock rate is 114.24 MHz
and hence the minimal event code interval is around 9
nanosecond. Every 20 ms, depending on the beam mode, 11
or 12 events are generated and distributed to more than 60
EVRs all over the SuperKEKB.
The EVG uses optical fiber to transmit a 16-bit word by
8b/10b encoding. Inside this 16-bit word, 8-bit distributed
bus running in parallel and independent of the other 8-bit
allows distribution of timing signals updated with the event
clock rate. Almost all 256 event codes except for some special functions can be defined by users. The EVG distributes
signal by fanout Units to an array of EVRs. Each EVR isable to generate pulse with an associated delay and width todevices. It must be emphasized that the MRF timing system
is well integrated with EPICS system with the help of mrfioc2 device support module [4]. Figure 1 is the scheme of
the MRF timing system.

Figure 1: The MRF event system.

The EVR decodes the optical signal and writes the eventcode into a 32-bit wide FIFO memory with attached timestamp information. This FIFO memory can hold up to 512
events and timestamps and each pair consists of an 8-bit
event code and a 24-bit timestamp register value. The operation of an event writing to FIFO will trigger a VME interrupt
request.
The hardware sequencer RAM is a vital units in EVG
which provides a method of transmitting or playing backsequences of events with defined delay time. The users areallowed to change the event codes, delay timestamp andsequencer trigger source but these changes are only temporarily saved in the software sequencer. When users are
satisfied with the changes then they can use ‘commit’ action
to update the changes to the hardware sequencer [5].

Pulse to Pulse Modulation
The SuperKEKB LINAC is operated via event-based,
global, and synchronized controls to inject beams with different parameters into four separate storage rings simultaneously. One of the crucial technologies used during theoperation is Pulse-to-Pulse Modulation (PPM) [6]. A single
LINAC would behave as four independent virtual accelerators with hundreds of independent parameters modulated
pulse-by-pulse at 50 Hz (Fig. 2).

Figure 2: A single linac behaves as four virtual accelerators
to inject their beams into four separate storage rings.
Since each ring requires different injection energy, magnetic field in the injector linac has to be changed pulse bypulse according to the characteristic of the beam [7]. Bymeans of PPM simultaneous, top-up injection to several
separated rings could be accomplished. Every pulse corresponds to a beam mode while every beam mode contains
several event codes [8,9].

The operator can set the beam mode sequence according
to requirements of rings and the EVG will generator eventcodes sequence as well as sending to the EVR. Each eventcode has its own meaning, either for triggering particularcomponent or indicating the pre-trigger signal of the next
beam mode. The pre-trigger signal is extremely importantbecause some critical devices like kicker magnet requires
some charging time to work properly [10].

System Set-up of Main Timing Station
The Main Timing Station is the core part of our Event
Timing System. Figure 3 is the schematic views of the Main
Timing Station. Three EVGs and one EVR are constructed
as three-layers system. One EVG (upper-EVG) is configured as the main events trigger while the other two EVGs
(lower-EVG) in the downstream will receive signal frommiddle EVR and generate actual event codes to control substations. These modules are placed in the VME64x sub-rack.
The lower-EVGs are utilized owing to the requirements of
the Damping Ring operation. One lower-EVG controls the
LINAC before the DR (upstream LINAC) while the othermanages the beam after DR (downstream LINAC).

Figure 3: The Main Timing Station consists of three EVGs,
one EVR, a TDC, two fanout modules and a distributed
shared memory module.
Besides, the time-to-digital converter (TDC) and the distributed shared memory are configured in the IOC [11].
Since the LINAC hardware must run with the same phaseof AC power 50 Hz (AC50) to maintain stable operation, theTDC module was installed for monitoring AC50 and synchronizing the MR injection operation. A 50 Hz generator
module whose input is the AC power supply would generate
the AC50 NIM standard signal.

Because of the restriction of the MR revolution frequency,
the “rough” synchronization system for the MR injection isdeveloped to reach our demand and here, for convenience, wecall this “rough” synchronization system “16/18 sequences
injection”. The detail of this algorithm is described in thispaper [1]. With the help of “16/18 sequences injection”,
every pulse is synchronized with AC50 despite of the fluctuation of AC50. The distributed shared memory is used
for fetching the delay value after the calculation of bucket
selection in another IOC and setting to the lower-EVG.

Trigger Line
Figure 4 show the trigger line of the Main Timing Station IOC. The upper-EVG receives MR revolution frequency
signal to trigger event sequence and current beam mode
event will be sent slightly earlier than next beam mode event(we call it ‘pre-trigger’). This pre-trigger signal causes twolower-EVGs perform PPM operation which changes software sequencer event code and timestamp to next pulse’s
value. After a fixed delay time, current beam mode event
will trigger the two lower-EVGs’s hardware sequencer tosend out actual event codes to sub-stations. The delay timeof two lower-EVGs depends on the measurement value ofthe AC50 through TDC. IOC reads the TDC value and set
back to lower-EVGs hardware sequencer by EPICS record.
Consequently, every pulse beam is always synchronized with
the AC50 by means of this technology.

sync module which receives PF (PF/AR) revolution frequency, event clock and AC50 as input.

EVENT TIMING DIAGNOSTIC SYSTEM
Theoretically, the timing system in SuperKEKB must operate periodically in 50 Hz, many devices like power supply,
pulsed magnets and beam position monitors would be triggered every 20 ms for assuring that bunches are accelerated
at relative the same location. And only 2 ms width of timing
fluctuation is accepted [12]. However, during the operation we noticed that the time interval between kicker charge
trigger and beam trigger is abnormal. And that may cause unstable beam and if beam bunch is not correctly transferred, inthe worst situation, the beam would hit the Belle II detector
and damage it. Besides this, lack of a reliability monitoring
system makes it difficult to evaluate whether correct eventcodes are received by devices or not.
Apart from that, during the 21st KEKB Accelerator Review Committee in 2016, it was suggested that an operational
diagnostic program of event system is needed. The collection of the event codes and timestamp information received
in EVR is beneficial to diagnose abnormal events and avoid
erroneously trigger.

Data Acquisition
In order to improve the stability and maintenance of timing
system, an EVR based fault diagnosis system is developed torecord all the events information. The MVME5500 on-board
CPU controller and VxWorks-6.8.3 is used for developing
this system. Two EVRs are installed in the VME rack tomonitor upstream LINAC and downstream LINAC timing
signals separately. A straightforward method to record all
event codes used in injector linac is to create a number of
EPICS PVs and monitor all of them using Channel Access.
After practical tests, we found that the VME CPU is not fastenough to handle nanosecond level Channel Access datatransmission and many event data would lost. Concerning
this, a substitution data transmission approach, Network File
System (NFS) protocol, is adopted. Figure 5 is the pictureof the fault diagnosis system.

Figure 4: The trigger lines of the Main Timing Station.
To guarantee the beam trigger signals are delivered todevices, we install an EVR in the downstream of lower-EVG
to receive the real trigger signal and use its output as a ‘Set
EVG’ signal to update the yet changed software sequencer
value to the two lower-EVGs’s hardware sequencer. In next
pulse, this hardware sequencer would be triggered by next
pulse’s current beam mode event.
For the beam transferring to PF (PF/AR) ring, a special
case is that the hardware sequencer trigger source is PF(PF/AR) trigger signal instead of the current beam mode
event. The PF (PF/AR) trigger signal is the output of triple-

The NFS is a distributed file system protocol, allowing
a user on a client computer to access files over a computer
network much like local storage is accessed [13]. The sourcecode of mrfioc2 module is modified so that an EPICS threadis created aiming for directly write event code and its timestamp information to local hard disk. Inside this EPICS
thread, a large size ring buffer is created to buffer the eventcode and timestamp from the EVR FIFO area. Under thiscircumstance, one thread is used to receive the event codefrom EVR hardware and the other thread buffers them andsend them by NFS protocol. Since we utilize the low-levelsystem call, the CPU load is dramatically decreased and thissystem is very stable.

time interval between some beam modes are abnormal. After detailed analysis, we categorize these data as two main
abnormal conditions. Both of them will be thoroughly explained.

Figure 5: Two EVRs are installed in VME crate to record
all the event codes and timestamp from upstream LINAC
and downstream LINAC, respectively.

Data Processing
Every day hundreds of megabytes data will be saved tohard disk. Noted that the binary format file rather than
ASCII format is chose to save the hard drive space andreduce the network transmission burden. To check thesedata we developed a program using Python script to identify
the abnormal conditions. The sequence of event code ischecked as well as the time interval of continuous beammode. The time interval threshold is set to eliminate systemerror and distortion. The results of diagnosis are sent out by
email to notify the control group person as well as saved tothe hard disk. This Python script automatically runs every
day.

Figure 7: Too long delay of AC50 cause the ‘Set EVG’ signal
arrives late than next beam mode event.

Beam Mode Replacement
This kind of abnormal condition can be described as current beam mode is replaced by the next beam mode. As theFig. 7 shows, owing to the drastically drift of AC50 and thechange of “16/18 sequences injection”, The AC50 signal
comes very late and thus ’Set EVG’ signal is later than next
beam mode’s pre-trigger which means hardware sequencer’s
value is changed to subsequently beam mode. The current
beam mode is substituted by next beam mode. Based on our
log data, such situation might occur when AC50 drift value
is larger than 1 ms every 16/18 pulses.

Redundant Beam Mode

Figure 6: The abnormal beam mode occurrence times during
two weeks before the summer maintenance of SuperKEKB
in 2019.
Figure 6 is the picture of detected abnormal event sequences including repeat beam mode and unusual beammode time intervals. The reason of such abnormal situations
will be explained later.

FAULT DIAGNOSISBased on the analysis of the statistic data. We found thatsome beam modes are wrongly triggered twice as well as

When we checked the timestamp information of the log
data, sometimes an extra beam mode is inserted between
a positron mode and a PF (PF/AR) beam mode. As theFig. 8 shows, if a positron beam mode arrives earlier, withthe trigger of lower-EVGs’s sequencer, the event codes aresent to the devices which causes a ‘Set EVG’ signal. This
‘Set EVG’ signal will change the sequencer trigger source to‘PF Trigger’ and sequencer is triggered again later when ‘PF
Trigger’ signal comes. Thus, an extra PF beam mode is sent.
The unexpected delay value of the positron beam mode is
the direct reason of this kind of error.

Solutions
To solve the beam mode replacement problem several
actions could be took. Firstly, since the original reason isthe drastic fluctuation of AC50, a filter could be used during
the transformer process in 50 Hz generator to stabilize theAC50. Besides that, the AC50 comes too later is the directoperators. Furthermore, by using the event timing data andAC50 signal, we plan to develop a fault prediction system
based on time series forecasting models or deep learningalgorithms. Later developments will be investigated basedon the achievements.

REFERENCES[1] H. Kaji et al., “Installation and commissioning of new Event
Timing System for SuperKEKB”, in Proceedings of 12th Annual Meeting of PASJ, Tsuruga, Japan, 2015, paper FROL15.
[2] T. Korhonen, “Review of Accelerator Timing Systems”, in
Proc. 7th Int. Conf. on Accelerator and Large Experimental
Control Systems (ICALEPCS’99), Trieste, Italy, Oct. 1999,
paper FB1I01, pp. 167–170.

Figure 8: Owing to the wrong value of bucket selection
delay, a beam mode signal is trigger earlier and results tothe wrong occurrence of the next beam mode.
cause, so the interval between lower-EVG delay time andAC50 could be decreased by changing the ‘16/18 sequences
injection’ setting.
The change of hardware sequencer trigger source makes
it possible that PF (PF/AR) beam mode is triggered twice
if positron beam mode arrives earlier than usual. Thus, theredundant beam mode error can be temporarily avoided byseparating positron mode and PF (PF/AR) mode. Much
more data is required to figure out the reason of the positron
beam mode bucket selection abnormal delay value.

SUMMARY AND OUTLOOKThe stability of timing system is crucial system to maintain a steady operation of our project. The introduction of
PPM technology and construction of the DR make it much
more complicated to diagnose the potential fault. Thus, wedevelop an event fault diagnosis system to monitor the timing
system signals and make fault analysis every day.
During the SuperKEKB phase-3 operation, we find several timing system abnormal conditions through our fault diagnosis system and after detailed inspection we find the flawof our timing system in some extreme situations and correspondent suggestions based on the original cause and directcause are proposed to improve the system. After the summer maintenance until the middle of October, SuperKEKB
will start again and much more timing system data could beacquired by our fault diagnosis system which contributes tofigure out deepest reason of abnormal beam mode. We believe this kind fault diagnosis system is of great significancenot only for our timing system but also for other facility’s
timing system.
Depending on the fault analyses, in the near future, wewould implement a mechanism for on-line alarming to thewebsite,[3] MRFtiming-system

http://mrf.fi/index.php/[4] Mrfioc2 website,
mrfioc2

http://epics.sourceforge.net/[5] EPICS device driver for MRF VME-EVG-230, Sep. 2011,
pp. 6-9; http://epics.sourceforge.net/mrfioc2/
evg-usage.pdf[6] K. Akai et al., “SuperKEKB collider”. doi:10.1016/j.
nima.2018.08.017[7] Y. Enomoto et al., “Pulse-to-pulse Beam Modulation for 4
Storage Rings with 64 Pulsed Magnets”, in Proc. 29th Linear
Accelerator Conf. (LINAC’18), Beijing, China, Sep. 2018, pp.
609–614. doi:10.18429/JACoW-LINAC2018-WE1A06[8] K. Furukawa et al., “Pulse-to-pulse Beam Modulation andEvent-based Beam Feedback Systems at KEKB Linac”, in
Proc. 1st Int. Particle Accelerator Conf. (IPAC’10), Kyoto,
Japan, May 2010, paper TUOCMH01, pp. 1271–1273.
[9] N. Higashi et al., “Construction and Commissioning of Direct
Beam Transport Line for PF-AR”, in Proc. 8th Int. Particle
Accelerator Conf. (IPAC’17), Copenhagen, Denmark, May
2017, paper WEPAB044, pp. 2678–2680.
[10] H. Sugimura et al., “Synchronized Timing and Control System Construction of SuperKEKB Positron Damping Ring”, in Proc. 16th Int. Conf. on Accelerator
and Large Experimental Control Systems (ICALEPCS’17),
Barcelona, Spain, Oct. 2017, pp. 229–231. doi:10.18429/
JACoW-ICALEPCS2017-TUCPL02[11] T. Suwada et al., “Wide dynamic range FPGA-based TDC
for monitoring a trigger timing distribution system in linear
accelerators”, Nucl. Instr. Meth., vol. 786, pp. 83–90, 2015.
doi:10.1016/j.nima.2015.03.019.
[12] H. Kaji et al., “Bucket selection system for SuperKEKB”, inProceedings of 12th Annual Meeting of PASJ, Tsuruga, Japan,
2015, paper THP100.
[13] vxworks kernel programmers guide 6.8, Wind
River, Alameda, CA, USA, Nov. 2009, pp. 433-440.
http://read.pudn.com/downloads305/sourcecode/
embedded/1357999/vxworks_application_
programmers_guide_6.8.pdf