Chapter 13
Accelerator Control System
13.1

Injection Control

13.1.1

Overview

The injection control is implemented by synchronously operating the various hardware at injector linac (LINAC), beam transport (BT), and main rings (MRs). The
system is robust with respect to the standard control system based on the EPICS
network[1] since it utilizes two kinds of dedicated optical networks.
One of the injection control networks is Event Timing System[2]. It delivers
timing-triggers to individual hardware along the beamline. The pulsed-operationsof beamline hardware are implemented synchronously so that the beam pulse canbe transported from electron guns to injection points at MRs.
The other network is Bucket Selection[3]. It selects the RF-bucket at MR
which the next beam-pulse is injected (injection-bucket). Then it calculates theappropriate timing of LINAC operation and provides to Event Timing System.
This process is implemented on pulse-by-pulse in 50 Hz.

13.1.2

Event Timing System

Star-topology network
The Event Timing System network is configured as the star-topology. Figure 13.1
shows its schematic view. The Event Timing modules are placed on the centerand idividual edges. The modules on the center and edges are master module andslave modules, respectively. The master module controls all slave modules while
the slave modules control the individual hardware along the beamline.
More than 30 slave modules are installed along the beamlines of LINAC, BT,
and MR. They control the accelerator hardware synchronously with the Event
650

Figure 13.1: Schematic view of Event network: one master module is installedon the center of network. It provides Events indicating the timing of LINAC
operation toward all slave modules at individual edges. The timing-triggers areprovided from the slave modules to individual hardware at the beamline.
Timing System functions.
Event Generator and Event Receiver
There are two kinds of Event Timing modules, which are the Event Generator(EVG) and Event Receiver (EVR). The EVG module connects with EVR modules
via the optical cables. They perform one way communication from EVG to EVR.
In general, the EVG is installed at the center of network as the master module while
the EVRs are installed at the individual edges of network as the slave modules.
Both EVG and EVR are the module beased on the FPGA and SFP connector.
We mostly utilze VME-EVG-230 and VME-EVG-230RF[4] for EVG and EVR, respectively. Also some modules developed at SINAP (Shanghai Institute of Applied
Physics) are utilized at the MR region.
The EVG delivers two byte data packet called “Event” towards the EVRs.
The half (one byte) of Event is the Event-code which is utilized to distinguish thekind of Event. There are 256 kinds of Events. The EVG controls the EVRs by
managing the Event-code and delivery timing.
The EVR has two functions, such as the production of timing-trigger andthe CPU interruption. The EVR function is implemented when it receives the
Events. The arbitral functions can be programmed on the individual Event-codes.
In SuperKEKB, we classify two kinds of Events. One is “timing-Event”. The
EVR provides timing-trigger when it receives the timing-Event. The other is“preparation-Event”. The EVR launchs the CPU interruption to the CPU module651

on the same VME-bus.
By managing the delivery of timing-Events and preparation-Events, the EVG
can control the beamline hardware via the EVR’s reactions.
Pulse-to-Pulse Modulation
LINAC provides quite different four kinds of beam-pulses towards four rings at
KEK. The particle, energy, charge, and direction of beam-pulse at LINAC aresummarized in Table 13.1. We perform the top-up filling operation to four rings
simultaneously. It is one of the specific features of LINAC.
Table 13.1: Beam-pulse at Linac: LINAC provides the beam-pulses toward SuperKEKB MRs and two light sources such as Photon Factory (PF) and PF Advanced Ring (PF-AR). Their beam types are summarized.
Direction Particle Energy (GeV)
LER
HER
PF
PF-AR

PositronElectronElectronElectron

4.0
7.0
2.5
6.5

Charge (nC)
4.0
5.0
0.2
5.0

We develop the injection control system which can switch the beam direction,
on pulse-by-pulse, in 50 Hz. More than 200 parameters of LINAC hardware must
be changed when we change the beam-type. The injection control system switches
those parameters synchronously in 50 Hz. It is called Pulse-to-Pulse Modulation(PPM).

Figure 13.2: Example of Event handling for Pulse-to-Pulse Modulation: The 50 Hz
Event delivery is implemented within the 2 ms of time windows which come every
20 ms. One set of timing-Event and preparation-Event are delivered from EVG ineach pulse.
The Event Timing System functions are fully utilized for implementing PPM
and realizing the simultaneous top-up filling operations for 4 rings. Figure 13.2 is652

the schematic view of Event delivery. One set of timing-Event and preparationEvent is delivered from EVG for every LINAC operation in 50 Hz. The beam
operation is carried out with the timing-triggers generated with the timing-Event.
Just after beam operation, typically 1 ms after, the CPU interruption is launchedwith the preparation-Event. The LINAC parameters are switched to be those forthe beam-type in the next injection. The time interval from the CPU interruption
to the next beam operation is at least 17 ms. Therefore, the enough time for the
preparation of next injection is reserved.
The preparation-Event and timing-Event are defined for individual beam-types.
The beam-type of preparation-Event is same as that of the timing-Event in thenext pulse. Note the beam-types of timing-Event and preparation-Event in one
set are not always same.
The parameters for all LINAC hardware are switched synchronously since all
EVRs receive same preparation-Event at the same time.
Main Timing Station
The followings are required to the timing-triggers for the beam injection at SuperKEKB.
1. The 50 Hz Event delivery of EVG is implemented by the 2254 times divided
MR revolution (rev2254).
2. The arbitrary delay can be added to the timing of Event delivery for Bucket
Selection and it can be changed on pulse-by-pulse in 50 Hz.
3. The LINAC beamline before DR (1st LINAC) can inject positron-pulse into
DR. That after DR (2nd LINAC) can extract positron-pulse from DR andtransport it into MR. The 1st LINAC and 2nd LINAC can separately carryout these operations.
4. One entire process of positrons injection becomes longer than LINAC pulse
period of 20 ms since we require at least 40 ms of damping time at DR.
Therefore the above mentioned injection and extraction processes at DR
must be implemented in the different LINAC pulses.
5. The EVG accepts additional two inputs and they are utilized for the injections to two light sources (PF and PF-AR). Three inputs to implement the
EVG operation can be switched on pulse-by-pulse in 50Hz.
6. The LINAC operation is implemented on the same phase of AC power supply
50 Hz (AC50) with the time window of 2 ms.

653

Figure 13.3: Picture of MTS (left) and schematic view of its module configuration(right): the EPICS IOC is built with the CPU, three EVGs, EVR, TDC, anddistributed shared memory modules. The Event modules configures three layers.
We configure the Main Timing System (MTS) to realize the Event delivery
which satisfies all above requirements. Figure 13.3 shows the picture of MTS
and schematic view of its configuration. MTS consists of three EVGs and one
EVR. They are installed on the same VME-bus and the EPICS Input/Output
Controller (IOC) is built with the CPU module. In terms of trigger line, Event
modules configure three layers.
There is one EVG at the upper-layer (upper-EVG). It is the most upstream oftrigger line. The upper-EVG delivers Events to the EVR installed at the middlelayer (middle-EVR). The middle-EVR provides three important timing-triggers.
Two of them are delivered to the two EVGs at the lower-layer (lower-EVGs). The
remaining one is utilized as the start signal of TDC measurement. The lower-EVGs
provide Events toward the EVRs installed in the beamline.
Figure 13.4 shows the time chart of MTS operation. The Event delivery of
upper-EVG is implemented by the input trigger of rev2254. The upper-EVG
provides the timing-Event in 50 Hz with its Event delivery sequence. The 16 or
18 timing-Events are provided in each input signal. The sequence lengths in bothcases become 320 ms and 360 ms and longer than one LINAC pulse period of 20 ms.
The 50 Hz operations of middle-EVR and lower-EVGs are implemented with this654

Figure 13.4: Time chart of MTS operation: the upper-EVG provides Events tomiddle-EVR. The middle-EVR provides TTL timing-triggers to lower-EVGs. The
lower-EVGs provide Events to EVRs installed along beamline. The upper-EVG istriggered with the 2254 times divided MR revolution and provides Events in 50 Hz.
These Event deliveries becomes source of LINAC 50 Hz operation. The timingtriggers for DR injection and extraction are provided with different Event pulses.
However their time synchronization is guaranteed since the processes providing
those Event pulses are originated from the same upper-EVG sequence.
50 Hz Event delivery. Therefore it satisfies the requirement item #1.
The arbitrary delay which is described in the requirement item #2 is added on
the two lower-EVGs. All input triggers from middle-EVR are delivered with thesource of rev2254 so that the timing of injection-bucket can be realized by adding
the appropriate delay to the Event delivery of lower-EVGs. Of course, this delay
can be changed in 50 Hz. Therefore the requirements from Bucket Selection aresatisfied. Note, the timing of LINAC operation can be adjusted within 2 ms time
window. This time window is constraint from the specification of LINAC hardware
such as the klystron.
The requirement items #3 and #4 are satisfied in the following way. One oflower-EVGs provides Events toward EVRs installed at 1st LINAC while the otherprovides Events toward those at 2nd LINAC. Therefore the different operations canbe implemented at two beamline regions by applying the different Event deliveries
to lower-EVGs. The Event deliveries for the injection and extraction of positrons
at DR are implemented with the different LINAC pulses even though they are in asingle injection process. However, as shown in Figure 13.4 (b), the input triggers
for two lower-EVG processes are generated in the same upper-EVG sequence.
Therefore the time synchronization between injection and extraction of positrons
is realized.
655

The VME-EVG-230 module has three TTL-level input channels. One of themcan be utilized as the input channel to implement the Event delivery sequence. It
can be switched in every pulse. Therefore the requirement item #5 is satisfied.
As described in the requirement item #6, the LINAC hardware must be operated on the same phase of AC50. The AC50 does not have perfectly equivalentintervals and the timing is drifted as shown in Figure 13.5. It is the side effect when the power company coordinates electricity. The upper-EVG adjusts the
50 Hz Event delivery and follows the opration phase of AC50 by using two sequencelengths. It is demostrated in Figure 13.6. The next trigger of rev2254 signal after
16 triggers sequence comes 2.504 ms earlier than usual 50 Hz timing while thatafter 18 triggers sequence comes 2.852 ms later. Therefore the timing of Event delivery can be intendedly adjusted to the drifted AC50 phase by using these timing
difference. The 16 (18) triggers sequence is selected and implemented when the
AC50 phase become earlier (later). This process is automatically inplemented bymonitoring the AC50 phase with the TDC.

Figure 13.5: Measurement of AC50 phase: the timing of operation phase of AC50
is measured with TDC. The precise 50 Hz triggers are utilized as the start signal
for the TDC measurement. The measurements are repeated during 1000 seconds.
The vertical axis is time interval from TDC start to the operation phase of AC50.
The timing of operation phase is considerably drifted as compared with the precise
50 Hz.
There is the TDC module[6] which monitors the AC50 phase. It is installed
at MTS and receives the start triggers from EVR. Therefore the timing relation
between the input trigger of lower-EVGs and AC50 can be monitored in every
pulse. The sequence length of upper-EVG is decided with this information.

656

Figure 13.6: Comparion of cycles among two upper-EVG sequences and rev2254:
the rev2254 is in 44.1 Hz and its cycle is 22.68 ms. The next rev2254 signal comes
17.496 ms (22.852 ms) after the last Event delivery of 16 (18) triggers sequence.
After 16 injection sequence, the timing of all Events in the next sequence is shiftedto be 2.504 ms earlier than usual 50 Hz. It happens to the other direction after
18 injection sequence and the timing is shifted to be 2.854 ms later. These timing
shifts can be utilized for following up the drifted timing of AC50 phase.
Main Ring Sub-timing Station
MR Sub-timing Station (MR-STS) is one edge of Event Timing System network.
MR-STS is complex of several Event Timing modules while all other edges of
network are configured with just a single EVR module.
Figure 13.7 is the picture of MR-STS. This is VME type EPICS IOC. We put
another EVG module, VME-EVO[7]. It is developed at SINAP. The Event delivery
sequence of VME-EVO can be implemented with the upstream Event as the input
trigger. This is one of the special functions for EVG developed at SINAP. The
input SFP channel of VME-EVO is connected with the lower-EVG at MTS. Thenthe Events for controlling injection hardware at MRs are newly generated at thisEVG.
By installing new EVG, the various functions can be integrated at MR-STS.
Totally 27 kinds of timing-triggers are generated with only one Event from MTS
at LINAC. The arbitrary delay parameters can be set on all timing-triggers individually.
Three EVRs are installed in the downstream of EVG at MR-STS. One is installed on the same VME-bus with the EVG while remaining two are installed at
D7 and D8 sub-buildings, remotely. All of them are developed at SINAP.
The VME form factor EVR, VME-EVE[7], is installed on the same VME-bus.
It delivers timing-trigger towards hardware belonging BT, such as the BPMs. All
output channels have fine delay function based on the GTX technogy. It is oneof advantages of EVR developed at SINAP. Three kinds of timing-triggers areprovided from this module.
657

Two standalone EVRs installed at D7 and D8 sub-buildings on the downstreamof EVG. Figure 13.8 is the picture of standalone EVR installed at D7. This module
consists of the EVR, CPU, and ether network port in the 1U height of 19 inch box.
The EPICS is running on this CPU and we can maintain the EVR from EPICS
network. They delivers timing-triggers for MR hardware such as the injection
kicker and septum magnets. The EVR at D7 (D8) takes care of the LER (HER)
injection and each module provides the 4 kinds of septum magnet triggers and 8
kinds of kicker magnet triggers.
The installation of EVRs at D7 and D8 sub-buildings saves the usage of opticalcables. The timing-triggers for septum and kicker magnets are parallelly deliveredin the previous KEKB project. However, now, one optical cable for each subbuilding is enough to deliver those triggers.
The standalone EVR has disadvantage on the CPU. Its performance is low withrespect to the VME type CPU module and insufficient for the PPM operation.
Therefore all PPM functions for MR injection hardware are integrated on the
VME-EVO. The delay for individual hardware is added on the timing of Eventdelivery. We develop the entire MR-STS system in such kind of way.
The configuration of MR-STS has the feasibility of future extention. As described in the previous paragraph, the functions on the standalone EVR is simplified. It makes the standalone EVR maintenance-free. Therefore, More standalone EVRs can easily be installed in the downstream of VME-EVO with the
non-expert’s work. This is strong advantage. Even though the requirements anddemands of Event Timing System are raised, there are less experts in the world.

13.1.3

Bucket Selection

Loop-topology network
Figure 13.9 is the schematic view of the Bucket Selection system. It consists ofthe three nodes which are connected via the loop topology network. All nodes areequal level. It means there is no relation like master and slave among them. Threenodes are located on MTS at LINAC, the Central Control Building (CCB), andthe D7 sub-building. Figure 13.10 is the KEK map which explains the locationsof three Bucket Selection nodes.
The nodes are the EPICS IOCs which have a distributed shared memory module. They can retain the same information on their memory. Besides they canrequest the CPU interruption with each other via the optical network.

658

Distributed Shared Memory
The reflective memory, VME-5565[5], is utilized as the distributed shared memoryof Bucket Selection. It is a VME form factor module with the DMA transfer
function. Figure 13.11 is the picture of VME-5565.
Table 13.2 shows the data transfer rate between two nodes of the Bucket Selection system. We write the test data into the distributed shared memory of one
node and confirm the same data on the other node. The transfer rate is determined from the period of spent time for this process. The measured transfer rateis slightly smaller than its specification described in the manual. The measured
data includes the data transfer between the distributed shared memory and CPU
module via the VME-bus. This part is actually bottle neck to suppress the datatransfer rate. However this performance of real system is enough for the purposeof Bucket Selection.
Table 13.2: Transfer rate between two Bucket Selection nodes: the rate is determined in two data packet size. The specification is cited from the manual.
Packet Size

Measurement (Mbyte/s) Specification (Mbyte/s)

4 byte packet
8 byte packet

10.9
45.5

43
No data

By using the data synchronization of distributed shared memory, we synchronize the 1MB of CPU memory of three EPICS IOCs. It is quite robust with respect
to the data transfer via the EPICS network.
The VME-5565 has another important function. It can launch the interruption
signal to the CPU module on the same VME-bus. This interruption can be requested from other nodes via the loop network. Therefore, one node can implement
the CPU process of the other nodes with this network interruption.
Main Timing Station node
One node of Bucket Selection is placed at MTS. Honestly, it is the EPICS IOC for
the Event Timing System which is described in the section 13.1.2. The distributed
shared memory is installed together with EVGs in the same VME-bus.
The Bucket Selection process is started on this node. It is implemented in the
PPM process of this IOC. Firstly, the necessary information for carrying out Bucket
Selection, like “which ring do we inject the next LINAC pulse” and “how manybunches we inject in the next LINAC pulse”, is written into the shared memory.
Then the network interruption is launched to the CCB node to implement the
Bucket Selection criteria.
659

The timings for the next LINAC operation are picked up from the sharedmemory when it receives the network interruption returned from the CCB node.
They are set on the EVGs as the delay of Event delivery. Also it is the part ofPPM process.
Central Control Building node
The node which decides the injection-bucket is placed at CCB. The CCB node receives fill pattern request from the operator in advance. The selection of injectionbucket is implemented with the CPU interruption from the distributed sharedmemory during the accelerator operation.
The fill pattern is requested by uploading the ascii file. In this file, the information like “injection” or “not injection” is defined for all of 5120 RF-buckets at
MR. The files are uploaded for LER and HER, separately.
The Bucket Selection process at CCB node is launched when it receives the
network interruption from the MTS node. It becomes following way:
1. Select the injection-bucket for the next LINAC pulse from the fill patternlist
2. Calculate the LINAC operation timing
3. Write them into the distributed shared memory
note, this process is implemented in 50 Hz. The different criteria for items #1 and#2 can be set for the LER and HER injections. Then, after finishing all processes,
the CCB node launches the network interruption to the MTS node to continue thePPM process with the Bucket Selection information.
There are two ways to decide the injection-bucket. One is the normal modeand the injection-bucket is selected in regular order from the list. The other is the
Bunch Current Equalizing mode (BCE). The RF-bucket with the smallest bunch
current is selected as the injection-bucket in this mode. All bunch currents for both
LER and HER are recoreded on the distributed shared memory and the Bucket
Selection can utilize them.
Figures 13.12, 13.13, and 13.14 shows the performance of BCE. The beam
current of individual operation bunches are fairly equalized when we turn on the
BCE. The uniformity which is defined from the r.m.s. of distribution in Figure
13.14 (b) is 9%. It becomes improved when we perform the accelerator operation
with more large bunch currents.
D7 sub-building node
The node at D7 sub-building is the EPICS IOC with the bunch current monitor(BCM). This node provides bunch current information on the distributed shared660

memory so that the CCB node can utilize it for the BCE operation. The update
rate of BCM data is every injection for the injection operation and every 1 second
for the no-injection operation, respectively.

661

Figure 13.7: Picture of MR-STS IOC: the EVG and EVR modules developed at
SINAP is installed in the VME type EPICS IOC. The EVG delivers new Events
when it receives Events from MTS. They are delivered to the EVR installed in thenext slot and those installed at D7 and D8.

Figure 13.8: Picture of standalone EVR installed at D7: the CPU, EVR, andethernet port are housed in the 1U height rack. The 12 timing-triggers for the
LER injection are provided from this module. The triggers are provided also fromrear panel. The same module is installed also at D8 and provides the timingtriggers for the HER injection.

662

Figure 13.9: Schematic view of Bucket Selection network: the distributed sharedmemory is labeled “RM”, namely reflective memory. They configure the looptopology network.

㻰㻣

Main Timing Sta!on

PF-AR

㼕㼚㼓
㻌㻾
㼍㼕㼚
㼕㼚㼓
㻌㻹
㻌㻾
㻷㻮
㼍㼕㼚
㻷㻱
㻌㻹
㻷㻮
㻷㻱

PF

㻷㻱㻷㻮
㻯㼑㼚㼠㼞㼛㼍㼘㻌㻯㼛㼚㼠㼞㼛㼘㻌㻮㼡㼕㼘㼐㼕㼚㼓㼀㼟㼡㼗㼡㼎㼍㻌㻴㼍㼘㼘䠄㻵㻼䠅

Figure 13.10: KEK map: the locations of three Bucket Selection nodes are illustrated with the orange stars.

663

Figure 13.11: Picture of VME-5565: the optical link is configured on the front
panel.

664

Figure 13.12: Bunch Currents at LER on 23:06 of March 1st, 2016: the data
are taken with BCM just after finishing the stacking the beam current. LER is
operated with 1182 bunches. The currents of individual bunches vary widely.

Figure 13.13: Bunch Currents at LER after turning on BCE: the BCE operation isturned on just after we take data in Figure 13.12. These data are taken 13 minutes
later that. The bunch currents are fairly equalized.

665

Figure 13.14: One dimentional histogram of bunch current measurements: (a) the
bunch currents data in Figures 13.12. The r.m.s of distribution is 0.019 mA. (b)Those in Figure 13.13. The r.m.s of distribution is improved to be 0.009 mA.

666

Bibliography[1] EPICS website: http://www.aps.anl.gov/epics/[2] H. Kaji et al., “Construction and Commissioning of Event Timing System at
SuperKEKB”,
in Proc. IPAC’14, Dresden, Germany.
[3] H. Kaji et al., “Bucket Selection System for SuperKEKB”,
in Proc. of 12th Annual Meeting of PASJ, Fukui, Japan.
[4] MRF website: http://www.mrf.fi/index.php/vme-products[5] Abaco website: https://www.abaco.com/products/pcie-5565piorc[6] T. Suwada et al., “Wide dynamic rage FPGA-based TDC for monitoring a
trigger timing distribution system in linear accelerators”,
Nucl. Instrum. Meth. A 786, 2015, pp.83-90.
[7] M. Liu et al., “Development Status of SINAP Timing System”,
in Proc. ICALEPCS2013, San Francisco, CA, U.S.A.

667