Proceedings of IBIC2012, Tsukuba, Japan
MOPA36
DEVELOPMENT OF BUNCHCURRENT AND OSCILLATION RECORDERFOR SuperKEKB ACCELERATOR

Abstract
A High-speed digital signal memory has beendeveloped for the bunch current and oscillation recorderfor SuperKEKB. It consists of an 8-bit ADC and a FPGA
daughter card consists of Spartan-6 and DDR2 memories
commercially available on a double width VME card. The
block-RAM on the FPGA is used to transfer bunch
current data with low latency for prompt bunch current
measurements, and the large DDR2 memory is used forlong-duration position recording, such as post-mortem
bunch oscillation recording. The performance of theboard, including data transfer rate, will be presented.

INTRODUCTION
The construction of the SuperKEKB accelerators to
upgrade the KEKB B-factory has started in FY 2010 andin progress almost on schedule up to now. On
SuperKEKB rings, we almost double the stored current,
reduce the beam emittance down to about 1/10, squeeze
betatron functions at the interaction point to achieve 40
times larger luminosity than KEKB. As the physical anddynamic aperture of the rings will be much smaller thanthat of KEKB, a positron damping ring is underconstruction to reduce the beam size of the injected beam.
Table 1 shows the main parameters of the SuperKEKB
accelerators (High Energy Ring: HER, Low Energy Ring:
LER and Damping Ring: DR).
Since the luminosity of a collider is proportional to thebunch current product of each beam, and it is almost
normal that they push the bunch currents as near as
available to the beam-beam limit, it is fairly important tomeasure the bunch currents and to keep the filling pattern
as flat as possible for stable operations and effective
tunings. In other rings such as the damping ring or storagerings for SR use, though the priority of measuring prompt
bunch current to flat the filling pattern is not so high, it isstill meaningful to record or control the bunch fillinginformation to understand the beam behaviour such as tostudy the collective effects.
In KEKB, we have used the bunch current monitor andthe bunch oscillation recorder based on the hardware twotap filter for the bunch feedback systems[1]. It had an 8bit ADC (MAX101), fast data demultiplexer which
demultiplex the 8-bit data to 16 ways, and 20MB SRAMS,
controlled by the extended VME interface. During
injection, the injection trigger signal interrupted the bunch
current recorder to stopped the recording, then initiate thedata transfer process using the VME interrupt to transfer
the bunch current information to a reflective memory___________________________________________

board on the same VME bus. One of the other reflective
memory board connected with the optical fibre wasplaced at the gun control rack of the linac (~600 m far
from the bunch current monitor) to select the injection
bunch from the bunch current information[2]. This bucket
selection system (bunch current equalizer) has worked
very well to stabilize the bunch filling pattern fairly flat
even after large loss of bunches due to beam-beam kick or
beam instability. This enabled us to make fine tuning of
the colliding condition to get higher and stable luminosity.
Actually, when we had trouble on the bunch current
monitor such as jumping the timing and couldn’t use thebunch current equalizer, the operation of the ring was
fairly difficult and the peak and integrated luminosity ofsuch period was obviously lower than other time.
We have developed a new bunch current recorder basedon current FPGA technology (18K10). With the aid of thegreat flexibility and the capacity of the FPGA, the board
supports the harmonic number of h=5120 for
SuperKEKB-LER and HER, h=230 for SuperKEKB
Damping Ring, h=640 for PF-AR and h=312 for PF ring
just by setting the DIP-switch on the board. Also, we havedesigned the board to be usable for the large-scale
memory board which records all the bunch oscillations
over long time, 0.16 s for SuperKEKB rings. The
performance of the board, including data transfer rate will
be shown.
Table 1: Main parameters of SuperKEKB rings (HER:
electron, LER: positron, DR: positron damping ring)
EnergyCircumference
RF frequency
Revolution
Beam current
H. number
Bunch number
Bunch length
H emittance
Coupling

LER

HER

DR

4.0
3016
508.886
0.0994
3.8
5120
2500
6
3
0.4

7.0
3016
508.886
0.0994
2.6
5120
2500
5
5
0.3

1.05
GeV
135
m
508.886 MHz
2.2
MHz
0.08
A
230
4
5
mm
13
nm rad10

OUTLINE OF THE SYSTEM
Hardware Design
The block diagram of the bunch current recorder board(18K10) is shown in Fig. 1. It has an trigger input, an
revolution input, an RF clock input and twocomplementary DATA inputs. Both trigger and revolution#makoto.tobiyama@kek.jp

ISBN 978-3-95450-119-9
Feedbacks and Beam Stability

1c 2012 by the respective authors — cc Creative Commons Attribution 3.0 (CC BY 3.0)
Copyright ○Makoto Tobiyama# and John W. Flanagan,
KEK Accelerator Laboratory, 1-1 Oho, Tsukuba 305-0801, Japan

MOPA36

Proceedings of IBIC2012, Tsukuba, Japan

VME P1/P2

NIMLVDS

TRIGGER

IRQ
DTACK

Control CPLDA32
NIMLVDSREVOLUTION

D32/BLT

RF(509MHz)

10E195Delay
16bit to32bit

SIG IN +6dB
THS4303 14dB

SIG IN --Data
ADC
MAX108

RF/2

Spartan-6
XC6SLX45-3CSG324C FGPA

Reset

Flash ROM

DDR2 128MB

c 2012 by the respective authors — cc Creative Commons Attribution 3.0 (CC BY 3.0)
Copyright ○Figure 1: Block diagram of the bunch current recorder (18K10).
inputs have buffered outputs. All the inputs and theoutputs are designed to match impedance of 50 . On the
RF clock input, we have inserted a digital delay(MC10E195) with a resolution of 10 ps and a range from0 to about 10 ns, which is controlled through VME bus.
By using this delay, we can adjust the ADC timing
without inserting analog delays before the bunch signal
inputs. The two complementary signal inputs (SIG IN+
and SIG IN-) are amplified with
DC amplifiers(THS4303, DC-1.8 GHz bandwidth) with the gain of 14
dB. Since we have inserted 6 dB attenuator between theDC amplifier and the ADC input, the total gain from the
input to the ADC will be 8 dB (2.5 times). MAX108 (8bit, DC-2.2 GHz BW, 1.5 GSPS max) is used as theAnalog-to-Digital converter (ADC). The ADC has a builtin 1:2 demultiplexer and reference clock outputs (RF/2)
which make latter signal processing much easier thandirectly handling 508 MSPS signal. The reset timing of
the built-in demultiplexer is supplied from the FGPA card
if needed. To cool the ADC, we added a heat-sink withsmall DC fan. The temperature of the ADC is monitored
using built-in diode of the ADC with the control CPLD.
For the signal processing, we have used a FPGA
daughter-card module (Mars MX1-45-3C) commerciallyavailable from Enclustra[3]. Figure 2 shows the Mars
MX1-45-3C daughter card. It has an Spartan-6 FPGA(XC6SLX45-3CSG324C), 128MB of DDR2 SDRAM on
a SO-SIMM form factor card (68 x 30 mm, 200 pins).
The usable I/O are totally 108 pins. We have assigned 35
pairs as LVDS signals (70 pins) and 38 pins as LVTTL
signals. Timing signals are converted to LVDS level to fit
this interface except for the MAX108 outputs. The
LVPECL outputs of the ADC are terminated to 50  justbefore the SO-DIMM connector, then received as LVDS
signals with the FPGA. With this scheme, attenuation of
the signal level due to the large stray capacitance of the
SO-DIMM connector has been compensated. It also has a

Quad SPI flash of 16 MB on the board, which is reprogrammable through the external J-TAG connector, toboot the FPGA code automatically after power cycle. To

Figure 2: FPGA daughter card (Mars MX-1).

Figure 3: Photo of the bunch current recorder board(18K10). Spartan-6 FPGA is mounted on SO-DIMM likedaughter card. MAX108 ADC is placed under the fan andthe heat sink.

ISBN 978-3-95450-119-9
2

Feedbacks and Beam Stability

ISBN 978-3-95450-119-9
Feedbacks and Beam Stability

3c 2012 by the respective authors — cc Creative Commons Attribution 3.0 (CC BY 3.0)
Copyright ○Proceedingsof IBIC2012,
Tsukuba,The
VME Japan
address, IRQ number and IRQ MOPA36
vector is
extract the data directly to VME databus, an 8-bit
data
bus with data strobe timing has used. The 32-bit data defined on the DIP and rotary switch on the mother card.
which fit the D32 data width of the VME bus is It occupies 512 byte of VME address space. For the ADC
multiplexed with the data strobe timing. As the FPGA data, only one 32 bit address space (D32) is prepared. The
card has a voltage regulator (1.8V and 1.2 V) needed for data is accessed through this windows using FIFO logic.
the FPGA and DDR2 memory on it, only 3.3 V is For the BLT access, 256 byte address space is accessible
for the ADC data area but the VME address itself is
supplied from the mother board.
The FPGA includes all the logic for the bunch current ignored during BLT access and is handled as if it accesses
mode corresponding to SuperKEKB (h=5120), PF-AR the first address of the ADC address space.
(h=640), PF (h=312) and Damping Ring (h=230) in bothbunch current mode and large scale memory mode. The
EVALUATION OF THE BOARD
mode can be selected with the setting of the DIP switch
EPICS Settings
on the mother board.
To control the function of the FPGA and to commuWe have been using EPICS system as the main control
nicate with the VME bus, we use two CPLDs. The first scheme on KEKB accelerators and will continually be
one is used to communicate with the VME bus (ALTERA used on SuperKEKB rings. The version of the EPICS
EMP7256A), and the other is used to control FPGA and used on this evaluation is R314.12 on VxWorks 6.2. As
ADC (ALTERA EMP7256A). Both CPLDs are also re- the VME system controller (IOC), we have used theprogrammable through external JTAG connector. The Emerson MVME5500 (MPC7457 1 GHz processor)
photo of the 18K10 is shown in Fig. 3.
which support block data transfer, such as MBLT or BLT
by using the Universe II chip functions. Figure 4 shows
Recording Function
the photo of the evaluation setup. Independent EPICS
In the bunch current recording mode, we limit the device support and databases is prepared for each bunch
memory size as the ring memory with the length of current mode and large scale memory mode. For the
harmonic number. Therefore, only the block-RAM in the interrupt handler, the device support is designed to read
FPGA is used. With the start command, the board the board status register at first which automatically reset
memory address is at first synchronized to using the the interrupt status of the board, then forward link of theexternal revolution timing, then the data are recorded until EPICS database initiate the reading function of the data.
the stop signal arrives. After receiving the stop signal, the During reading process, we disable both the interrupt andrecording is stopped at the end of the ring memory board restart function. After finishing the reading, theaddress, then the interrupt is initiated. The data device support enables the interrupt and restart the board.
acquisition will not restart until receiving the startcommand again from the VME.
For the memory mode, the synchronization mechanismof the revolution timing to the memory address is thesame as in the bunch current mode. We can select the
memory size from 4k, 8k and 16k turns of revolution. In
case of SuperKEKB, 4k turns corresponds to 40 ms
which is roughly same as the transverse radiation
damping time. The trigger position can be selected bycommand from the VME with (1) -100%: stop recording
just after receiving the stop signal. (2) 0%: stop afterrecording half of the memory after the trigger. (3) 100%:stop recording after the full length of the memory filledwith the data after the trigger. ADC data is buffered in the
FPGA block-RAM, then transferred to DDR2 memory Figure 4: 18K10 in the VME subrack. VME extension left
with burst mode. When reading the memory, the data on to the 18K10 is used to monitor the VME bus timing
the DDR2 memory is at first transferred to the block- using logic analyser.
RAM in the FPGA, then transferred to the VME bus
using long FIFO buffer to compensate the refresh cycle of Synchronization Test
It is always important that the recording data is
the memory.
correctly synchronized to the revolution of the accelerator,
VME Functions
especially when we use the recorded bunch current data to
The 18K10 is designed to support the following VME select minimum bunch current bucket on the bucket
selection system to equalize the filling. With the simple
functions:algorithm to equalize the bunch filling pattern, jitter of the A32D32 supervisory data access (AM code : 0x0D).
 A32 supervisory block transfer (BLT, 0x0F) on ADC bunch current monitor easily yields fatal effect such as
injecting huge bunch current into one special bunchdata region only.
within fairly short time.
 VME Interrupter (ROAK).

MOPA36
Proceedingsof IBIC2012,
Tsukuba,
PPC750Japanwas used as IOC, 1.4 ms typically. It is
We have examined the jitter rateby inputting
the when
known pulse signal (attenuated revolution signal with well known that the VME data cycle of the MVME5500
pulse width of about 150 ns) to the signal input and is slower than PPC750 due to huge overhead in PCI-VME
checked the change of the falling edge ADC data using bridge and is also fairly difficult to speed up with normal
EPICS sequencer. The bunch current mode for methods.
SuperKEKB (h=5120) was used on this examination. By
monitoring two 18K10 board simultaneously with thestop trigger rage about 300 Hz, the jitter error was
confirmed to be less than 5 108 on both board after
adjusting the timing of the revolution timing input to be
the good point with enough timing margin. We confirm
the synchronization mechanism used on this board isworking well as expected.

c 2012 by the respective authors — cc Creative Commons Attribution 3.0 (CC BY 3.0)
Copyright ○ADC Test
The Peak-to-Peak and RMS error of the system has
been measured by fitting the recorded data of the clear
sinusoidal signal input. The bunch current recording
mode was used on this test.
The RMS error of the fitted data was 0.98 bit and thepeak-to peak error was about 3.7 bit. No significant
pattern was observed in the amplitude error distribution.
We can conclude the fast ADC and the pre-amplifier isworking well. On the large scale memory mode, it
sometimes shows a large spike, especially when the data
passes 0x7F to 0x80. Since the ADC is working well on
the bunch current mode, this spike should be coming fromthe miss-timing of FPGA or DDR2 memory. Detailed
investigation is now in progress.

Data Transfer RateAs the maximum injection rate of the KEKB andSuperKEKB is 50 Hz, the data transfer time from the18K10 to the IOC should be much shorter than 20 ms tokeep the calculation time of the bunch current equalizer.
The bunch current monitor used in KEKB showed aninterrupt response time of typically 21 s and datatransfer time of about 1.4 ms (5120 bunches) under
EPICS R313 with Power PC 6750 IOC (PPC750). For
SuperKEKB, the situation might be more severe if thetransient beam loading effect is much stronger than
KEKB; in such case we need to use I/Q detection using
two set of 18K10 for one ring and perform amplitude (andphase) calculation using the IOC CPU resource. For thelarge scale memory mode, as we need to record a large
amount of data to relatively slower remote disk using
NFS or tftp protocol through terribly crowded control
network, the data transfer time from 18K10 to the IOC
might not be the rate-limiting step, though the speeding
up of the data transfer surely make total waiting timeshorter when transferring large data such as 16k turns, 80
MB data.
The measured interrupt response time of the 18K10
with MVME5500 was 9.5 s. Figure 5 shows the typical
data transfer cycle (A32, D32) with AM code of 0x0D
(A32D32 supervisory data access). The typical datatransfer cycle was around 1.5-1.7 s per D32 read process.
To transfer 5120 data using MVME5500 with VxWorks
6.2, it required 2.1 ms. It is much slower than the case

Figure 5: A32D32 normal VME cycle.
We have implemented block transfer function using the
DMA access of the Universe II chip on the MVME5500.
On the block transfer mode[4], we at first allocate
memory space of the CPU (not in the VME address space
but the memory directly accessible from the CPU). VME
master at first asserts the first VME address, AM code of
0x0F (A32 supervisory block transfer: BLT mode) thendrive AS* (AS* is kept low during entire BLT cycle).
18K10 places D32 data on the VME bus then drive
DTACK*. VME master transfers the D32 data to theallocated CPU memory and release DS*. 18K10 releases
the D32 bus and prepare next data. While AS* is kept low,
the above cycle repeats up to a pre-defined cycle limit.
After the block transfer finishes, we transfer the data fromthe allocated CPU space to the device support memory
space, which also should be on the CPU memory space,to make needed calculation and to transfer the data to
EPICS database. As the memory space on the 18K10 for
the ADC data is 256 bytes, which is 64x4 word, themaximum length of the BLT cycle is limited 64 cycles
per access though the maximum number of BLT cycle byFigure 6: A32D32 BLT access.

ISBN 978-3-95450-119-9
4

Feedbacks and Beam Stability

MOPA36

SUMMARY
We have developed the bunch current and bunch
oscillation recorder (18K10) using FPGA daughter card
commercially available. For the bunch current mode, theperformance of the board such as ADC error, timing jitterand data transfer rate was evaluated to be sufficient as
expected. In the large scale memory mode, it shows someunstable behaviour both in regards to the acquired data
and the timing jitter. Detailed investigation is in progress.

ACKNOWLEDGEMENT
The authors would like to express their sincere
appreciations to KEKB control group for their help todevelop the device support with new EPICS environment.
The DMA transfer function of the Universe-II chip on
VxWorks has been developed by Mr. T. Okazaki ofTonichi-Giken Co. Ltd.

REFERENCES[1] M. Tobiyama and E. Kikutani, Phys. Rev. ST Accl.
Beams 3,012801(2000).
[2] E. Kikutani, et. al., in proceedings of APAC01,
Beijing, 2001, p.669.
[3] http://www.enclustra.com/en/products/fpgamodules/mars-mx1/[4] W. D. Peterson, “The VMEbus Handbook”, VITA.

ISBN 978-3-95450-119-9
Feedbacks and Beam Stability

5c 2012 by the respective authors — cc Creative Commons Attribution 3.0 (CC BY 3.0)
Copyright ○IBIC2012, Tsukuba, Japan
VME64x specification is 256. FigureProceedings
6 shows theofVME
bus timing during BLT access.
Typical access cycle time was around 550 ns per D32
read. To transfer 5120 data with the BLT access took 1.1
ms, which is about the half of the normal data transfer
time and about 30% faster than when using a PPC750 asIOC. By extrapolating this transfer rate to large memorytransfer, we can estimate the data transfer time. For
example, in case of 4k turns of data transfer (40 ms = 20
MB), it will need about 4 s which is much shorter than the
PPC750’s case of about 16 s (measured). The overhead
due to limited BLT cycle (64 cycles) is estimated to be
negligibly small in KEKB bunch current mode (about 20s) and the 4k turns of memory (about 20 ms).
For the use the of bunch current equalizer, it isrequired to transfer the bunch current data from the18K10 board to the linac gun trigger control systemthrough reflective memory earlier than the next injection
timing. We have already started the evaluation of thereflective memory (VMIVME5565) using the long opticalfiber from the central control room to the linac gun
control, about 500 m. The data transfer rate between thetwo boards through optical fiber is 178MB/s byspecification. As it supports DMA data transfer on the
VME bus, we hope the latency of the system to be shortenough. Total performance of the system will be
evaluated within this fiscal year.