The Main Injector Beam Position Monitor Front-End Software

Luciano Piccoli, Stephen Foulkes, Margaret Votava and Charles Briegel

Fermi National Accelerator Laboratory

P.O. Box 500

Batavia, IL, USA 60510

**Abstract.** The front-end software developed for the Main Injector
(MI) BPM upgrade is described. The software is responsible for
controlling a VME crate equipped with a Motorola PowerPC board running
the VxWorks operating system, a custom-made timing board, and up to 10
commercial digitizer boards. The complete MI BPM system is composed of 7
independent units, each collecting data from 19 to 38 BPM pickups. The
MI BPM system uses several components already employed on the successful
upgrade of another Fermilab machine, the Tevatron. The front-end
software framework developed for the Tevatron BPM upgrade is the base
for building the MI front-end software. The framework is implemented in
C++ as a generic component library (GBPM) that provides an event-driven
data acquisition environment. Functionality of GBPM is extended to meet
MI BPM requirements, such as the ability to handle and manage data from
multiple cycles; perform readout of the digitizer boards without
disrupting or missing subsequent cycles; transition between closed orbit
and turn-by-turn modes within a cycle, using different filter and timing
configurations; and allow the definition of new cycles during normal
operation.

Keywords: Beam Position Monitor.

PACS: 07.77.Ka

# Introduction

The Main Injector (MI) is a complex particle accelerator used for
several experiments at Fermilab. The machine has the ability to
accelerate protons and antiprotons from 8 GeV to 150 GeV. Protons from
the Main Injector are used for antiproton production, as well as
neutrino production. The Main Injector is also used for moving
antiprotons between the Accumulator, Recycler and Tevatron for collider
operations.

The front-end software is one of the components developed for the
ongoing MI Beam Position Monitor (BPM) upgrade \[1\]. The functionality
of the software system is based on the recent upgrade to Fermilab's
Tevatron \[2\].

The following section describes the components of the upgraded BPM
system; software requirements; existing framework developed as a basis
for the Tevatron BPM upgrade; extensions for supporting unique MI
abilities and modes of operation; performance measurements and
conclusion.

# Front-End system Overview

The upgraded MI BPM system is composed of components located in the
tunnel close to the BPM pickups and in service buildings. In the tunnel,
the signals harvested by the pickups are merged by a combiner box
(Figure 1, part a) and sent to service buildings via cables of varying
lengths (100 ft. to 1200 ft.). There are seven service buildings located
around the MI ring, each receiving signals from 19 to 38 pickups.

EMBED Word.Picture.8 ![](media/image1.wmf){width="3.5729166666666665in"
height="2.3645833333333335in"}

**FIGURE 1.** Main Injector BPM upgraded hardware components. Signals
harvested from the pickup are combined while in the tunnel and sent
upstream to the transition boards in the service building. Transition
board outputs are connected to digitizer boards controlled by the
front-end software.

Each service building has two VME crates (Figure 1, part b). The first
crate contains a set of transition boards \[3\] which receive the
signals from the combiner boxes. The crate controller card receives
commands from the main crate and configures the transition boards to
output either the 2.5 MHz or the 53 MHz signals coming from the combiner
boxes.

The main crate contains a processor board, a custom-made timing board
and commercial digitizer boards. The processor board chosen for the MI
BPM upgrade is the Motorola^TM^ MVME5500-0163 with 512 MB of memory. It
runs the VxWorks^®^ Real Time operating system (version 5.5). Due to the
potentially high amount of data to be collected, this board was chosen
because it is possible to install a memory extension of 512 MB, totaling
1 GB.

Next to the processor board sits the custom-made timing board. It is
based on the board developed for the Tevatron BPM upgrade, with minor
hardware modifications and additional firmware functionality. The board
generates timing signals that control the acquisition of the BPM
signals. Timing signals are based on inputs from the accelerator
controls clock system, which include signals synchronous to the beam:
Main Injector RF clock (RFCLK); the Main Injector beam sync clock
(MIBS); the Recycler beam sync clock (RRBS); the Booster extraction
signal (BES); and asynchronous signals: Machine Data (MDAT) and the
Tevatron event clock (TCLK). The board provides synchronization signals
for up to ten digitizer boards. Signals to the digitizer boards are
propagated through individual cable connections between the boards.

The digitizer boards are commercial boards from Echotek^TM^ corporation
(model ECDR-GC814-FV-2) \[4\], containing high-speed digitizers for the
analog inputs. The board also contains a digital down converter, which
produces in-phase (I) and in-quadrature (Q) outputs. Each sample is
represented by one I/Q value pair. One digitizer has eight input
channels, which handle signals from four BPM pickups.

# Software requirements

The upgraded BPM system for the Main Injector has requirements that
extend the functionality of the original system, while functioning in a
similar manner in order to minimize disruptions and avoid redesigning
the user interfaces.

Basic data acquisition modes that must be supported by the front-end
system are turn-by-turn and closed-orbit modes, gathering data from
particles bunched at either 53 MHz or 2.5 MHz.

The operating mode of the front-end system is defined by the Main
Injector state, also called cycle. A number of injections and
extractions may occur during a cycle, and the front-end must to be
programmed to arm the desired measurements at the correct time.

On injections and extractions the system is set up for taking
turn-by-turn measurements, each of which measures 512 consecutive turns
starting at a given bucket. Injections and extractions are armed by
accelerator clock signals (BES, MIBS or RRBS). The system is able to
collect up to 20 of these measurements, also referred to as flash
turn-by-turn, during a single cycle.

Between injections and extractions the system is able to take closed
orbit measurements at 500 Hz and allow beam position and intensity to be
calculated. Closed-orbit measurements can be stored in separate buffers
for later retrieval at the reception of predefined TCLK events.

For every MI state, the front-end software must keep all data collected
from the last instance of a state. These are overwritten every time a
new instance of the same state is started.

The following section describes some key aspects of the GBPM framework,
which is used as basis for the MI BPM front-end software.

# The generic BPM framework

Most of the basic functionality required for the MI BPM front-end
software is implemented by a generic framework (GBPM) developed for the
Tevatron BPM upgrade \[5\]. GBPM defines modules that allow the
composition of an event-driven data acquisition. Figure 2 shows how the
components interact with each other to form a simple data acquisition
system.

![](media/image2.wmf)

**FIGURE 2.** Generic framework developed for the Tevatron BPM upgraded
and reused for the MI BPM upgrade project.

Any data acquisition is started by an *event*, which can be defined, for
example, as a timer interrupt or a user command. The event is passed
onto one of the active entities in the system: *control task*, *data
acquisition task* or *alarm task*. The tasks are constantly waiting for
events coming through *event queues*. The reception of an event triggers
the data acquisition and any required processing.

The sequence of data acquisition takes the following form: suppose that
an event source (timer interrupt at 1 Hz) is programmed to generate
events. The event is passed onto any tasks that are listening for events
from the generator. One of the listening tasks can start collecting data
from a device at the reception the event and re-arm the device after
reading it out.

The GBPM framework also abstracts the process of reading out a device
and saving the data into memory. Any entity in the system capable of
generating data is called a *data source* and any entity that can store
data is a *data buffer*.

The alarm task is dedicated to receive alarm events for any other part
of the system. According to the alarm received the task may forward it
to users through the controls network layer.

The controls network layer is the connection between the framework and
external applications. The GBPM supports primarily ACNET; however, it is
possible to extend it to use other control systems, such as EPICS.

In order to meet the front-end software requirements, the GBPM
functionality was extended. The following section describes the
additions.

# MIBPM Extension

The GBPM used as a base for the MI BPM front-end software required the
addition of a few capabilities. The constant and fast transition between
MI cycles requires the front-end to reconfigure itself to quickly handle
distinct data-taking configurations. Data from states must be kept in
memory until overwritten by another instance of the state and must be
available for applications via the controls network.

## Multiple state support

A Main Injector state is identified by a combination of TCLK and MDAT
events. The reception of a state change TCLK event informs that the next
state number is available from the MDAT line. This combination of
signals defines the next series of data-taking commands to be executed.

The list of commands for all states is sent through the controls network
to all MI BPM front-ends, making sure all have the same set of
instructions, as shown in figure 3.

Commands available to operators are: filter, closed orbit, flash,
turn-by-turn and raw. The filter command allows the operator to select
the transition module frequency (2.5 MHz or 53 MHz), as well as
attenuation; closed orbit enables the collection of averaged positions
at 500 Hz; flash starts a 512 point turn-by-turn measurement;
turn-by-turn allows the acquisition of beam position on 2048 consecutive
turns; and the raw command disables filtering at the digitizer boards,
allowing operators to calibrate the timing for turn-by-turn measurements
and diagnose simple problems like a disconnected input.

All commands have a starting time, specified in ms, that is relative to
the state change TCLK signal. It is possible to combine all commands,
except the turn-by-turn and raw, which cannot be mixed with closed
orbits and flash measurements.

![](media/image3.wmf)

**FIGURE 3.** Iteration between front-end components for supporting
multiple Main Injector states. The list of commands is received from the
controls network, which is activated by a TCLK event and identified by
an MDAT value. The control task programs the state timer in the timing
board and also configures the digitizer for the next measurement.

## Buffer organization

For every defined MI cycle the front-end reserves a set of buffers,
which are capable of keeping data for all potential measurements. A
buffer set includes closed-orbit related buffers profile (128), display
(1), user TCLK (128), flash average (1); and turn-by-turn related
buffers turn-by-turn (2048) and flash turn-by-turn (512). The numbers in
parenthesis are the total of samples that each buffer can hold.

The profile, display and user TCLK buffers pull data from the 500 Hz
closed-orbit buffer when a specific TCLK signal is decoded. The profile
and display buffers have predefined TCLK trigger events, while the user
TCLK buffer has a configurable event. The flash average is a calculated
closed-orbit measurement based on the first injection flash taken in the
cycle.

The system has a pre-allocated buffer pool of flash turn-by-turn
buffers, each capable of storing 512 measurements. The variable number
of flash turn-by-turn measurements per cycle requires the front-end to
use dynamic buffer assignment in order to keep optimum memory usage. A
single cycle is allowed have up to 20 flash turn-by-turn buffers used.

## Modes of Operation

The system defines two modes of operation for regular data taking:
closed orbit and turn-by-turn modes, and another two modes used for
diagnostics: raw and safe modes. All modes of operation can be
configured to measure data at 2.5 MHz or 53 MHz, depending on the
configuration of the transition boards.

### Closed Orbit Mode

Closed-orbit measurements are taken by configuring the digitizers to run
in an averaged mode (narrow bandwidth mode). The timing system is
programmed to generate a trigger for the digitizers at a rate of 500 Hz,
represented by the 2 ms interrupt in figure 4. After sampling the beam
for approximately 40 turns, the software reads out a single I/Q value
per digitizer channel through the VME backplane and makes it available
for plotting through ACNET. If a profile, display or user TCLK is
decoded, the closed-orbit measurement is also saved into the appropriate
buffer.

![](media/image4.wmf)

**FIGURE 4.** Front-end components involved with closed-orbit
measurements.

### Turn-by-Turn Mode

On turn-by-turn mode the digitizers are configured to measure particle
positions once every turn (wide bandwidth mode). The timing system is
configured to start generating triggers synchronously with the beam. A
trigger is issued every turn to collect data starting at a specific
bucket that is configurable by the operator.

Due to timing restrictions and potential high quantity of data, the
transfer from the digitizers to the processor board are delayed until
the cycle ends, at the reception of an end-of-beam TCLK signal.

After the data transfer is complete, the measurements are routed to the
appropriate buffers. For injections and extractions, data are copied to
the flash turn-by-turn buffers (512 points each), while for
user-triggered turn-by-turn, the data is saved into the turn-by-turn
buffer (2048 points).

![](media/image5.wmf)

**FIGURE 5.** Front-end components involved with closed orbit
measurements.

### Raw Mode

Raw mode samples the beam at the maximum digitizer frequency, about 75
Mhz, with all filtering turned off on the digitizer card. The readout
also occurs at the end-of-beam TCLK and data is saved into a special
buffer that is accessible only through a diagnostics application. The
raw mode can be used to define the timing parameters for the system and
helps to detect faulty connections.

### Safe Mode

The safe mode is designed to be used as an option when the system is not
receiving reliable synchronization signals, or when timing has changed.
In this mode the digitizers are configured in wide bandwidth mode and
12,000 measurements are taken continuously, instead of one measurement
per turn.

The processor uses an algorithm to search for intensity peaks on the
data samples. The selected measurements are saved into the first flash
turn-by-turn buffer.

## Digitizer Configuration

Most functionality of the digitizer device driver from the Tevatron BPM
upgrade was used for the MI project. However, due to timing requirements
for MI state data acquisition, the driver functionality was extended.

The digitizer is configured to generate four parallel data streams, with
data for closed-orbit and turn-by-turn measurements at both 2.5 MHz and
53 MHz. The data path for data from a channel pair is depicted in figure
6.

![](media/image6.wmf)

**FIGURE 6.** In normal data-taking the BPM signals pass through the
digitizer gray chip and FPGA, which is configured to generate four
streams of data; two with closed-orbit measurements and two for
turn-by-turn measurements.

Closed-orbit data (2.5 MHz and 53 MHz) are always read at 500 Hz, while
turn-by-turn data is kept within the digitizer memory until the
end-of-cycle signal is received. There is enough time between cycles to
allow for data to be transferred across the VME backplane into the
processor memory.

# Performance Measurements

The front-end software performance was measured in the test stand setup,
which contains a fully-configured crate with ten digitizer boards. The
timing of the most critical functions was measured and the results are
listed below.

The closed-orbit measurements are taken at every 2 ms. Most of the time
(approximately 1.3 ms) is used by the digitizers to collect data. The
data transfer time is minimal and does not have any impact on the
performance. The remaining 25% of a closed-orbit measurement is used for
calculating the position and intensity of the beam based on the I/Q
values from the digitizers and also to move data internally to other
buffers as needed (e.g. profile and display closed orbit measurements).

During turn-by-turn measurements data are taken for every turn by the
digitizers and stored in memory. The transfer of data from the
digitizers to the processor starts only after the current MI cycle
finished by the reception of the end-of-beam signal. Between cycles
there is a 500 ms time window that is used to move data from the
digitizers into the processor memory. The maximum amount of data to be
transferred from the digitizers is 12,000 measurements, and for a
fully-loaded crate the total transfer time takes 150 ms. The remaining
time is also used for position and intensity calculation and for moving
data to the correct memory buffers.

During a cycle the front-end is required to switch between closed-orbit
and turn-by-turn mode within 10 ms, which is the time the machine needs
to transition between bunch structures. The amount of time to transition
between different modes was measured at 1.8 ms, allowing plenty of time
for any additional processing during a mode change.

# Conclusion

The front-end software for the Main Injector BPM upgrade has been
described. The system structure uses the framework designed and
implemented for the Tevatron BPM upgrade. In order to meet the upgrade
requirements the framework had its functionality extended. It also
required changes in the digitizer driver and in the timing hardware to
support multiple MI cycles.

The system will be commissioned starting in May of 2006 after the
machine is back up from a long shutdown. The original BPM system will be
used to bring the MI back into operations, except for a fraction of one
service building that is under control of the upgraded system.

# Acknowledgments

This work was supported by Fermi National Accelerator Laboratory,
operated by Universities Research Association Inc. under Contract No.
DE-AC02-76CH03000 with the United States Department of Energy.

# References

1\. S. Wolbers et al., *Fermilab Main Injector Beam Position Monitor
Upgrade*, in 12^th^ Beam Instrumentations Workshop (BIW06), Batavia, IL,
May 2006.

2\. S. Wolbers et al., *Tevatron Beam Position Monitor Upgrade*, in
Particle Accelerator Conference (PAC05), Knoxville, TN, May 2005.

3\. M. Wendt and S. M. Rapizarda and A. Saewert, *Analog Signal
Pre-Processing for the Fermilab Main Injector BPM upgrade*, in 12^th^
Beam Instrumentations Workshop (BIW06), Batavia, IL, May 2006.

4\. Echotek Corporation, *Echotek ECDR-814x user manual*, Huntsville,
AL, \[Online\]. Available: http://www.echotek.com/a2d/ecdr-814
AD/ECDR-814Rev3p0062001.pdf.

5\. L. Piccoli, M. Votava and D. Zhang, *The New Tevatron Beam Position
Monitor Front-End Software*, in 14^th^ IEEE-NPSS Real Time Conference,
Stockholm, Sweden, June 2005.