Current DSP Applications in Accelerator Instrumentation and RF
Brian Chase, Anne Mason, Keith Meisner
Fermi National Accelerator Laboratory
P.O. Box 500, Batavia, IL 60510

Abstract
The Beams Division at Fermilab is applying Digital
Signal Processors (DSPs) to a diverse group of control andmeasurement systems. This paper presents Fermilab′s useof the Analog Devices SHARC DSP on custom VXI andcommercial VME cards in a variety of systems. It explores
how these systems take advantage of an architecture thatallows for software and hardware re-use, flexibility, andevolution in systems with real-time and computationallyintensive requirements.
1 Introduction
When Fermi Run II physics resumes in 1999, theBooster, Main Injector, Tevatron, and Recycler Ring will
all have low level RF systems built around the SHARC
DSP. Other key control and instrumentation systems using
the SHARC will also come on line for the next run. This
work started in 1994 for the Low Level RF group upon
reception of an engineering sample DSP. Since this time
the scope of application has increased beyond Low Level
RF to projects in other departments in the division.
Hardware and software design that emphasizes
fundamental functional modules has provided a powerfultoolbox that facilitates these and future projects.
The focus of this paper is the implementation of systems
using the SHARC DSP, and a description of the hardware
and software architecture that takes advantage of the
SHARC′s capabilities. The scope and history of DSP
projects in the Beams Division includes:‘94 - Prototype Direct Digital Sythesizer‘95 - Main Injector LLRF
‘96 - Booster LLRF
- Tevatron LLRF
- Baseband Processor for PET RFQ LLRF
In Progress:- Anti-proton Source BPM- Bucket Generator for Recycler LLRF
- Beam Transfer Synchronizer for LLRF
- MDAT transmitter- Time-line generator- Baseband Processor for MI LLRF

can drive the external port, two serial ports and 6 link
ports. The link ports provide inter-processor
communication in multi-processor systems. Each link port
is a half duplex six line bus that can move a byte of data at
each clock cycle.
The SHARC derives much of its performance fromparallelism in multifunction instructions, data address
generators, and hardware loop control. All instructions are
executed in one clock cycle, eliminating the need forpipelining instructions. An example of a core multifunction
instruction is:
F12=F1*F4, F2=F8+f12, F3=F8-F2, F1=pm(I11,M15);
This computation is a parallel floating point multiply,
addition, subtraction, and a move instruction using the
program memory data address generator. This example
instruction, as all instructions, executes in one clock cycle.
Because the memory is dual ported, the IOP moves data atfull speed without affecting the core processor. At a 66
MHz clock rate, the DSP core peaks at 198 MFLOPS while
the IOP DMA moves data at 396MBytes/s over the 6 link
ports.
Several features of the SHARC architecture have proved
critical to the success of these projects, and will be
discussed as they occur in the system descriptions.
3 Custom SHARC to VXI interface
Custom VXI modules that are in use share a basic
definition that takes advantage of both the DSP and VXIarchitectures.
VXI Data Bus
Trigger Bus

IRQs

Flags

Host port
External Port

Special
Function
Hardware

SHARC DSP

Link

66MByte/s

SerialLink

6

Local Bus to Moduleon the Left

66MByte/s
6

66M Bits/s

Local Bus to Moduleon the Right

Figure (1) SHARC to VXI Interface
2 SHARC architecture
The ADSP-2106x SHARC (Super Harvard Architecture
Computer) is a high performance floating point digital
signal processor. The SHARC combines the DSP corefrom the ADSP-21020 DSP with dual port SRAM and an
integrated I/O processor (IOP). Four independent buses
allow dual data fetches in a single cycle while allowing
independent DMA activity by the I/O processor. The IOP

The backplane has 32 bit access to all SHARC memoryand control registers. Any of the eight VXI trigger lines
may be mapped to any of the SHARC interrupt request
lines. SHARC flag lines can map back to the trigger bus or
to LEDS and test points on the front panel. Two of the six
link ports are brought off board on the VXI local bus, auser defined 12 line bus that connects modules in adjacentslots. SHARCs on VXI modules are daisy chained over

this private link bus in a multi-processor configuration.
One of two SHARC serial port is brought out to the local
bus and tees off in both directions.
4 DDS for main injector and tevatron
The VXI Direct Digital Synthesizer (DDS) is a custom
module designed to provide LLRF frequency and phase
control. It uses the standard SHARC to VXI interface.
DDS functionality is achieved by three RF synthesizer
channels. Each channel consists of a Numeric Controlled
Oscillator (NCO), two DACs, and an IQ modulator. The
NCO is a Harris HSP45106 with 32 bit center frequency
and 16 bit phase control registers with milli-Hertz and 100
micro-radian resolution. NCO outputs are TTL sine andcosine values that drive separate 12 bit DACs. The
nominal 3MHz DAC outputs are filtered to remove alias
spectra, and drive the I and Q ports of a Mini-Circuits
MIQC-88M modulator. The 50MHz reference RF input
drives the modulator LO port and 25MHz NCO clocks.

VXI
BUS

SHARC

VXI
Interface

40 bit Floating Point
Digital Signal Processor

Link Ports

SPORT

Host Port2 Mbit Memory

32 bit Data bus
20 bit Addr bus
Local
Bus
Trigger

Control andInterface LogicBus(3) Direct Digital
Synthesizers
3 ChannelCoupled
3Frequency
ChannelDirectIndependent
DigitalPhase
DirectDigitalSynthisizer
Synthisizer

V=Asin( ω t+ φ)Figure (2) DDS Block Diagram
The DSP and its software control the frequency andphase registers of the three synthesizer channels. These
three DDS outputs create the RF bucket while controlling
bucket area, beam energy, and beam azimuthal positions.
Frequency and phase agile RF bucket manipulations must
be done gently and with minimal RF phase noise to avoid
increasing the beam longitudinal emittance. This requiresthat DSP code execution is deterministic, that frequency
and phase calculations are smooth and accurate, and thatsidebands created in the I/Q modulation process are small
and outside the RF cavity bandwidth.
Many DSP code processes are required for the RF bucket
and beam control. Each repetitive process executes at oneof three basic rates that correspond with specific process
bandwidth requirements. These DSP rates are 720hz,
16.6Khz, and 100Khz.
The DDS SHARC calculates the LLRF feedforward
frequency program (Frf) from real time bend bus current(I) on an accelerator data link (MDAT) that updates at
720hz.
−1
K 2
Frf = FE
= F∞ β = F∞ 1 + 
2( ∆r = 0)

I 
where

F∞ =

hc

2π R 0The DSP interpolates Frf with a 2760 tap FIR lowpass

filter and writes the result to the RF synthesizer channel
frequency registers at 100kHz. The DSP also applies apreviously filtered frequency feedback term at 100kHz.
An example of a 720hz phase control process is DDSSHARC calculation of the synchronous phase angle ( s)required for beam acceleration. The independent variablesfor Φs are the time derivative of beam energy (Edot) and adigitized RF cavity accelerating voltage (Vg). Edot isderived by DSP scaling of successive I values on the
MDAT link.

 2πRs dE −1φ s (t) = sin  ecVg β s dt Φs is summed with other 720hz phase terms, the sum islinearly interpolated, and updates the RF synthesizer phaseregisters at 100Khz.
Seven additional DDS phase terms, grouped by function
and bandwidth, are currently used for Main Injector andTevatron LLRF system beam control. All terms areinterpolated and output at 100Khz. Adiabatic 720hz terms
include: static offsets for beam transfer phasing, groupdelay compensation, slow arbitrary waveforms forcounterphasing, and bucket cogging waveforms. Terms for
non-adiabatic 16.6Khz control include: phase steps fortransition crossing, and fast arbitrary waveforms forcounterphasing. The 100Khz DSP process includes phasefeedback terms and updating synthesizer registers. The
DSP applies all terms and process categories
simultaneously. Lock stepping of the three process rates isaccomplished by a DSP timer and software scheduler thatis phase locked to the 720hz MDAT link with a software
PLL.
The DDS is a critical component of the Main Injector
and Tevatron LLRF systems. The functionality of thesesystems has evolved to stress even the real time capabilityof the single DDS SHARC. Fortunately, other VXI
modules developed for these LLRF systems also use the
SHARC. The DDS SHARC has evolved to become thecrate master DSP. It controls one or more slave DSPs withbi-directional link port communication. The DDS DSP
master scheduler lock steps all slave DSP scheduler
processes, sends all required data, and applies slave DSP
computational results received over the link port. The link
port communication requires very little master or slave
DSP core processor involvement. This approach is used toeasily allocate compute intensive and synchronous systemprocesses among multiple DSPs.
5 Booster DDS
The Booster LLRF is built around a SHARC DDS very
similar to those used in the Main Injector and Tevatron. A
slightly modified design is required due to the wider
frequency range and higher frequency slew rate of the
Booster. Instead of calculating RF frequency in real time,
the SHARC plays a waveform array of frequency program
data to the DDS. The array is downloaded to the SHARC
memory over ACNET and plays out at a one microsecond
update rate over 33 millisecond acceleration cycle to theDDS. The DDS runs on its own 575 MHz clock andoutputs the Booster 38-53 MHz RF directly. In the same
one microsecond interrupt service routine, the SHARC
reads the digitized phase error signal, subtracts an offset
and multiplies the result by a loop gain factor. This is thensummed with the DDS frequency program to close a 150
kHz bandwidth phase locked loop to the beam.

Feedforward
Feedback Error
Program
Waveform
Amplitude &Phase set

The Baseband Processor (BBP) module is the second
DSP design that uses the SHARC to VXI interface. Four
AD9026 12 bit 26 MHz ADCs and four AD9713B 12 bithigh speed DACs are connected directly to the 48 bit databus of the SHARC.
RF IN

VXI
BUS

VXI
Interface

SPORT

Host

32 bit Data bus
20 bit Addr bus
Local
Bus

TriggerBus(4) 12 bit 26 MHz ADC
Control andInterface Logic

clip
lpf

Zn

+

+

Drive OutΣΑe j∆ω t + ϕ

1 cycle delay

Learning
Gain
Curve
Learning
Loop
Gain

Scale

Mag
R-P

Φ

FFT

Det Wave

Cavity tuning
Error

Det Phase
Mag
R-P

Φ

Legend256 point plot waveforms

Det Freq Wave
Det Freq Phase

Settable Parameters
Real value
Complex Value

Figure (4) PET Feedback

RF Return(2) RF Modulator/
Demodulator

SHARC
Link

RF Drive

-

+
Σ
+

Σ

Detected
CavityWaveforme-j∆ω t6 Baseband processor+(4) 12 bit 26 MHz DAC

Figure (3) Baseband Processor
Two of the 10 DMA channels in the SHARC IO
processor have hardware DMA request and grant lines. By
properly configuring the DMA controller, the IOP will play
and acquire waveforms without external hardware or
memory. The IOP can be configured to play or acquire at
the full DSP clock rate, or it can ping pong these modes sothat playback and acquire happen at the same time but at
half the full clock rate. The high speed, high spectral
purity digitizers are combined with signal analysis softwarein the DSP to provide the building blocks for a vector
signal analyzer. The combination of the DACs and DSP
software is the basis for a vector signal modulator.
The analog IO from the ADCs and DACS is connected totwo general purpose daughter cards. PET daughter cards
perform RF IQ modulation and demodulation. Threemodels have been built to support 425MHz, 212.5MHz and
53MHz. These cards frequency translate baseband vector
signals up to and down from their center RF frequency.
For the PET LINAC isotope production accelerator, theBBP provides the LLRF. This RFQ accelerator requires
four complete and independent LLRF systems to drive one
station at 212MHz and three stations at 425MHz. The
LLRF systems produce 80µs pulses of RF at the 360Hz
machine cycle rate. With IQ modulation, the DSP has
complete control and feedback regulation of RF amplitude,frequency, and phase.
Two LLRF systems are
implemented on each BBP module.
At the 360Hz rate, two one kWord waveforms are played
to the RF modulators. SHARC interrupts to handle spark
protection are enabled during the pulse. This SHARC codeis written in assembly. At the 10Hz rate of the control
system, the waveform processing is done by SHARC ″C″
code. The program waveform array is built from theparameters of pulse length, rise time, amplitude, RF

frequency and phase. With feedback off, the program
vector array plays out. With feedback on, the complex
arrays are processed as shown in the feedback diagram in
figure four.
The processing includes vector array
multiplication, addition, filtering, and limiting.
The
complex time domain signal is transformed by FFT tomeasure the resonant frequency and correct the cavity
tuning control loop. Ten 256 sample waveforms per
system are returned by the control system for 10Hz display.
7 Recycler bucket generator
The Recycler Ring low level RF system combines theuse of many previously described SHARC tools. Three
DDS modules provide nine RF clock inputs to a new
SHARC based module called the Recycler Bucket
Generator (RBG). The RBG contains eight arbitrary
waveform generator channels clocked by the DDS outputs
as shown in figure five. The waveform channels are
summed to form the LLRF output that drives the Recycler
cavity high level system. The DSP in the Bucket
Generator is the master of the eight waveform channels aswell as the three DDS modules. The link ports on the local
bus tie all four DSPs together. Most of the RBG ’ ssoftware design, uses existing libraries for frequency andphase control.
The Recycler Bucket Generator presents many newchallenges. Beam in the Recycler is not contained in
traditional RF buckets. Instead, it is contained between
pairs of rectangular pulses that form barrier buckets. The
barrier bucket waveform is amplified to drive four 50 Ohm
wideband cavities. Three sections of beam are containedby 6 barrier pulses. Each barrier pulse has independent
width, position, amplitude, and shape control
programmable as a function of time.
Each waveform channel is clocked by an independent
Direct Digital Synthesizer. Each synthesizer is programmed to the same frequency, while phase isprogrammed to advance or retard the pulse output of thecorresponding arbitrary waveform channel. Each channel
may output from one of two memory blocks. While one
output is being played, the other may be reprogrammed. In
this way the small changes to the shape of the barrier pules
may be made at a 720Hz rate.
High level commands for barrier bucket manipulations

53 MHz RF

Divide by 588

injection oscillation data, the DSP calculates the position
for each turn. For closed orbit mode, the DSP produces anaverage position.

ARB 1
Start

Σ
Divide by 588

HLRF
Drive

ARB 8
Start

SHARC

720 Hz Process
Recalculate Waveform
Phase and Frequency

Link Port
Divide by 588

Revolution Marker to
Beam Sync Clock

Figure (5) Recycler Bucket Generator
are passed from the system CPU over the VXI bus to the
RBG. The RBG then smoothly controls any changes tobucket parameters at a rate that maintains adiabatic beam
control. It also defines bucket position and velocity interms of 53MHz RF frequency and phase. Frequency andphase parameters are passed to the three DDS modules
over the link port at 720Hz. These parameters areinterpolated and played out by the DDS modules at a
100kHz rate.
8 AP BPM digitizer
The SHARC DSP is used in a new beam position
measuring (BPM) system for Fermilab′s Antiproton Source
Accumulator. The Accumulator is a storage ring holding
beam at two momenta and orbits, for stored antiprotons,
and beam injection or extraction. The new BPM system
supports measurement modes that include closed orbit foreither beam, and turn by turn for injected beam. The BPM
system uses split plate beam detectors spaced around theaccelerator. The detector plates provide A and B signalsthat are conditioned by tunnel mounted preamps. The
beam signals are cabled to equipment galleries forprocessing by analog modules and digitizers. The digitizer
is a Fermilab VXI design, consisting of eight 12-bit
channels with one integrated SHARC DSP. Each channel
uses a super Nyquist digitizer with a sampling rate of
25.6MHz and an analog bandwidth greater than 100MHz.
The digitizer writes into memory that holds 128k samples
with 12 bits per sample.
This application of the SHARC is different from theothers presented in this paper, in that the SHARC is being
used as a batch mode computational engine to produce
beam position from the raw sampled data. There are eightchannels per card. With two channels per BPM detector,
each DSP processes data from four BPM detectors.
Measuring the beam position begins by downloading
algorithms to the DSP. The algorithm chosen depends
upon the system mode and choice of the user. After thedigitizers are triggered and channel memory is filled, theDSP combines the raw data and calibration numbers toarrive at the position data. In the case of turn-by-turn

9 Time line generator and MDAT transmitter
The goal of this project is to generate a variety of clock
and data signals for use by Fermilab accelerators. In
particular, the time line generator sources events thattrigger accelerator operations. The MDAT module
transmits magnet current and beam energy information in
real-time to accelerator devices. The serial signals to begenerated are Manchester Bi-Phase Mark encoded. The
clock signals come in two frequencies, both of which arecontinuous wave with intervening ″ones″ between events.
Data signals are burst mode with a logic low level between
frames of data.
Once again, the SHARC DSP was chosen for its
extremely flexible architecture including configurable
serial ports, ample on-chip memory, and floating point
capability. However, in this case an off-the-shelf VME
card by Alex Computer Systems was chosen.
The Alex Computer Systems SHARC6000 VME cardcomes with two SHARCPAC mezzanine slots. Eachmezzanine mounted SHARCPAC module contains one
SHARC DSP and one Altera Flex8K programmable logic
device (PLD).
The first signal to be generated, MDAT or Machine Data,
was also the most complex from the viewpoint ofcustomization. Frames of 28 bits in length are transmitted
in TDM mode with timeslots of 10 sec synchronized witha line locked 720Hz decoded clock event. The 720Hz
event inputs to the PLD. Since the SHARC serial port can
accept an external frame sync signal, the PLD wasprogrammed to produce a one sec pulse train occurring
only at the appropriate timeslots for the data frames. Thissignal triggers the DSP to output the data frame.
The ability to vary the frame sync pulses with respect tothe decoded 720Hz event is key to the proper operation of
the entire MDAT link. This can be accomplished in two
ways. The decoded 720Hz event can be delayed prior to its
use in the PLD, thereby shifting the entire train of frameswith respect to 720Hz; or the constants used to compare to
the counter output can be changed to vary the inter-frame
spacing. To change the PLD, which has in-circuit
programmability, ALEX tools are used to deliver a new
configuration though the DSP serial port. One of the
SHARCPAC serial ports was used for this task, while oneof the motherboard SHARC serial ports was used totransmit the MDAT signal.
The second task of the Altera part is to take the NRZ dataand clock from the SHARC serial port and encode theminto a Manchester Bi-Phase Mark signal. The SHARC
transmit clock can be programmed to be gated on only
during valid data. The clock edge for sampling data canalso be selected such that the NRZ data and clock are in the
low state between frames. Sequential logic ensures that allcases of clock and data input will result in a 28-bit framewith a low state between frames.
The ALEX SHARC6000 module with customized Altera

circuitry for MDAT transmission will be installed in the
MECAR (Main Injector Extraction Control andRegulation) VME crate, where it will be used to transmit aminimum of six frames. MECAR calculates programmed
current and its derivative, and reads measured current andits derivative, which will be encoded onto MDAT by thismodule. Programmed current is a 32-bit IEEE floating
point value that is broadcast as two MDAT frames. Future
expansion of 32-bit values such as programmed
momentum and its derivative are anticipated.
The ALEX SHARC6000 replaces a complex andinflexible system of ribbon cabling to a suite of CAMAC
modules which have their signals or′d together at a repeater
chassis. The MECAR MDAT suite of frames are output
from the front panel of the ALEX SHARC6000 directly to
the MDAT link repeater. Future expansion only requirechanging the buffer size in SHARC memory andreprogramming the PLD in circuit through the APAC509A
SHARC serial port.
The Time Line Generator (TLG) is a source for one setof Tevatron Clock (TCLK) events known as machine
resets. Reset events determine the operation of theaccelerators for a given period of time called a″supercycle″. The TLG receives its instructions for which
events to generate from a software application. Other types
of TCLK events are line synchronous frequency events,
reflected events from Beam Synchronous Clocks, software
generated immediate events and general events which are
triggered by reset events plus delays. The plan is to replace
the TLG with a VXI module containing a SHARC andAltera parts. A VXI DDS module will be the source of line
synchronous frequency events. A single VXI cratecontaining two modules with SHARC-based architecture
will replace not only the TLG but the entire cadre of timer,priority encoder, parallel-to-serial converter and phase lockfrequency modules which currently create TCLK. SHARC
serial ports utility for receiving and transmitting serial
signals, combined with the ease of programming andaccess to memory, make powerful tool for serial signal
generation.
10 Future directions
As the field of digital signal processing grows, highvolume applications are driving the market. The fastest
growth area in is application specific ICs. These parts often
greatly exceed the capabilities of the general purpose DSP.
One common example of this is the Numeric Controlled
Oscillator discussed in this paper. Another example is the

Digital Radio Receiver developed for the cellular basestation market. The Harris HSP50214 converts a digitized
IF data stream into decimated and filtered baseband data.
It has a 52 MSPS front end process rate while maintaining
a 100dB SFDR. This class of parts may greatly simplify
high performance BPM and other instrumentation systems.
One last technology to watch is FPGAs in combination
with DSP libraries. These are the chameleons of the field,
able to be reprogrammed on the fly. This allows the
hardware to be optimized for the task at hand.
11 Conclusions
In just a few years, technology previously unexplored in
the Fermilab Beams Division has been successfully appliedin a variety of systems for beam control, clock generation,
data distribution, and beam diagnostics. These systems all
have demanding real time requirements.
SHARC
technology initially developed by the LLRF group has
easily migrated to several other Fermilab engineering
departments.
This is evidence for good working
relationships between departments and maturity of the DSP
industry. DSPs are no longer considered to be exotic
technology mastered by a few dedicated experts. Instead,they represent an accessible mainstream engineering tool
for a wide spectrum of applications.
The design of Fermilab DSP based hardware and DSP
software emphasizes powerful functionality in basic
components. The DSP architecture is matched with the
VXIbus specification to create an in-house standard. Thisapproach optimizes module compatibility to encourage andfacilitate reuse of VXI SHARC modules in a variety ofsystems, by greatly reducing new hardware and softwaredevelopment time. DSP functional components are generalbuilding blocks that are combined and configured withbasic control parameters, as needed for specificapplications. These DSP tools offer effective resources formany future projects.
References[1] ADSP-2106x SHARC User′s Manual, First Edition, ©
1995 Analog Devices Inc.
[2] Time and Data Distribution Systems at the Fermilab
Accelerator, Beechy, D. and Ducar, R. Nuclear
Instruments and Methods in Physics Research A247
(1986) pp. 231-238.
[3] Applications of DSP, Enhanced Digital Radio, Ron
Klopfer, DSP & Multimedia Technology, September
1997.