Architecture of the APS Real-Time Orbit Feedback System*
J.A. Carwardine and F.R. Lenkszus
Argonne National Laboratory, 9700 South Cass Avenue, Argonne, Illinois 60439, USA

Abstract
The APS Real-Time Orbit Feedback System is designed
to stabilize the orbit of the stored positron beam against
low-frequency sources such as mechanical vibration andpower supply ripple. A distributed array of digital signal
processors is used to measure the orbit and computecorrections at a 1kHz rate. The system also provides
extensive beam diagnostic tools. This paper describes thearchitectural aspects of the system and describes how theorbit correction algorithms are implemented.
1 Introduction
The APS is the foremost third-generation synchrotron
light source in the United States, delivering intense x-rays
to as many as 35 insertion-device and 35 bending-magnet
beamlines. As with other light sources, orbit stability is
critical in order to achieve the optimum performance forthe x-ray users. At APS the rms orbit motion must notexceed 5% of the beam size, translating to limits of 17µm
rms horizontally and 4.5µm rms vertically.
The APS real-time orbit feedback system provides realtime (dynamic) control of low-frequency orbit
disturbances. The design provides for both global (longspatial wavelength) orbit correction and for local correction
at the x-ray source points, although so far, only the global
system has been fully implemented.
The global feedback system has been in routine operation
with users since June 1997, producing up to 20dB
attenuation of orbit motion in a band up to 30Hz. Typicalorbit motion with feedback is 8µm rms horizontally and3.5µm rms vertically, both measured up to 30Hz.
Ultimately, a bandwidth of 100Hz is desired.
2 Overview
The concept of global orbit correction has been wellestablished in the literature [1,2]. Once a set of beam
position monitors (BPMs) and correctors has been chosen,
a linear ′response matrix′ can be determined that describes
the change in the beam orbit when corrector magnet
currents are changed. Given appropriate systemdimensions, the relationship can be inverted
mathematically, producing a matrix mapping the measured
orbit to a set of corrector currents that will minimize theorbit. This matrix is defined as the ′inverse response
matrix.′ Since the mapping is linear and time-invariant, thesame matrix can be used for static orbit correction that is
applied infrequently and for real-time orbit correction that*cancels dynamic orbit errors.
The APS orbit feedback system makes orbit corrections
at a 1kHz sample rate, using in each plane 160 BPMs tomeasure the orbit and 38 dipole magnets to correct the orbit
errors. In all, 317 dipole magnets (correctors) are available
in each plane, but there are only 38 high-bandwidthcorrectors suitable for global orbit feedback. The remaining
correctors will be used for local correction at the x-ray
source points.
3 Hardware layout
The APS system is entirely digital, using digital signal
processors (DSPs) to perform the orbit correction
calculations and to implement closed-loop feedback oneach corrector magnet.
Since the BPM and corrector hardware are distributed
around the 1.1km storage ring, the orbit feedback hardware
is also distributed, with a total of 20 ′slave′ stations
accessing BPMs and correctors and performing orbit
feedback calculations. A separate ′master′ station controls
the slaves and performs real-time data analysis.
Figure 1 shows an overview of the slave stations. Eachstation receives digitized data from 16 BPMs and sends
digitized setpoints to as many as 32 correctors. There arealso x-ray BPMs available on each beamline that measure
the trajectory of the x-ray beam as it travels from thestorage ring to the users experiment. Each station contains
one or more DSP processors for performing the real-time
calculations. A dedicated fiber-optic ″reflective memory″
network provides deterministic data transfer between
stations. All the stations are connected to the APS controls
network.
Odd Sector
Correctors

Odd Sector
BPM Crate

X-ray BPMs
from previous
stationEven Sector
Correctors

8 BPMs8 BPMsEven Sector
BPM Crate

Feedback 'Slave'
StationReflective Memory Network

to next
stationControls Network

Figure 1: Slave station interfaces.
The global algorithm is divided by output channel (i.e. bycorrector), as illustrated in Figure 2. Each ′corrector error′
is produced from the vector dot product of BPM errors andWork supported by U.S. Department of Energy, Office of Basic Energy Sciences under Contract No.W-31-109-ENG-38

one row of the inverse response matrix. By separating theinverse response matrix into rows, the algorithm is split
into 40 identical calculations, one for each corrector. (Theorbit feedback system has 40 corrector channels of which
two do not actually connect to corrector magnets.) Eachcalculation requires the same vector of BPM values asevery other, but uses a different row of the inverse response
matrix. Each slave station handles the calculations for twocorrectors in each plane (four in total). The resulting′corrector errors′ are subsequently used to generate theupdated corrector setpoints. The channels can be treatedindependently so long as they remain synchronized in time.
corrector 'errors'

row 1

corrector 1

row 2

corrector 2

Station 1/2

BPM errors

inverse response matrix

row 3

Station 3/4

row 4

Station 39/40

corrector 3=

corrector 4

row 39corrector 39

row 40corrector 40

Figure 2: Separating the global algorithm.
4 Slave stations
A more detailed view of the slave stations is shown in
Figure 3. Each station is implemented in a VME cratecontaining a Motorola MVME167 processor that runs
EPICS core routines under VxWorks [3]. Real-timeprocessing is performed with Texas Instruments TMS320
C30 and C40 floating-point DSP processors that reside on
Pentek VME boards. Interfaces to BPMs and correctors areprovided by custom electronics (FSIC, CMPSI, MSI, XRI).
Synchronized 1kHz clock ticks are delivered to all thestations from the APS timing system.
Controls Network

both global and local orbit correction at a 2kHz samplingrate. The two main processors will have direct access to the
VME bus, with each hosting two auxiliary processors overa MIX bus. Currently only the main C30 DSP is installed,
and this is just sufficient to implement global orbit
correction in both horizontal and vertical planes at a 1kHz
rate. It takes the processor about 900µs to complete eachcycle, of which 200µs is consumed in computing the two
vector dot products.
The reflective memory network (supplied by VMIC)
provides an essential link between the slave stations since
all the BPM values must be transferred to all stations inorder to implement the global orbit correction algorithm.
The network is implemented with a ring topology overfiber optic and is accessed like any other block of memory
in VME-space. It provides deterministic data transfer ratesof 29.6Mbytes/second. Despite some reliability problems
with earlier versions of the product, this system has provento be a very effective method of transferring data at highspeeds in a deterministic way.
In addition to transferring BPM errors from slave toslave, the reflective memory is used to transfer real-timequantities from each slave to the master station and toprovide an interchange of control and status information
between the master and slave stations. A flowchart of theslave software is shown in Figure 4.
wait for clock tickread 32 BPM values from hardware
calculate BPM errors
write BPM errors to refl. mem.
wait for other slaves to write BPM errors

read 320 BPM errors from refl. mem.
calculate corrector errors
apply corrector errors to regulator PID

from x-ray BPMs

write new corrector setpoints
Reflective
Memory68040

XRI

write other parameters to refl. mem.

VME BusFigure 4: Slave station flowchart.
C30

C40*

MIX

C30*

FSIC

MIX

C30*

C40*

1kHz clock

P2

C40*

CMPSI

MSI

* Available but not yet installed

to correctors

from rf BPMs

Figure 3: Slave station architecture.
The reason for the mix of C30 and C40 processors is
mainly historical. The C30 processors were purchased
some time ago, before the C40 versions were available.
The more recently acquired C40s were chosen because of
their improved performance over the original C30s.
The figure shows that six DSP processors will be
installed in each slave; these will be used to implement

When a clock tick is received, each slave reads the latest
values from its local BPMs and writes the computed errors
to reflective memory. Once all slaves have completed theirwrites, they all read back the entire vector of 160 BPM
errors (320 for the two planes). Corrector errors are thencomputed from the dot product of the BPM error vector
and the appropriate row of the inverse response matrix. The
corrector errors are then applied to a digital PID regulator
before being sent to the corrector power supplies.
The process of reading 320 BPM values from reflective
memory has proven to be a significant bottleneck, taking
almost 200µs of the available 1ms. The problem isworsened by the fact that VME block transfers are notsupported on the DSP boards used in the slave stations.
Adding a second main DSP processor will allow us tospread the burden of reading the BPM values fromreflective memory. The plan is to separate the task suchthat one main DSP will handle only horizontal orbit
correction and the other will handle only vertical orbit
correction. The two DSPs will be staggered in time, withthe first DSP reading its BPM values while the other waits.
Once the first DSP has finished accessing the reflective
memory, the second will start reading its BPM values.
Thereafter the two DSPs will proceed independently. This
arrangement does mean that there will be a delay between
writing new horizontal and vertical corrector setpoints, butsince the two correction planes are essentially decoupled,
there will be minimal impact from doing this. By splitting
the two correction planes, it will be possible to double thesampling rate to 2kHz despite the bottleneck of reading theBPM values from reflective memory.
The four auxiliary DSPs will be used to implement local
correction at the x-ray source points. At this stage it isunclear whether the local feedback algorithm will be
implemented using ′local bumps′ or using a second inverse
response matrix [4]. Depending on the chosen algorithm,
each slave may have to compute as many as eight local
bumps or 28 additional vector dot products.
5 Master station
The original system design did not include provision fora ′master′ station. This was added in order to simplify thetask of synchronizing the operation of the 20 slave stations.
We have subsequently extended the functionality of themaster in order to include real-time data analysis.
The master station, shown in Figure 5, differs from theslaves in that it has no access to BPMs or correctors andhas a different arrangement of DSP processors.
Controls Network

FSIC

1kHz clock

VME BusC40
C40

6 Real-Time beam diagnostics
Significant effort has gone into providing real-time beamdiagnostics through the orbit feedback system, taking
advantage of the fact that 320 BPM values are collected
and processed at a 1kHz rate. Initially the diagnostics were
aimed at quantifying orbit motion power spectra and orbit
feedback system performance. More recently we havefound the diagnostics equally useful for analyzing transient
orbit motion that dumps stored beam, as a tool for locating
the sources of rms orbit motion.
‘ Dspscope’This is the most elementary diagnostic, providing 40
channels of user-selectable data collection. The system cancollect 4000 data points from a variety of sources,
including any of 640 BPM values and 76 corrector errors.
There are also extensive triggering capabilities. Data isaccessed as EPICS waveform records.
‘AC Voltmeter’

Reflective
Memory68040

such as regulator PID settings and detecting problems withthe slaves via various status flags. For example, if any
control parameters exceed operating limits in one of theslaves, the master turns off all the other slaves in order toprevent dumping stored beam. Each slave also increments a′heartbeat′ register on each clock tick. Missed heartbeats
are detected by the master station and reported to the APS
control system.
The system can operate in several modes including thenormal ′closed-loop′ mode. For example, we can drive anyof the correctors with a synthesized function (e.g., asinusoid) in order to carry out beam-related measurements.
We can also run the orbit correction in ′single-step′ mode
for algorithm checkout.

C40

C40

Reflective
Memory*C40
PMC* Future installation

Figure 5: Master station architecture
The single C40 DSP implements supervisory control andsome real-time analysis. The array of four C40s was added
to increase real-time processing capabilities. The station
relies entirely on the reflective memory network for datatransfer between stations.
The figure shows a second reflective memory board
connected via a PMC bus to the array of four DSPs. This
will be added in order to speed up read access to thereflective memory and will considerably improve theeffective throughput of the four C40 processors.
Supervisory tasks include delivering system parameters

This diagnostic follows the time evolution of a userselectable frequency in various signals. There are actually
two AC Voltmeters, the first operating on the 40 selectablechannels of dspscope and the second operating
simultaneously on all 320 BPM values in either plane. The
320-channel AC Voltmeter is implemented using four C40
DSPs. It takes about 800µs to read the 320 BPM values andcompute one Fourier component in each.
When used with the ′drive corrector′ mode, the AC
Voltmeter provides a fast method for measuring response
matrices, allowing measurement of a 320-BPM by 38corrector response matrix in about ten minutes.
Corrector Error Statistics
At each time step, a running mean and variance of eachcorrector error is computed. These statistics are useful for
detecting malfunctioning of the orbit feedback system itself
and for detecting problems with BPMs.
Corrector Error History Buffers
In addition to computing corrector error statistics, we
also store the last 128mS of each corrector error in acircular buffer that is frozen when stored beam is dumped.
Since the corrector errors provide a measure of the location

of any source of motion, we can often use these history
buffers to localize any source of unwanted beam motionthat results in an unintentional beam dump.
7 Real-Time algorithms
A major consideration in selecting algorithms for the
real-time analysis is that the system is configured to run thesame identical code on each clock tick. Consequently,
algorithms that operate on blocks of time-sequence data are
more difficult to implement. Our algorithms for computing
error statistics and frequency components are all ′sliding′algorithms, updating previous results given the latest data
sample. The algorithms are straightforward to derive andare described here for reference.
The mean value is simply the output of a lowpass IIR
filter. The variance is computed from the square of thedifference between the instantaneous value and the mean,
with a lowpass filter generating the expected value.
The sliding Fourier transform takes advantage of the factthat the previous value of the Fourier component can be
updated given only the latest data sample and the sample
from N time steps earlier (for an N-point discrete Fouriertransform). The equation is as follows:
Y (n) =k[]Y ( n − 1) + x ( n ) − x ( n − N ) . ek− j 2πkNwhere Yk(n) is the kth (complex) Fourier component at time
step n, Yk(n-1) is the previous Fourier component, x(n) is
the latest data input, x(n-N) is the data input at time step (nN), and N is the number of points in the discrete Fouriertransform. It takes N time steps for the calculation to becompleted, but thereafter, the Fourier component is
updated at every time step. The algorithm is very efficient
on a sample-by-sample basis, even though it requiresconsiderably more computations in total than thecorresponding FFT algorithm.
8 Code development
The DSP software is developed on a Unix workstation
using the Texas Instruments TMS3203x/C4x Code
Generation Tools. While most of the code is written in ‘C’,
procedures performing operations on arrays are written in
C-callable assembly language routines in order to improve
performance. The C code is compiled with optimization.
This can cause unexpected behavior, particularly when thecompiler rearranges code to minimize DSP pipeline
conflicts. When this occurs, inspection of the resulting
compiler assembler output is necessary to diagnose andcorrect the problem.

Code is downloaded to the DSP over the controls LAN
from the control system file server using tools in SwiftNet,a product available from the DSP vendor [5]. These tools
run on the VME controls processor under the VxWorks [3]real-time kernel. Our system is configured to download the
DSP software each time a feedback VME crate is booted.
We have not made extensive use of debugging tools.
Rather, we have used reserved ′test′ locations in dualaccess RAM on the DSP to pass debugging information to
the VME controls processor. This method has little impact
on the DSP algorithm and allows us to debug at the normal
feedback system operating speed.
To date we have not used a DSP real-time kernel, buthave chosen to run a single task which linearly executes the
feedback code. Thus we don′t incur any overhead due tocontext switching. We will revisit this issue as we add
additional DSP processing power to the system.
9 Interface to the control system
The DSP operation is controlled and monitored throughdata structures residing in dual-access RAM on the DSP
board. Elements of these data structures are interfaced to
EPICS process variables through EPICS sequence
programs. Values such as BPM readings and computed
corrector values are deposited in a data structure by the
DSP. A sequence program on the controls processor
periodically scans the data structure and deposits values incorresponding process variables. The process variables arecontrol system entities and therefore are accessible by all
the standard control system tools.
A separate EPICS sequence program transfers local
control information such as inverse response matrix rows,
BPM selection, etc., to the DSP resident data structure andsets a flag commanding the DSP to load the new values
into fast local static RAM. Global controls such as
feedback loop open/close, filter cutoff frequencies, etc., arepassed to the slave stations via the reflective memory. The
master station writes global control values to reflective
memory and sets a flag that commands slave stations toread in the new control information.
References[1] “ Synchrotron Radiation Sources - A Primer,” H.
Winick, pp 344-364, World Scientific, 1994.
[2] Krinsky et. al., AIP Conf. Proc. 249 (1992).
[3] Wind River Systems, Alameda, CA, 94501.
[4] Carwardine et. al., Proc. IEEE PAC, (1997).
[5] SwiftNet Node Software for VxWorks 5.x, Pentek
Upper Saddle River, N.J. 07458.