[^1]

Abstract--- In the Fermilab Beam Loss Monitor System, inputs from ion
chambers are integrated for a short period of time and digitized which
are then processed to create the accelerator abort request signals. The
accelerator power supplies employing 3-phase 60Hz AC cause noise at
various harmonics on our inputs which must be eliminated for monitoring
purposes. Due to accelerator ramping, in which both the sampling
frequency and the amplitudes of the noise components change, traditional
digital filtering can partially reduce certain noise components but not
all. A non-traditional algorithm was developed in our work to eliminate
remaining ripples. The sequencing in the FPGA firmware is conducted by a
micro-sequencer core we developed: the Enclosed Loop Micro-Sequencer
(ELMS). The unique feature of the ELMS is that it supports the "FOR"
loops with pre-defined iterations at the machine code level, which
provides programming convenience and avoids many micro-complexities from
the beginning.

*Index Terms*---Digital Data Processing, Embedded System,
Micro-processor, Micro-sequencer, FPGA, Reconfigurable Computing.

# INTRODUCTION

[The]{.smallcaps} new Fermilab Beam Loss Monitor (BLM) readout system
\[1\] is designed to perform several tasks: to provide a flexible and
reliable abort system to protect Tevatron magnets; to provide loss
monitor data during normal operations of the Tevatron, Main Injector and
Booster; and to provide detailed diagnostic loss histories when an abort
happens. Beam losses are detected using ion chambers.

The inputs from ion chambers are integrated for a short period of time,
typically 21 µs, and digitized to 16 bits. The digital data are used to
construct several numbers, i.e., fast, slow and very-slow sliding sums,
which are a measure of the integrated loss over a variety of time scales
up to 64k cycles. The abort request signals for each channel are made in
firmware by comparing these sums as well as immediate measurement with
thresholds. The system abort signal is made by checking the number of
channels and types of abort request signals.

For the Main Injector BLM system, an integration sum for each channel is
accumulated.

In addition to producing abort request signals, the sliding sums are
also readout to monitor the beam loss. However, the accelerator power
supplies employing 3-phase 60Hz AC cause noise at various harmonics on
our inputs which can be larger than the beam loss data in some channels.
Both analog method, i.e., appropriate grounding scheme for the input
cables and digital method are employed for noise reduction. A special
challenge for the digital processing in accelerator systems is ramping,
i.e., accelerating particles from lower energy to higher energy. During
accelerator ramping, both the sampling frequency and the amplitudes of
the noise components change. Traditional digital filtering process was
implemented and it partially reduced certain noise components but not
all. A non-traditional algorithm was developed in our work to eliminate
remaining ripples.

An FPGA firmware core for our sequencing control, called the Enclosed
Loop Micro-Sequencer (ELMS) is also described in this document. The
primary difference between the ELMS and regular
micro-processor/micro-sequencer is that the "FOR" loops with pre-defined
iterations at the machine code level is supported in the ELMS and it is
self-sufficient to run multi-layer nested-loop programs.

# The Digitizer Card

A Digitizer Card (DC) integrates, digitizes and processes 4 channels of
ion channel inputs. The partial block diagram for the FPGA calculating
the sliding sums is shown in Fig. 1.

![](media/image1.wmf){width="3.3513888888888888in"
height="1.7381944444444444in"}

Each input from the ion chamber is integrated by two integrators
alternately in ping-pong fashion. The output voltages of the integrators
reflect the charges due to beam loss which are then digitized by an ADC
device (AD7654AST) about 21 µs per sample and input into an FPGA (with
project name Sums03) for digital processing.

A total of 16 sliding sums are to be kept in the FPGA. (In addition to
the fast, slow and very slow sliding sums, the immediate measurement is
implemented as a sliding sum with sum length=1). They are compared with
corresponding thresholds pre-loaded into the FPGA to produce 16 abort
request signals indication the channel and type of the abort request.
The FPGA also produces several other "de-rippled" values for monitoring
purposes which will be described in detail in later sections. If all
sums were kept using accumulators, the FPGA would easily consume several
thousand logic elements, out of 5980 logic elements in the Altera
Cyclone EP1C6 device we use.

On the other hand, during a 21 µs period, there are more than 1000 clock
cycles at 50 MHz inside the FPGA. Clearly it is more economical to
calculate 16 sliding sums and to perform other tasks sequentially using
one set of data processing resources. The sequence is conducted by the
"Seq128" block with an ELMS block inside.

For the Main Injector BLM system, integrations must be computed. In
order to compute the integrations properly, the pedestal for each
channel is first calculated. At the beginning of each beam extraction
when there is no beam, about 750 ADC values for each channel are
accumulated as pedestal.

For the Main Injector the very-slow sliding sum of each channel has a
sum length of about 64 and now represents a smoothed version of the
input. For each measurement, the pedestal is subtracted from the
very-slow sliding sum with an appropriate scaling and the difference can
be optionally compared with a user-defined value called "squelch level".
If the difference is bigger than the squelch level, the input signal is
considered bigger than noise and then it is added into the integration
sum. Else, the input signal is considered below the noise level and the
integration sum is kept unchanged.

# The De-ripple Process

In addition to calculating the 16 sliding sums and generating
corresponding abort request signals, the Digitizer Card also outputs
various values for monitoring of the beam loss.

The signal cables in the MI tunnel pick up noise generated by equipments
powered by 3-phase 60Hz AC. It contains primarily harmonics of 60Hz,
180Hz and multiples of 360Hz. The noise level is higher than desired
signal and the noise peaks exist at both higher and lower frequencies
comparing with the signal spectrum. A typical set of raw measurement
data and their spectrum are shown in Fig. 2 and Fig. 3.

A natural approach of eliminating noise is filtering. In fact,
calculating sliding sums can be viewed as a digital filtering process.
The fast sliding sum over length of 128 shown in Fig. 2 has reduced
noise level from about 10 ADC counts in raw data down to about 2 ADC
counts. Some beam loss can be seen from fast sliding sum plot.

![](media/image2.png){width="3.4611111111111112in"
height="2.3097222222222222in"}

![](media/image3.wmf){width="3.520138888888889in"
height="2.2881944444444446in"}

However, the sliding sum is a low pass filter with sinc(x) function
shaped frequency responds. The zeros of the sinc(x) is very sharp and it
is hard to align with noise peaks, especially when the sampling
frequency is not integer multiple of 60Hz and when accelerator ramps
which varies the sampling frequency. Also the side lobe of the sinc(x)
is not low enough. Therefore, the plot of fast sliding sum still
contains glitches along with unfiltered 60Hz and 180Hz components.

The de-ripple process further eliminates remaining noise components so
that smaller beam losses become visible. The process takes following
several steps:

1\. Calculating the Cascaded Integrator-Comb (CIC) sums.

2\. Waveform extraction, storage and validation.

3\. Waveform subtraction.

## Calculating the Cascaded Integrator-Comb (CIC) Sums:

The cascaded integrator-comb (CIC) digital filter \[2\] of order N
contains N cascaded stages. Each stage is a moving average filter which
is a CIC filter of order 1. Sliding sum can be viewed as a CIC filter of
order 1 with un-normalized gain. The CIC sums implemented in the sum03
FPGA firmware is the CIC filter of order 2, which can be viewed as the
sliding sum of the sliding sum of the raw data.

The frequency responds shape of the CIC sums is sinc^2^(x) in which the
zeros become the second order ones that provide deeper attenuation to
the noise peaks even though the peaks are not precisely aligned with the
zeros. The side lobes also become lower. In Fig. 4, the sliding sums
(FS) and the CIC sums of a set of typical measurement data are plotted
(with appropriate scales and an artificial offset). It can be seen that
the CIC sums are significantly smoother than the sliding sum.

![](media/image4.png){width="3.5388888888888888in"
height="2.0708333333333333in"}

The CIC sum y\[n\] and sliding sum s\[m\] of input sequence x\[j\] with
sum length K in our work are defined as:

$y\lbrack n\rbrack = \sum_{m = n}^{n - (K - 1)}{s\lbrack m\rbrack}\ s\lbrack m\rbrack = \sum_{j = m}^{m - (K - 1)}{x\lbrack j\rbrack}$

The sum length K for the CIC sums in our firmware is chosen to be
124-128, which brings the first zero to 360Hz. It can be reasonably
assume that the CIC sums are band limited to 360Hz.

In the practical firmware, the accumulations above are implemented
recursively as shown in Fig. 5.

![](media/image5.wmf){width="0.9194444444444444in"
height="1.0868055555555556in"}![](media/image6.wmf){width="1.2201388888888889in"
height="1.054861111111111in"}

Both left and right diagrams in Fig. 5 are valid CIC sum implementations
and the resource usages are comparable. However in our application, we
would like avoid adding a separate storage for s\[n\], given that a
record of up to 64K raw measurement points x\[j\] are available. The
formula for CIC sum is altered as show in the right diagram which
contains two recursive accumulations:

$\begin{array}{r}
u\lbrack n\rbrack = u\lbrack n - 1\rbrack + x\lbrack n\rbrack - 2x\lbrack n - K\rbrack + x\lbrack n - 2K\rbrack \\
y\lbrack n\rbrack = y\lbrack n - 1\rbrack + u\lbrack n\rbrack
\end{array}$

This way, the only additional storage is the intermediate value u\[n\]
which takes one memory space and no additional long record of
intermediate values is to be stored.

## Waveform Extraction, Storage and Validation:

The "de-ripple" process is simply subtracting the noise waveform from
the current CIC sum. The waveform is a record of previous CIC sums of
one period (1/60Hz = 16.7ms) long which are believed not being
contaminated by abrupt beam loss (although a DC or very slow beam loss
is allowed). In the BLM digitizer FPGA firmware, the calculated CIC sums
are directly stored as the waveform data.

Since the CIC sums are band limited to about 360Hz, it is possible to
decimate the y\[n\] sequence to save storage space without losing
information. The decimation counter is a 24-bit accumulator that
increases by 22336 for every input point or about every 21s. The top 7
bits are used as address to the waveform WF storage memories. This way,
128 CIC sum points are stored for the time period of 1/60Hz. The
separation of two decimated points is 5 or 6 raw data points. The
effects of non-uniform decimation are negligible for our application,
although interpolation algorithms are available to reduce the effects.
The block diagram of the de-ripple processor is shown in Fig. 6.

![](media/image7.wmf){width="3.3180555555555555in"
height="1.9027777777777777in"}

The waveform storage memories are split into two pages for each channel.
The CIC sums are written into the page as the tentative waveform which
must be validated through the period. Meanwhile, for each channel, a sum
of the waveform is accumulated to calculate the waveform mean WM value.
After accumulating 128 points for the entire period, the sum is simply
the waveform mean scaled by factor of 128, or 7 bits. This is the reason
of choosing the decimation scheme mentioned above.

Two CIC sums are calculated, the current one y\[n\] and the one a period
before y\[n-L\], where L is the length corresponding to a period which
is about 752. During the period, the absolute value of the difference of
the two CIC sums is constantly compared with a parameter MaxDY. If the
difference between the two CIC sums is bigger than the predefined limit
MaxDY, there may be an abrupt beam loss in the period. The waveform then
is considered invalid. If the differences in the entire period are
within the predefined limit, the waveform becomes valid.

At the end of each period, if the waveform is valid, a 1-bit counter PG
flips, which swap the tentative waveform page to the usable waveform
page. The new tentative waveform is stored in another page until it
becomes valid waveform at the end of another period.

At the initial time after reset, the firmware logic forces the value
(WF-WM)=0. After the waveform of a period become valid, it always
outputs the latest valid one.

## Waveform Subtraction:

Once the waveform become valid, waveform subtraction can be performed to
get the de-rippled sum DR. The stored waveform WF contains DC component
which represents the slow beam loss at the period when the waveform is
recorded. To assure the slow beam loss at current time is corrected
preserved, the DC balanced waveform (WF-WM) is used in the subtraction.
As mentioned above, the WM is the waveform mean accumulated during the
waveform recording period and therefore the mean of (WF-WM) over the
same period is zero or DC balanced. The full de-ripple process is shown
in Fig. 7.

![](media/image8.png){width="3.4256944444444444in"
height="1.729861111111111in"}

In order to see the curves clearly, each curve are added with an offset.
(Or they will all overlap on each other.) The second curve is the CIC
sum with sum length of 128. For reference, the sliding sum FS with same
sum length is also shown as the first trace. The waveform WF in general
takes 3 periods to become valid. In the first 3 periods, WF and WM are
forces to be 0. The de-rippled outputs DR, the bottom curve, first
follow the CIC sum for 3 periods since WF is invalid. Then it becomes a
smooth curve with 60Hz and 180Hz ripples canceled. In the DR sum, both
abrupt and slow beam losses become visible.

# The Enclosed Loop Micro-Sequencer

## Sequence Control Options:

As mentioned earlier the functions in the Sum03 FPGA are performed
sequentially. Sequence control is normally implemented using either
finite state machines (FSM) or embedded micro-processor cores. When an
input data item is to be fed through a fast and very simple process,
typically using a few clock cycles, FSM is a suitable means of sequence
control. FSM also responds to external conditions promptly and
accurately. However, the sequence or program in the FSM is not easy to
change and debug, especially when irregularities exist in the sequence.
Also, the state machines occupy logic elements no matter how rarely they
are used. So it is not economical to use FSM to implement the
occasionally-used sequences such as initialization, communication
channel establishment, etc.

Embedded microprocessor is another option of sequence control. The
drawback of microprocessor is large resource usage. The micro-processor
is a better choice only if a data item is to be processed with a very
complicated program, typically using thousands of clock cycles.

When a data item is to be processed with a medium length program, e.g.,
using a few hundreds clock cycles, a micro-sequencer become a better
option. We have developed a micro-sequencer in our FPGA called the
Enclosed Loop Micro-Sequencer (ELMS). The primary difference between the
ELMS and regular micro-processor/micro-sequencer is that the ELMS
supports "FOR" loops with predefined iterations at the machine code
level and is self-sufficient to run multi-layer nested-loop programs.

## Description:

A detailed block diagram of the ELMS is shown in Fig 8. The program is
stored in a 36-bit x 128-word ROM in our example. Clearly the
instruction width and memory depth can be flexibly chosen for different
applications if it is necessary. Also, ROM's in FPGA are typically
implemented with dual-port random access memories (RAM's), which allows
the users to overwrite its contents so that new programs can be loaded.
However, if the program is not to be changed during operation, a block
memory organized as a ROM with the program pre-stored is more
convenient.

![](media/image9.wmf){width="3.3666666666666667in"
height="2.4611111111111112in"}

Both unconditional and conditional branches are supported as in regular
micro-processors. We have used non-pipelined branch logic in our example
for simplicity.

The Loop & Return Registers (LRR) along with a 128-word stack are the
primary elements designed to support the constant iteration "FOR" loops.

Some ELMS instructions are shown in Table I.

The ELMS instructions are 36-bit words. When any of the bits 32-35 is
set, the word represents a program control instruction. Otherwise, it is
treated as a user instruction. In the ELMS, the only build-in
instructions are the program control instructions. All other
instructions can be freely defined by the users.

## The Branch Instructions

The unconditional branch instruction JMP is implemented as in typical
micro-processors. When bit 35 is set, bit field desA (Only lower 7 bits
are used in our example.) is selected as the PC for next clock cycle.

The conditional branch instruction JMPIF is signified when bit 32 is
set. An input line CondJMP is supplied from external user logic as the
branch condition, i.e., the PC jumps to desA only when CondJMP is high.
The branch condition in the ELMS is treated as a result from the
external data processing resources. It is the users' responsibility to
generate this signal and assure that it is valid when reaching the
conditional branch instruction. This design arrangement allows us to
avoid using an ALU in the sequencer.

In the non-pipelined design, the branch logic is the most latency
critical part. When a JMP or JMPIF instruction is present at the output
of the ROM, the signals must flow through several layers of
multiplexers, arriving at the address registers of the ROM with
sufficient setup time. We have been able to compile the non-pipelined
design in an Altera Cyclone FPGA device EP1C6Q240C6 \[3\] with a 153 MHz
maximum operating frequency.

To increase the operating frequency further, a pipelined design can be
used, i.e., assigning registers on both input and output ports of the
ROM. We have compiled a pipelined version in same device with a 250 MHz
maximum operating frequency. However, a pipeline bubble (no-op
instruction) or out-of-order time slot must be added after the JMP or
JMPIF instructions.

In our application, the clock inside the FPGA is 50 MHz. That's why we
chose a non-pipelined design in our example.

The branch instructions are to be used only when it is necessary.

## The FOR Instruction

Supporting FOR loops with predefined iterations at machine code level is
a special feature of the ELMS.

When bit 33 is set, the instruction starts a FOR loop in which the bit
fields BckA, EndA and cnt are pushed into the corresponding LRR/stack.
The PC is incremented until reaching EndA, and then it is set back to
BckA. This continues for (cnt+1) passes. Then the stack is popped on the
last pass of the loop.

A program segment with a FOR loop may look like the following:

FOR BckA1 EndA1 5

*Initialization Processes*

BckA1

*Repeating Processes*

EndA1

After the FOR instruction, the instructions before PC = BckA1 are
executed once, essentially serving as initialization. Then the
instructions between PC = BckA1 and EndA1 (inclusive) are executed
(cnt+1) or 6 times in this example. Note that there is no conditional
branch instruction at EndA1. The ELMS conducts the loop sequence by
itself.

Another interesting point is that the LRR + stack structure appears like
a Branch Target Buffer (BTB) in advanced micro-processors \[4\]. Indeed,
the LRR + stack stores information of the targets to be branched to.
However, the PC jumps in ELMS are pre-defined by the FOR instruction and
are not base on predictions. The sequencing performance of the ELMS is
deterministic rather than statistical.

## The CALL and RTN Instructions

The CALL instruction is implemented as a combination of the FOR and JMP
instructions with cnt automatically set equal to 1. At the CALL
instruction, the PC jumps to desA while BckA and EndA are pushed into
the LRR/stack. When PC reaches EndA or when a RTN instruction is seen,
the PC jumps back to BckA and the stack is popped. Note that in addition
to a regular return instruction, the return point from the subroutine is
also pre-defined to be EndA, which allows an alternative means of
subroutine return that provides extra convenience.

A program segment with CALL/RTN instructions may look like the
following:

CALL BckA1 EndA1 DesA1

BckA1

*Processes after Subroutine Return*

DesA1

*Subroutine*

EndA1 RTN (optional)

After the CALL instruction, the PC jumps to DesA1 to execute the
subroutine. Once PC reaches EndA1, it returns to BckA1. The instruction
at EndA1 needs not to be RTN. Therefore any program segment can be
called as a subroutine.

The RTN instruction is provided primarily for possible early returns in
the subroutines. The RTN instruction may also be used when early breaks
are needed in the FOR loops.

## Nesting Loops

Multi-layer FOR or CALL loops can be nested. When an inner layer starts,
the parameters of the unfinished outer loop are pushed into the stack,
which allows the outer loop to continue after the inner loop finishes.

Note that in the FOR loops, inner loops can be nested not only in the
repeating processes, but also in the initialization processes. This
design arrangement provides convenience for the programmers when
subroutine calls or FOR loops are needed in the initialization, such as
presetting an array.

Up to 128 layers of loops can be nested. It is the users' responsibility
not to nest more than 128 layers of loops. This should be sufficient for
practical applications. For example, if 64 layers FOR loops, each
iterating 2 times, are nested together, it will take the sequencer to
complete more than 2000 years at 250 MHz operating frequency.

## The User Instructions

When the bits 32-35 of the instruction word are all 0, the word
represents a user instruction. The users have maximum flexibility to
define their own instruction sets based on the application. We would
like to present the instruction set we used for the Fermilab BLM system
as an example shown here in Table II.

A user instruction contains four instruction fields, 4 bits each: SEQA,
SEQB, SEQC and SEQDQQ and two address/data fields: ADH and ADL. Each
instruction field is decoded into up to 15 control signals with names
shown in Table II. The SEQDQQ are delayed by a pipeline step before
being decoded. The control signals generated from SEQDQQ are essentially
register enable signals for reading out contents from the Parameter RAM
and the Sum Keeping RAM that are registered on input port. The ADH and
ADL field are used to provide addresses for memory access or to specify
initial values for some registers.

Sometimes, several control signals must be turned on simultaneously.
While defining the instruction set, signals that might be turned on
simultaneously are carefully assigned into different column in Table II.

## Sample Codes

The ELMS codes for calculating 16 sliding sums in our application are
shown in Table III.

After reset, the PC starts from 00. The sequencer runs into a dead loop
at PC = 03. The unconditional instruction, JMP to 03 is "executed" every
clock cycle. However, there is no bit flipping at all. The sequencer and
the logics it controls are effectively in a sleep mode that consumes no
dynamic power.

When an external "do sums" signal arrives, the "RUNat04" signal in Fig.
8 is turned on for a clock cycle that forces the PC to become 04. The
ELMS then goes through the sequence of calculating the sliding sums. The
FOR instruction at PC = 07 sets the outer loop for 4 types of the
sliding sums (immediate, fast, slow and very-slow). Then the FOR
instruction at PC = 0A sets the inner loop for 4 input channels. The
type and channel of the sliding sums are indexed by two counters that
are initialized and incremented by the SetType, IncType, SetCh and IncCh
instructions, respectively.

The "compiler" we used is a Microsoft Excel spread sheet. The search and
index functions are used to find labels and instructions. Each row is
composed as a 36-bit integer in the column "code". The columns "PC" and
"code" are taken into another worksheet which is then saved as a text
file. The text file can be directly used as a "memory initialization
file" that specifies the ROM contents in the FPGA.

# FPGA Implementations

The firmware with project name Sums03 that calculates the sliding sums,
generates abort request signals and performs de-ripple functions has
been implemented in a low cost FPGA device, EP1C6Q240C6. The ELMS has
been used in the block Seq128 in the FPGA for sequence control. For
evaluating the idea of the ELMS, a bare ELMS circuit plus three 8-bit
accumulators has also been compiled and simulated in a test project
ELMS1 using same FPGA device. Compiled results are shown in Table IV.

It can be seen that the resource usage of the ELMS is very small,
leaving most portions of the FPGA for data processing functions defined
by users. As a result of using ELMS, a significant portion of the
resources for calculating the sliding sums and the integration sums are
reused multiple times for each measurement. Without resource reusing,
the whole function would not fit our FPGA.

It is possible to find off-the-shelf micro-processor/micro-sequencer IP
with comparable resource usage. However, the supporting of predefined
FOR loops at machine code level is a special and convenient feature of
the ELMS. In fact, because of its simplicity, the ELMS itself can become
an off-the-shelf solution for future projects. On the other hand, it is
not difficult to add the FOR loop support to the future version of
existing IP.

Again because of the simplicity, it is very easy to compile the ELMS to
a high operating frequency. The non-pipelined and pipelined versions of
the ELMS1 project are compiled with 153 MHz and 250 MHz maximum
operating frequencies, respectively, where 250MHz is the upper operating
limit of the M4K memory block in the device. The project Sums03 does not
need a high operating frequency since its internal clock is only 50 MHz.
The Sums03 project is compiled with maximum operating frequency 61 MHz.

# Discussion

Several design considerations of the firmware are to be discussed in
this section.

## Several Remarks about the De-ripple Process

Traditional digital filtering performs well on eliminating high
frequency noise. In our firmware, we used CIC filter of order 2, i.e.,
the CIC sums to eliminate noise above 360Hz. To eliminate low frequency
components, however, traditional digital filter will need sufficiently
long record usually many (1/60Hz) periods of measurement data. In our
system, there are not very many periods after a reset and the amplitudes
and phases of the 60, 120 and 180 Hz noise components can be different
over a few (1/60Hz) periods before and after the accelerator ramping.

Therefore, a noise waveform subtraction approach is chosen for our
de-ripple process in which a period of noise waveform is stored for
subtraction from the later measurements. Since the noise components many
change over a few periods, the waveform is updated constantly. To assure
the waveform is free of abrupt beam loss, it is being validated while it
is recorded using periodic condition (assuming the abrupt beam loss is
not periodic). Finally, the effect of DC beam loss in the waveform is
canceled by subtraction of the waveform mean from the waveform.

## The Sequencer without Data Processing Resources

In history, there are computers employing the "Harvard" architecture
\[5\] in which storages of program and data are physically separated.
Most of today's general purpose micro-processors use the "Princeton"
architecture in which the program and data are stored in the same
external memory. However, inside the micro-processor, the program and
data are usually stored in separate caches and at this level it is the
Harvard structure again.

In the ELMS, the data and program are further separated beyond the
Harvard architecture. A micro-sequencer is not a CPU since the sequencer
itself does not have capabilities for general purpose data processing.
The micro-sequencer controls external data processing resources by
toggling control signals.

In FPGA computing, this arrangement allows maximum flexibility in the
data domain. The widths of data words, addressing modes and number of
processing channels etc. can be chosen by the designer without any
restrictions as in general purpose micro-processors.

Without data processing resources, conditional branches are harder to be
implemented and therefore are discouraged in ELMS while loops using the
built-in FOR loop support are encouraged. On the other hand, because of
the FOR loop supporting in ELMS, a farther separation of program and
data than the Harvard architecture becomes practical and feasible.

## Predefined Iteration FOR Loop Support

Using loops in the program is a primary means of code reusing.
Supporting block-styled predefined iteration FOR loops without using a
conditional branch instruction is a unique feature of the ELMS. Of
course, the ELMS must still support conditional branch instruction JMPIF
since the FOR loops can only replace conditional branches in many but
not all instances.

In advanced micro-processors, branch penalty \[6\]\[7\] becomes more
serious as the pipeline becomes deeper and deeper. An attempt to solve
the problem is by using branch prediction with additional resource and
there are good algorithms in this area.

When a FOR loop with predefined iteration is programmed, the execution
route including where and how many times to loop is determined in
advance. There should be no branch penalty at all. However, when using
conditional branch to conduct the loop, the originally known sequence
becomes unknown and the branch condition must be evaluated each time the
end of the loop is reached. With FOR loops available at the machine code
level, it helps to ease the branch penalty problem.

In an FPGA, the clock speed difference between pipelined and
non-pipelined ROM is not very significant. In cases as in our example, a
clock frequency as low as 50 MHz is sufficient which makes non-pipelined
structures more preferable. The benefit of the FOR loop on reducing
branch penalty is therefore not very obvious. Nevertheless, the FOR loop
is still a convenient program instruction to achieve silicon resource
and code reuse.

In practice, indexes must be kept in loops to distinguish different
passes of the loops. In the ELMS, the pass counter for the FOR loop can
be viewed as an index. However, we chose for the user to implement
external user indexes rather than supporting them inside the
micro-sequencer. The pass counter is in the program domain while user
indexes belong to the data domain. It is more convenient for the users
to specify the parameters of the index counters such as number of bits,
incremental difference, reset or preload features etc.

In our example, the number of iterations "cnt" is an immediate value
that comes with the FOR instruction. However, there is no fundamental
reason why this value can not be stored in a user register. This way,
FOR loops with a variable number of iterations can be supported, which
is very useful in applications like matrix computation.

## Software Issues

The operation of the ELMS is conducted by a pre-stored program. Just as
in micro-processor computing, the software must be appropriately coded
and compiled for given computing tasks. Based on experience of
micro-processor computing, it is known that software engineering could
become a major effort in certain tasks.

In many cases complexity of software is only partially necessary for the
computing tasks and is partially artificial, essentially due to
complexity of the hardware or firmware. Therefore, the best way to
reduce software complexity is to simplify the hardware or firmware
design.

The architecture of the ELMS is directly reflected in its instruction
set. There are only a handful program flow control instructions that are
native to the ELMS. All the remaining ones are user instructions that
are application specific. Unlike in micro-processors that the users code
a program using an existing instruction set, in the FPGA with the ELMS
the users design the instruction set as well as program them into the
desired sequence.

For our practical design, we have used spread sheets as our tools for
keeping track the instruction set design, program coding, compiling as
well as documenting. This way, the effort of software design is
controlled within a reasonable fraction of the entire work.

# Conclusion

The firmware in the digitizer card for the Fermilab BLM system has been
commissioned and operates with specified functions.

The de-ripple process described in this document can be a useful noise
elimination tool for systems with periodic noise and non-periodic
signals. The signals can be brief abrupt ones or slow moving ones.

The ELMS provides an option of sequence control in an FPGA with very low
resource usage. It has been used for the Fermilab BLM system with
specified performance and a flexible reprogramming ability.

The FOR loop support in machine code level in the ELMS also provides
some hints on fighting branch penalty problems for advanced
micro-processor development. Clearly, there is a whole array of
associated issues that must be studied in the future.

# References {#references .Reference-Head}

1.  C. Drennan, et. al., "Development of a new data acquisition system
    for the Fermilab beam loss monitors," in *Nuclear Science Symposium
    Conference Record*, Date: 16-22 Oct. 2004, Pages: 1816 - 1819 Vol.
    3.

2.  R. Lyons, *Understanding Digital Signal Processing*, 2nd ed. Upper
    Saddle River, NJ: Prentice Hall, 2004.

3.  Cyclone FPGA Family Data Sheet, Altera Corp., San Jose, CA, 2003
    \[Online\]. Available: <http://www.altera.com/>

4.  G. Hinton, et. al., "The Micro-architecture of the Pentium 4
    Processor," in *Intel Technology Journal*, Vol. 5 Issue 1 (February
    2001).

5.  J. Hilburn & P. Julich, *Microcomputers/Microprocessors: Hardware,
    Software and Applications*, Englewood Cliffs, NJ: Prentice Hall,
    1976.

6.  D. Comer, *Essentials of Computer Architecture*, Upper Saddle River,
    NJ: Prentice Hall, 2005.

7.  Arvind et. al., "6.823 Computer System Architecture" MIT Open Course
    Ware, \[Online\]. Available:
    <http://ocw.mit.edu/OcwWeb/Electrical-Engineering-and-Computer-Science/6-823Fall-2005/CourseHome/>

[^1]: ^^Manuscript received April 30, 2007. This work was supported in
    part Operated by Universities Research Association Inc. under
    Contract No. DE-AC02-76CH03000 with the United States Department of
    Energy.

    The authors are with Fermi National Accelerator Laboratory, Batavia,
    IL 60510 USA (phone: 630-840-8911; fax: 630-840-2950; e-mail:
    jywu168@ fnal.gov).