**Tevatron Ion Profile Monitor (IPM) Project**

**IPM Front End (IFE)**

Kwame Bowie

August 10, 2005

Revised: December 9, 2005

**Introduction:**

This document will present the current state of the multi-channel QIE
board designed for the Tevatron IPM project. The motivation of this
document is to describe the board in its present state. This document
will discuss the board's architecture, and its operation within the
Tevatron IPM DAQ system.

**Tev IPM DAQ System Architecture:**

The IPM Front End (IFE) board is the front end portion of the IPM data
acquisition (DAQ) system. The following shows in block diagram form the
basic architecture and data flow of the DAQ system:

![](media/image1.wmf)

The IFE board is a single element of the Tev IPM DAQ system. The Tev IPM
DAQ system may be thought of as two separate subsystems: upstairs and
in-tunnel. The upstairs portion of the DAQ system all resides in a
single PC. The upstairs PC houses the Timing Card as well as the Buffer
Boards and implements all of the intelligence and control for the DAQ
system. The in-tunnel portion of the DAQ system all resides in a single
6U x 160mmEurocard crate. The in-tunnel portion includes sixteen (16)
front end cards as well as a single fanout card. The in-tunnel portion
intentionally has very little memory or configurable functionality
because it will sit very close to the beam line in a moderately high
radiation dosage area. For that reason, all components of the IFE and
fanout cards are designed to tolerate reasonable dosage levels, and
unnecessary logic is not present in the design. The remaining sections
of this document will discuss the IFE board design and functionality in
detail.

**IFE Specifications:**

The following specifications have been developed as the result of
several meetings with various members of the project group. The
specifications remain somewhat malleable, until the full IPM system has
been calibrated. This document breaks the specifications down into two
categories: functional and performance.

**Functional Description:**

-   Digitize the charge data of 8 QIEs operating in both calibration
    mode and normal non-inverting mode.

-   Relay the following flags to closest QIE clock cycle:

    -   Proton

    -   Pbar

    -   Proton Injection

    -   Antiproton injection

    -   Error bits

    -   Crossing(clock) counter

    -   Proton bunch counter

-   Multiplex data for serialization via an optical link

-   Integrate charge at rate of 2RF/7

> **Performance Requirements:**

-   Operation reliable up to 20 krad total ionizing dose (TID)

-   Noise level: 0.5 -- 1LSB RMS (1-2 fC)

-   Minimum signal (resolution): \~1 fC

-   Provide additional 4- 26 bits of header data for each integration
    cycle

-   High speed serial data out using optical link \@1.6 Gbps

![](media/image2.jpeg){width="2.863888888888889in"
height="2.686111111111111in"}

**IFE Board Architecture**

The block diagram below illustrates the main functional elements of the
IFE board. The following sections will provide more detail about each of
the blocks.

![](media/image3.wmf)

**QIE biasing and interface**

The core element of the charge integration and digitzation for the IFE
board is the QIE8 developed for CMS by Fermilab engineers. The principal
designer T. Zimmerman has authored several documents available for
review which provide a very detailed description of the operation and
performance of the chip. The QIE is an amazing chip that can operate in
many different configurations in order to support many different
detector signal sources. In this section, we will discuss the
configuration that we have selected for the QIE. The table below
illustrates the all of the possible modes of operation of the QIE.

  ----------------------- ----------------------- -----------------------
  **Range mode**          **Charge Polarity**     **Input Impedance**

  Calibration Mode        Non-inverting           50 ohm

  Normal Mode             Inverting               90 ohm
  ----------------------- ----------------------- -----------------------

The QIE may operate in non-inverting mode or inverting mode. Inverting
mode is designed to be used with positive current source detector
elements such as HPDs. The IFE board is designed to run with the QIE
operating in its non-inverting mode. The non-inverting mode of the QIE
is useful for detector elements generating negative current pulses such
as PMTs or similar devices. The micro channel plates (MCPs) collecting
ionization electrons provide a similar type of pulse when configured for
the IPM.

The QIE also has selectable input impedance. One may select between
90ohm or 50 ohm termination inside the QIE chip. The IFE board utilizes
the 50ohm input termination for the QIEs amplifiers. The selection of
the 50ohm termination was primarily due to cabling and connector
concerns.

The QIE integrates charges with a dynamic range of roughly 14 bits in
its NORMAL mode of operation. Although there are 14 bits of dynamic
range, the data is condensed into a single 9-bit data word with
non-linear transfer function. Although large dynamic range is quite
useful in many applications, the signals of interest for the Tev IPM
project reside in a fairly small window at the lowest end of measurable
charges for the QIE. In fact, quite frequently the signals of interest
are below the threshold of the LSB in NORMAL mode. The QIE has a
CALIBRATION mode in which the voltage vs. charge curve is linear with a
slope of \~0.9fC/ count. In many situations, several channels per board
will be expected to operate in NORMAL mode and the remainder in
CALIBRATION mode. The IFE board provides a single 8-bit DIP switch
between the two backplane connectors that enable switching to NORMAL
mode. If the DIP switch is not set, then the QIE on the associated
channel will always operate in CALIBRATION mode.

In addition to the mode settings, it is important to bias the QIE
correctly for the signals of interest and the selected sampling rate.
The QIE8 chip has a bias setting resistor that sets the bias currents
for all of the internal transistors and amplifiers. By decreasing the
bias, one may integrate over a longer period of time without saturating
the output. A resistance of 750Kohms provides an ideal bias for the
selected sampling rate of roughly 15MHz.

The QIE is capable of integrating and digitizing charge at a better than
40Mhz sampling rate. This high throughput may only be accomplished by
using a multi-cycle pipeline. The QIE has a four-cycle pipeline with
four identical and parallel charge integration pathways. Unfortunately,
these identical pathways have slightly different characteristics due to
wafer variations that simply cannot be perfectly matched and controlled.
Since these pathway-to-pathway variations will remain nearly constant
over time, a great increase in accuracy may be extracted by calibrating
each of these pathways. In order to calibrate these pathways, we must
know which pathway each sample represents. The QIE provides this pathway
information in the form of two additional bits for each sample called
CAPID bits. Without these CAPID bits and a full calibration, the QIE
charge measurement accuracy is of order 25% while with a full
calibration, the accuracy of 5% or better is achievable.

**Timing/Control Link:**

The timing/control link originates from the timing card in the upstairs
subsystem. The timing/control link signals are sent from a PECL driver
in the upstairs subsystem to the in-tunnel subsystem on a single CAT5
cable. Once the signals reach the tunnel, the signals are duplicated and
passed to the IFE boards by the Fanout Card on custom modular cables.
The timing/control link of each IFE is composed of 4 beam-synchronized
PECL pairs signals. The signals are listed below:

-   CLOCK (2RF/7)

-   PROTON marker

-   PBAR (antiproton) marker

-   CONTROL (5-bit serial encoded)

The PROTON and PBAR markers are provided by the timing card to ease the
load on the Buffer Board triggering logic. The Tevatron operates with a
known beam structure and bunch spacing. Relevant charge signals are only
expected during the known bunch crossings. Providing beam-synchronized
PROTON and PBAR signals to the Buffer Board allows these signals to used
as a trigger for storing data. Without the PROTON and PBAR signals, the
Buffer Board would need to trigger on data in a possibly very complex
manner. The two markers provide a very accurate and very simple trigger
for data storage.

**Clock Distribution:**

![](media/image4.wmf)

The clock distribution on the IFE board includes two (2) asynchronous
clock domains: the system or QIE clock, and the serializer clock. The
distribution of each of these clocks to the relevant portions of the
board is discussed in this section

The clock distribution scheme for the QIE clock is based on a tree of
point-to-point links as shown in the diagram. The fanout card
distributes beam timing information to each front end via a dedicated
point-to-point link. Each IFE board then duplicates the system clock
signal nine (9) additional times and sends the clock signal to the eight
(8) QIEs and an FPGA via point-to-point links. The total skew between
all clocks is less than +/- 2 nanoseconds.

The GOL serializer requires a very pristine clock signal. The GOL
expects a clock with less than 150 ps of jitter peak-to-peak. When
designing the clock distribution, each layer of logic adds a small
amount of jitter to a clock signal. The GOL's strict clock jitter
requirements suggested that minimum logic be placed between the origin
of the serializer clock and the GOL chip. The simplest solution turned
out to be placing a low-jitter clock on each board with a point to point
link to the GOL.

**Data Serialization**

The IFE board uses the GOL as its serializing circuit. The GOL is an
ASIC developed for the CMS project at CERN. The GOL has multiple
operating modes, but we will operate constantly in the high speed mode
allowing serialization at a bit rate of 1.6 Gbps. The GOL may be thought
of as a high speed parallel-to-serial shift register that receives a
32-bit input and transmits a 1.6Gbps output.

As the IFE board and DAQ system will be based upon two asynchronous
clocks, it is very important to understand the relationships between
these two clocks and its impact on the data. The two clocks that must be
present for the system to operate reliably are the system clock and the
serializer clock.

The system clock is based upon the tevatron RF cycle. The frequency of
the system clock is actually two-sevenths of the tevatron RF frequency
or 15.17 MHz. This beam-synchronized clock provides us our time
reference point for our signals of interest. Unfortunately the frequency
of this clock varies slightly with the Tevatron's mode of operation.
This variation in the system clock makes it unsuitable for use as the
serializer clock due to its frequency variation.

To enable reliable serialization, a separate high-precision crystal is
located on the IFE. The frequency of this crystal is defined by the
bit-rate of the serializer and for our purposes is a factor 40 times
smaller than the bit rate of serialization. The GOL ASIC accepts a 40MHz
clock from these crystals and serializes data at the bit rate of
1.6Gbps.

Since this system is based on two asynchronous clocks that are not pure
harmonics of one another, there is a definite need to ensure that data
is not lost or re-transmitted. Either situation makes the data
acquisition a more complex problem. To make the receiving of the serial
data the simplest, only relevant data will be transmitted. Any stale
data will not be transmitted during its respective serializer clock
cycle. Instead an idle (K28.5) character will be sent, which will notify
the receiver that the previous word was not valid data. One requirement
that will be made is that there must be a fixed-length data frame
format. This simplifies the encoding and decoding processes by not
requiring large amounts of memory and decision circuitry. The fixed
length data frames may be separated by IDLE characters of arbitrary
length.

Over time the two clocks will be related in a semi-periodic way, and
this relationship will be discussed below:

**[Digitizing 8 QIEs @ 1.6 Gbps using GOL ASIC:]{.underline}**

For an 8-channel board using the CERN GOL chip for serialization at
1.6Gbps, the serializer clock would be 40 MHz. The relationship between
these two clocks is such that it will take several cycles of the system
(QIE) clock for there to be an extra serializer clock cycle. The
calculations based upon ideal clocks (no frequency variation) would look
as follows:

Clock period relationship:

System clock period = 1 / 15 MHz = 65.908 ns

Serializer clock period = 1 / 40 MHz = 25 ns

Ratio of Serializer clocks per system clock period = 2.64:1

Data bus width relationship

QIE data bits generated each QIE clock = 58 bits (keeping only 1 CAPID)

GOL data bus width = 32 bits

Ratio of bits transferred per clock cycle = 1.81

These two relationships must be used when designing a fixed-length data
frame format. The clock period ratio of 2.64:1 defines the fixed clock
relationship and as a result, the maximum ratio of output (serializer
clock) data words-to-input (qie clock) data words. This ratio of course
refers to integer number of input data words and output data words. It
is possible to transfer less information than this ratio by utilizing
IDLE characters. However, it is beneficial to utilize as much bandwidth
as possible. The data per cycle ratio of 1.81 defines the minimum ratio
of serializer words to QIE words to ensure that there is not an overflow
and loss of relevant data. There are really three tradeoffs that must be
balanced when designing the data format:

1)  Bandwidth utilization

2)  Ease of encoding/decoding

3)  Information for each QIE clock cycle

The two ratios define the upper and lower bounds of the ratio of
serializer clock cycles per qie clock cycle of the fixed length data
frames. The following examples will demonstrate some of the tradeoffs.

**Single QIE clock cycle design:**

> The use of a single QIE clock cycle means that only integer ratio
> values are achieveable. As a result, a ratio of 2 provides the only
> solution that lies within the ratio boundaries. Thus the data frame
> has a length of 2 and it transmitted after every QIE clock cycle. The
> amount of data sent in this case is 64 bits, so 6 additional bits are
> provided every data frame. However, the ratio of 2 means that a
> significant portion of the time, IDLE characters are being sent and
> bandwidth is being wasted. Effectively, on average
> 2\*2RF/7\*32bits/cycle = 971 Mbps are being transmitted as the data
> rate. This data rate amounts to only 76% bandwidth utilization.
>
> **2 QIE clock cycle design**
>
> The use of 2 QIE clock cycles provides two acceptable solutions: a 4
> word data frame and a 5 word data frame. These two solutions provide
> ratios of 2 and 2.5 respectively. The second solution is much
> preferred as it provides the maximum in terms of bandwidth utilization
> and extra data per QIE clock cycle. Using a 5 word frame, the amount
> of extra bits per QIE clock is as follows:

Extra bits per QIE clock =
$\frac{(5\frac{\text{words}}{\text{frame}}) \ast (\text{32}\frac{\text{bits}}{\text{word}}) - (2\text{cycles}) \ast (\text{58}\frac{\text{bits}}{\text{cycle}})}{2\text{cycles}}$

Extra bits = 22 per QIE clock cycle

> Using the 5-word frame also yields a high bandwidth utilization of
> 1.21 Gbps of a possible 1.280 Gbps or 94 % utilization.

As you can see, using a 5 word frame based upon 2 QIE clock cycles
provides a good solution. An even better utilization of bandwidth may be
achieved by utilizing a 13 word frame based upon 5 QIE clock cycles (2.6
clock ratio, 98% bandwidth utilization, 25.2 bits per QIE clock).
However, as the data frame grows larger, more memory is required to
build and decode the frame and a more complex state machine is required
to control the frame builder and the frame decoder. In addition, the
decoder on the buffer board requires more memory and logic.

The frame length that the IFE currently implements is a 7-word frame
based upon 3 QIE clock cycles. This solution provides for frequent IDLE
characters, a simple encoder and decoder, and low memory requirements.
The data frame format is shown in the diagram below.

![](media/image5.wmf)

**Data Synchronization:**

The IFE board accepts charge signals for eight (8) channels as its input
and generates a high speed data stream as its output. These signals are
always running and are presented to the receiver boards without
interruption. The IFE board from a functionality perspective is simply a
time multiplexing and digitizing board. There is very little memory and
no trigger generation currently present in the IFE board design. It is
critical that IFE board data bytes may be associated with the correct
Tevatron events. To ensure that data can be correlated with Tev events,
synchronization must be accomplished.

There are essentially two requirements to achieve synchronization
between the different IFE boards in the Tev IPM DAQ system. The first
requirement is at least one common time reference. The timing control
link provides the common clock for all IFE boards. The second
synchronization requirement is that there must be a synchronization
input. The synchronization input is implemented with the CONTROL signal
within the timing/control link. The RESET command serves as the
synchronization command as it ensures that all QIEs begin integration
and transmission during the same Tevatron event.

Although all IFE boards begin integration and transmission during the
same Tevatron event, the data links on each IFE board will operate at
slightly different data rates. The key element to the synchronization is
the FIFO internal to each IFE and ability of the serializer to send IDLE
characters.

Since the actual data throughput is much less than the link data rate,
it is possible to transmit IDLE characters where stale data would be.
The Buffer Board then stores only the valid data and not the IDLE
characters to memory. The differing data link rates for the different
IFE boards means that some boards will have more IDLE characters, but
all boards will have the same amount of valid data. If only the valid
data is stored to memory and the first word from each IFE board is
aligned, and no data is lost, then the data is guaranteed to be aligned
and synchronization is achieved. The main challenge of this
synchronization technique is maintaining a solid lock on the data link
thus never losing data.

The Buffer Board must align the data from all IFE boards to synchronize
the boards. To align the data from the boards, there must be a known
time reference. There are several markers as well as several counters
that help accomplish this task.

.

**FPGA Design:**

The FPGA design for the IFE board has two key functions: mode control
and multiplexing/frame building. The FPGA has a few other
responsibilities, but these two are the key components of the FPGA
design.

The mode control portion of the FPGA design simply decodes the command
sent on the CONTROL signal from the timing/control link. The control
signal has available 8 different commands. The commands are encoded such
that only commands starting with a "11" will trigger the decoding
circuitry. The table below lists the commands that the IFE board
implements and the associated codes that must be present on the CONTROL
signal in five (5) consecutive system clock cycles.

**Code Function**

  ----------------- -----------------------------------------------------
  11000             No effect

  11001             Proton Injection Marker

  11010             Antiproton Injection Marker

  11011             QIE Mode0 (QIE in Calibration Mode)

  11100             QIE Mode1 (QIE in Normal Mode)

  11101             QIE Mode2 (Link test - outputs a 32-bit counter to
                    GOL)

  11110             QIE Mode3 (Reset QIEs)

  11111             IFE Board Reset
  ----------------- -----------------------------------------------------

The functions are implemented such that there must be a one-cycle
waiting period between sending back-to-back commands.

The other function of the FPGA is to implement the multiplexing and
frame building components scheme. The implementation of the multiplexer
and frame builder is accomplished by creating a high speed multiplexer
that feeds a FIFO. The high speed multiplexer generates a 7-word frame
after every 3 QIE clock cycles. This 7-word frame is then written into a
FIFO. The FIFO directly feeds the GOL. If the FIFO empties, then an IDLE
character is forwarded by the GOL. The interspersing of IDLE characters
ensures that stale data is never sent and the Buffer Board only keeps
valid data from each IFE board.

**Power Distribution:**

  ----------------- ----------------- ----------------- -----------------
                    Power per chip    Chips on board    Total power

  CMS QIE8          600 mW \@5 V      8                 4.8 Watts @ 5V

  Serializer        360 mW \@2.5V     1                 0.360 Watts
  (TLK2501/GOL)                                         \@2.5V

  VCSEL             37.5mW \@2.5V     1                 0.0375 Watts @
                                                        2.5V

  FPGA              700 mW \@3.3V,    1                 0.900 Watts
                    200mW @ 1.5                         \@3.3V, 1.5V

  PECL Receivers    110 mW \@3.3V     36                3.96 Watts \@3.3V

  LVDS-PECL         36mW @ 3.3V       36                1.3068Watts @
  terminations                                          3.3V

  PECL-PECL         52 mW @ 3.3V      12                0.622 Watts@ 3.3V
  terminations                                          

  PECL Clock fanout 400 mW \@3.3V     1                 0.400 Watts @
  (MC100LVEP111)                                        3.3V
  ----------------- ----------------- ----------------- -----------------

12.4Watts per board

The power consumption estimates in the table above are listed in
wattage. The values are listed in wattage to account for the varying
operating voltages of all of the chips. Listing the values in wattages
also enables the selection of an acceptable power supply. Different core
voltages will be converted using linear regulators to step down the
voltage from a higher voltage power supply. It is desirable to send a
moderately higher voltage from the power supply to the IFE board.
However, the linear regulators used by the IFE boards must dissipate
internally any excess voltage provided to them. A good, safe assumption
for the dropout voltage of a linear regulator is 1.5V. This means that
the power supply that provides the input for the QIE regulator should be
at least 7.0V and the regulator that provides the power for the
remainder of the board should be at least 4.8V.

The IFE board sits in a 6U x160 Eurocard chassis with a custom backplane
that provides two power buses per four IFE boards. It is expected that a
custom rad-tolerant supply will be designed to provide power for the 16
IFE boards within a front end crate. The ideal supply would be able to
generate the two voltages below at the corresponding currents:

QIE Supply: 7Volts at 20 Amps

3.3V-generating supply: 4.8 Volts at 40 Amps

The values above were rounded up to the nearest 10 amps to allow for
dose-related current increases.