**Fermi National Accelerator Laboratory**

**Particle Physics Division**

**Electrical Engineering Department**

**Accelerator Division Support Projects**

**Tevatron Ionization Profile Monitor (IPM) Project**

**IPM Front End (IFE) Hardware Manual**

**Kwame Bowie**

August 10, 2005

Revised: August 30, 2007

**1. Introduction:**

This document will present the current state of the multi-channel front
end board designed for the Tevatron IPM project. The motivation of this
document is to describe the board in its present state. This document
will discuss the board's architecture, and its operation within the
Tevatron IPM DAQ system.

**2. Tev IPM DAQ System Architecture:**

The IPM Front End (IFE) board is the front end portion of the IPM data
acquisition (DAQ) system. The following shows in block diagram form the
basic architecture and data flow of the DAQ system:

![](media/image1.wmf)

The IFE board is a single element of the Tev IPM DAQ system. The Tev IPM
DAQ system may be thought of as two separate subsystems: the upstairs
subsystem and the in-tunnel subsystem. The upstairs portion of the DAQ
system all resides in a single PC. The upstairs PC houses the Timing
Card as well as the Buffer Boards and implements all of the intelligence
and control for the DAQ system. The in-tunnel portion of the DAQ system
all resides in a single 6U x 160mm Eurocard crate. The in-tunnel portion
includes sixteen (16) front end cards as well as a single fanout card.
The in-tunnel portion intentionally has very little memory or
configurable functionality because it will sit very close to the beam
line in a moderately high radiation dosage area. For that reason, all
components of the IFE and fanout cards are designed to tolerate
reasonable dosage levels, and unnecessary logic is not present in the
design. The remaining sections of this document will discuss the IFE
board design and functionality in detail.

**3. IFE Specifications:**

The following specifications have been developed as the result of
several meetings with various members of the project group. This
document breaks the specifications down into two categories: functional
description and performance specifications.

**Functional Description:**

-   Digitize the charge data of 8 QIEs operating in both calibration
    mode and normal non-inverting mode.

-   Relay the following flags to closest QIE clock cycle:

    -   Proton

    -   Pbar

    -   Proton Injection

    -   Antiproton injection

    -   Error bits

    -   Crossing(clock) counter

    -   Proton bunch counter

-   Multiplex data for serialization via an optical link

-   Integrate charge at rate of 2RF/7

-   Data integrity checking to detect data link errors

> **Performance Requirements:**

-   Operation reliable up to 20 krad total ionizing dose (TID)

-   Noise level: 0.5 -- 1LSB RMS (1-2 fC)

-   Minimum signal (resolution): \~1 fC

-   Provide additional 4 timing flags every 66 ns (2RF/7) corresponding
    to accelerator events of interest (injections, revolutions)

-   Provide 7-bit proton revolution counter and 9-bit clock counter for
    data integrity checking and as a time reference

-   High speed serial data out using optical link \@1.6
    gigabits-per-second (Gbps)

-   Maintain serial data lock with receiver for entire duration for
    acquisition of at least 500K samples (33 ms)

![](media/image2.jpeg){width="2.863888888888889in"
height="2.686111111111111in"}

**4. IFE Board Architecture**

The block diagram below illustrates the main functional elements of the
IFE board. The following sections will provide more detail about each of
the blocks.

![](media/image3.wmf)

**4.1. QIE biasing and interface**

The core element of the charge integration and digitzation functionality
for the IFE board is the QIE8 chip developed for CMS by Fermilab
engineers. The principal designer T. Zimmerman has authored several
documents which provide a very detailed description of the operation and
performance of the chip. The QIE is an amazing chip that can operate in
many different configurations in order to support many different signal
sources. In this section, we will discuss the configuration that we have
selected for the QIE. The table below illustrates all six (6) of the
possible modes of operation of the QIE.

  ----------------------- ----------------------- -----------------------
  **Range mode**          **Charge Polarity**     **Input Impedance**

  Calibration Mode        Non-inverting           50 ohm

  Normal Mode             Inverting               90 ohm
  ----------------------- ----------------------- -----------------------

The IFE board is designed to run with the QIE operating in its
non-inverting mode. The non-inverting mode of the QIE is useful for
detector elements generating negative current pulses such as PMTs or
similar devices. The micro channel plates (MCPs) collecting ionization
electrons provide a negative current type of pulse when configured for
the IPM project.

The QIE also has selectable input impedance. One may select between 90
ohms or 50 ohms termination inside the QIE chip. The IFE board utilizes
50 ohms input termination for the QIEs amplifiers. The selection of the
50ohm termination was primarily due to cabling and connector concerns.

The QIE integrates charges with a dynamic range of roughly 14 bits in
its NORMAL mode of operation. Although there are 14 bits of dynamic
range, the data is condensed into a single 7-bit data word with
non-linear transfer function. Although large dynamic range is quite
useful in many applications, the signals of interest for the Tev IPM
project reside in a fairly small window at the lowest end of measurable
charges for the QIE. In fact, it is sometimes desired to see signals of
interest below the threshold of the LSB in NORMAL mode. The QIE has a
CALIBRATION mode in which the voltage vs. charge curve is linear with a
slope of \~0.9fC/ count. In many situations, several channels per board
will be expected to operate in CALIBRATION mode and the remainder in
NORMAL mode. The IFE board provides an 8-bit DIP switch that allows or
disables operation in NORMAL mode. If a particular DIP switch is not
set, then the QIE on the associated channel will always operate in
CALIBRATION mode.

In addition to the mode settings, it is important to bias the QIE
amplifiers correctly for the expected signals of interest and the
selected sampling rate. The QIE8 chip has a bias setting resistor that
sets the bias currents for all of the internal transistors and
amplifiers. By decreasing the bias, one may integrate over a longer
period of time without saturating the output. A resistance of 750Kohms
provides an ideal bias for the selected sampling rate of roughly 15MHz.

The QIE is capable of integrating and digitizing charge at a better than
40Mhz sampling rate. This high throughput may only be accomplished by
using a multi-cycle pipeline. The QIE has a four-cycle pipeline with
four identical and parallel charge integration pathways. Unfortunately,
these identical pathways have slightly different characteristics due to
wafer variations that simply cannot be perfectly matched and controlled.
Since these pathway-to-pathway variations will remain nearly constant
over time, a great increase in accuracy may be extracted by calibrating
each of these pathways. In order to calibrate these pathways, we must
know which pathway each sample represents. The QIE provides this pathway
information in the form of two additional bits for each sample called
CAPID bits. Without these CAPID bits and a full calibration, the QIE
charge measurement accuracy is of order 25% while with a full
calibration, the accuracy of 5% or better is achievable.

**4.2. Timing/Control Link:**

The timing/control link originates from the timing card in the upstairs
subsystem. The timing/control link signals are sent from a PECL driver
in the upstairs subsystem to the in-tunnel subsystem on a single CAT5
cable. Once the signals reach the tunnel, the signals are duplicated and
passed to the IFE boards by the Fanout Card on custom 8-conductor
twisted pair cables. The timing/control link of each IFE is composed of
four (4) beam-synchronized PECL pairs signals. The signals are listed
below:

-   CLOCK (2RF/7)

-   PROTON marker

-   PBAR (antiproton) marker

-   ENCODED CONTROL (5-bit serial encoded)

The PROTON and PBAR markers are provided by the timing card to serve as
triggers for the Buffer Board triggering logic. The Tevatron operates
with a known beam structure and bunch spacing. Relevant charge signals
are only expected during the certain bunch crossings. Providing
beam-synchronized PROTON and PBAR signals to the Buffer Board allows
these signals to be used as a trigger for storing data. Without the
PROTON and PBAR signals, the Buffer Board would need to trigger on data
in a possibly very complex manner. The two markers provide a very
accurate and very simple trigger for data storage.

**4.3 Clock Distribution:**

![](media/image4.wmf)

The clock distribution on the IFE board includes two (2) asynchronous
clock domains: the system or QIE clock, and the serializer clock. The
distribution of each of these clocks to the relevant portions of the
board is discussed in this section

The clock distribution scheme for the QIE clock is based on a tree of
point-to-point links as shown in the diagram. The fanout card
distributes beam timing information to each front end via a dedicated
point-to-point link. Each IFE board then duplicates the system clock
signal nine (9) additional times and sends the clock signal to the eight
(8) QIEs and an FPGA via point-to-point links. The total skew between
all clocks is less than +/- 2 nanoseconds.

The GOL serializer requires a very pristine clock signal. The GOL
expects a clock with less than 150 ps of jitter peak-to-peak. When
designing the clock distribution system, each layer of logic adds a
small amount of jitter to a clock signal. The GOL's strict clock jitter
requirements suggested that minimum logic be placed between the origin
of the serializer clock and the GOL chip. The simplest solution turned
out to be placing a low-jitter clock on each board with a point to point
link to the GOL.

**4.4 FPGA Design:**

The FPGA design implements all of the control functions as well as the
time multiplexing of the digitized QIE outputs. This section will be
dedicated to describing the details of each portion of the FPGA design.
There are four key elements to the FPGA design: board-level mode
control, QIE control, frame builder, and GOL control.

**4.4.1 Board-Level Mode Control:**

The board-level mode control module is responsible for receiving the
encoded control signal from the timing/control link and decoding the
commands. Once a command is decoded, the board-level mode control module
must forward the appropriate signals to the other modules within the
FPGA. The IFE board has four (4) persistent operational modes that once
issued will persist until the mode is changed again or the board is
reset. The IFE board also services several commands that do not affect
the overall board mode. In total, the ENCODED CONTROL timing signal may
issue 8 different commands. The commands are encoded such that only
commands starting with a "11" will trigger the decoding circuitry. The
table below lists the commands that the IFE board implements and the
associated codes that must be present on the ENCODED CONTROL signal in
five (5) consecutive system clock cycles.

**Code Function**

  ----------------- -----------------------------------------------------
  11000             No effect

  11001             Proton Injection Marker

  11010             Antiproton Injection Marker

  11011             QIE Mode0 (QIE in Calibration Mode)

  11100             QIE Mode1 (QIE in Normal Mode)

  11101             QIE Mode2 (Link test - outputs a 32-bit counter to
                    GOL)

  11110             QIE Mode3 (Reset QIEs)

  11111             IFE Board Reset
  ----------------- -----------------------------------------------------

The functions are implemented such that there must be a one-cycle
waiting period between sending back-to-back commands. Processing any of
the commands takes seven (7) 2RF/7 clock cycles or nearly 500 ns.

**4.4.2 QIE Control:**

The QIE chip is responsible for the integration of the analog charge
signals. The QIE chip is a very simple device to operate as it requires
very few control inputs for proper operation. The main control signals
that are responsible for the operation of the QIE as follows: RESET,
MODE, and RANGE. The IFE board uses DIP switches to control the value of
the RANGE control signal for each QIE. The RESET and MODE signals are
the responsibility of the FPGA design.

The QIE utilizes a synchronous RESET signal that must be stable for
several nanoseconds on either side of the clock edge. As a result, the
IFE FPGA design specifies that the RESET signal may change only on the
falling edge of the QIE clock.

The mode signal is latched during the reset of the QIE chip. As a
result, it is important that the mode is stable as the RESET signal is
asserted. The FPGA asserts the MODE signal on the rising edge before the
RESET signal is asserted.

**4.4.3 Frame Builder**

The frame builder is the workhorse of the IFE FPGA design. The data from
the IFE board is transmitted to the Buffer Board as a serial bitstream
in fixed length blocks which will be called frames in this document. The
frame structure serves many purposes, but perhaps the most elementary
function is that the frame allows the Buffer Board to know which sample
corresponds to which QIE on the IFE board. The IFE board has eight (8)
QIEs, and each QIE produces 7-bits of integrated charge data for a total
of 56 bits of charge data per board at each sampling instant. However,
the GOL may only stream out 32 bits at a time. This bus width constraint
means that multiple data words must be sent to relay the information
from each sampling instant. The frame allows the Buffer Board to know
which of these words corresponds to the data from each QIE.

The frame builder module is responsible for the time-multiplexing of the
data, the proper alignment of timing signals with charge data, and the
intermediate storage of data while the GOL is busy. This document will
primarily describe the implementation of the frame builder module and
only briefly describe some of the design tradeoffs that were weighed.

The data frame size is a parameter that affects many portions of the Tev
IPM DAQ system. One of the tradeoffs that had to be weighed was data
content versus encoder/decoder complexity. The GOL is configured on the
IFE boards to produce a constant output stream of 1.6 Gbps with 1.2 Gbps
being the maximum data payload. The essential charge and timing
information that must be sent to the Buffer Board is 62 bits every 66ns.
This 970Mbps data rate (76% utilization) provides reasonable margin for
extra information from each sampling instant. It was decided that a
single extra 32-bit word after every 3 charge and timing samples would
be sufficient to relay enough information for data integrity checks.
This extra word would increase the data throughput to 1.13 Gbps data
rate (88% utilization). It is advantageous to use somewhat less than the
maximum GOL data bandwidth because it allows for frequent IDLEs in the
data stream. The use of frequent IDLEs is beneficial for maintaining
SERDES lock, frame encoder and decoder simplicity, and synchronization
purposes.

The frame length that the IFE currently implements is a 7-word frame
based upon 3 QIE clock cycles. This solution allows frequent IDLE
characters, simple frame encoder and decoder designs, and modest memory
requirements. The data frame format is shown in the diagram below.

![](media/image5.wmf)

**4.4.4 GOL Control (Data Serialization)**

The IFE board uses the GOL as its serializing element. The GOL is an
ASIC developed for the CMS project at CERN. The GOL has multiple
operating modes, but the IFE always operates with the GOL in the high
speed mode which allows serialization at a bit rate of 1.6 Gbps. The GOL
may be thought of as a high speed parallel-to-serial shift register that
receives a 32-bit input and transmits a 1.6 Gbps serial data stream.

As the IFE board and DAQ system are based upon two asynchronous clock
domains, it is very important to understand the relationships between
these two clock domains and their impact on the data. The two clocks
that must be present for the system to operate reliably are the system
clock and the serializer clock.

The system clock is based upon the Tevatron RF cycle. The frequency of
the system clock is actually two-sevenths of the Tevatron RF frequency
(roughly 15.17 MHz.) This beam-synchronized clock provides the time
reference for our signals of interest. Unfortunately, the frequency of
this clock varies slightly with the Tevatron's mode of operation. This
variation in the system clock makes it unsuitable for use as the
serializer clock due to its frequency variation.

To enable reliable serialization, a low-jitter crystal oscillator with
very precisely known frequency is located on the IFE. The frequency of
this crystal is defined by the bit-rate of the GOL, which specifies 40
MHz.

Since the system is based on two asynchronous clocks that are not pure
harmonics of one another, there is a definite need to ensure that data
is not re-transmitted or lost. Either situation makes the data
acquisition a more complex problem.

It is important that only relevant data be transmitted. Any "stale" data
will not be transmitted during its respective serializer clock cycle.
Instead an IDLE character will be sent, which will notify the receiver
that there is not currently new data to store. By sending only new data
separated by IDLEs, we are able to be certain that we do not store any
stale data. The IDLE data stream is four (4) 8B/10B encoded characters.
The sequence is as follows: **IDLE = \<K28.5\>,\<D5.6\>, \<K28.5\>,
\<D16.2\>**

To be certain that data is never lost, one must ensure that the data
link has adequate bandwidth to relay the necessary charge and timing
information. In addition, one must be certain that the link is reliable
and does not lose lock during an acquisition. These two requirements are
satisfied by the design of the board. However there is a possibility
that as the board ages, there might be data loss as a result of slow
board failure. To ensure that this board failure is detected, data
integrity checks are embedded in the data stream in the form of
counters. If the counters agree with the expected values, then we assume
that no data was lost.

**4.5 Power Distribution:**

  ----------------- ----------------- ----------------- -----------------
                    Power per chip    Chips on board    Total power

  CMS QIE8          600 mW \@5 V      8                 4.8 Watts @ 5V

  Serializer        360 mW \@2.5V     1                 0.360 Watts
  (TLK2501/GOL)                                         \@2.5V

  VCSEL             37.5mW \@2.5V     1                 0.0375 Watts @
                                                        2.5V

  FPGA              700 mW \@3.3V,    1                 0.900 Watts
                    200mW @ 1.5                         \@3.3V, 1.5V

  PECL Receivers    110 mW \@3.3V     36                3.96 Watts \@3.3V

  LVDS-PECL         36mW @ 3.3V       36                1.3068Watts @
  terminations                                          3.3V

  PECL-PECL         52 mW @ 3.3V      12                0.622 Watts@ 3.3V
  terminations                                          

  PECL Clock fanout 400 mW \@3.3V     1                 0.400 Watts @
  (MC100LVEP111)                                        3.3V
  ----------------- ----------------- ----------------- -----------------

12.4Watts per board

The power consumption estimates in the table above are listed to
demonstrate the most power-hungry components on the board. In reality,
the power consumption of the board is slightly less than these estimates
which are based upon maximum current draw specifications provided by the
component manufacturers. It is not hard to see that the PECL receivers
and terminations dominate the board's power consumption. However, to
accommodate the rad tolerance specification, the higher power
consumption is an acceptable tradeoff.

When selecting or designing a power supply to support the IFE boards, it
is important to know the voltage and current requirements for each power
supply. The power distribution of the IFE boards has been segmented into
two domains: analog power and digital power. The analog power supply
provides 5.5--volt power to the QIE chips. The digital supply provides
three (3) different voltages: 1.5 volts, 2.5 volts, and 3.3 volts to all
of the other components on the IFE board. Each of the power systems
provides power to a rad-tolerant linear regulator which tightly
regulates the voltage around the desired voltage level. The linear
regulators require an input voltage of at least 1.5 volts above the
regulated voltage that they will output to maintain regulation. It is
important to realize that the linear regulators dissipate any extra
voltage drop. As a result, the power supply input voltage should not
exceed 3 volts above the regulated voltage. Combining these two
requirements constrains the power supply requirements for each IFE board
to the following:

**Analog Power Supply: 7V -- 8.5V @ 1.5 Amps**

**Digital Power Supply: 4.5V -- 6V @ 2.5 Amps**

The IFE board sits in a 6U x160 Eurocard chassis with a custom backplane
that provides two power buses that are shared among four (4) IFE boards.
It is expected that a custom rad-tolerant supply will be designed to
provide power for the 16 IFE boards within a front end crate. The ideal
power supply for the Tev IPM front end system would be able to generate
the two voltages below at the corresponding currents:

Analog supply: 7.5 Volts at 20 Amps

Digital supply: 5 Volts at 40 Amps

The values above were rounded up to the nearest 10 amps to allow for
dose-related current increases.

**4.6. GOL Jumperable Mode Options:**

There are several options that may be modified by the replacement or
removal of jumper resistors. The following table lists the supported
modes and the default configuration of the IFE boards. The operational
modes of the IFE board are highlighted in red.

  ----------------------- ------------------------- -----------------------
  *Parameter*             *[Option1]{.underline}*   *Option2*

  Clock input signaling   **Single-ended**          Differential

  Data Rate               800 Mbps                  **1.6 Gbps**

  Encoding scheme         CIMT                      **8B/10B**

  Output signaling        VCSEL laser driver        **Differential 50 ohm**
  ----------------------- ------------------------- -----------------------

**4.7 Optical Transmitter module:**

The optical transmitter module used for the IFE boards is a transceiver
package from Stratos Lightwave, but the receiver portion is left unused.
The transmitting laser operates at the **850 nm wavelength** and is
optimized for **multimode fiber**. The optical transmitter module was
selected as a more robust replacement for the single-ended standalone
vertical cavity surface emitting laser (VCSEL) diodes that were unable
to perform reliably on the prototype IFE boards. The optical transceiver
module has two advantages that allow it to be more reliable than the
VCSEL solution: a light-tight locking connection, and a differential
input. The combination of these two advantages, allow the optical
transceiver to operate reliably without losing lock with the buffer
board during an entire acquisition. The VCSEL diode was unable to
achieve this although it was significantly cheaper than the optical
modules.

**5. Interboard IFE Synchronization:**

Each IFE board accepts charge signals for eight (8) channels as its
input and generates a high speed data stream as its output. These
signals are always running and are presented to the receiver boards
without interruption. The IFE board from a functionality perspective is
simply a time multiplexing and digitizing board. There is very little
memory and no trigger generation capability currently present on the IFE
board design. Since the IFE boards do not produce their own trigger
signals, it is critical that IFE board data bytes be associated with the
correct Tevatron events which will serve as the trigger. To ensure that
data can be correlated with Tevatron events, synchronization must be
accomplished.

There are essentially two requirements to achieve synchronization
between the different IFE boards in the Tev IPM DAQ system. The first
requirement is that the system must have at least one common time
reference. The timing control link provides a clock and several timing
signals for all IFE boards that provide a common time reference. The
second synchronization requirement is that there must be a
synchronization control signal. The synchronization control input is
implemented with the ENCODED CONTROL signal within the timing/control
link. The RESET command issued through the ENCODED CONTROL signal serves
as the synchronization command. The RESET command ensures that all QIEs
begin initial integration and data transmission during the same 2RF/7
CLOCK cycle.

Although all IFE boards begin integration and data transmission during
the same 2RF/7 CLOCK cycle, the data links on each IFE board will
operate at slightly different data rates due to each board having its
own crystal oscillator. To accommodate these slightly different data
rates, it is necessary to select a data format that generates frequent
IDLE characters. The IDLE characters temporarily halt data storage of
the "faster" IFE boards and allow the "slower" IFE boards to catch up.
The implementation of this synchronization is done very simply on the
buffer board by enabling or disabling the input FIFOs of the associated
IFE on the buffer board pending the receipt of an IDLE character.

The above synchronization method does make two key assumptions:

1)  All data links will remain locked between a system RESET command and
    the end of the acquisition.

2)  It is possible to initially align all IFEs such that the 1^st^ word
    after a system RESET arrives in memory location 1 for all channels.

If the following two conditions are satisfied, then synchronization is
guaranteed by design. The first condition was a design requirement for
the IPM system from the start, and has been verified through testing.
The second condition was solved by embedding a Buffer Board reset
command into the data stream. The only concern was how to accomplish
this task. The simplest solution was to allow a long string of IDLE
characters to represent a Buffer Board reset command. The length of the
IDLE string had to be long enough to distinguish between a normal IDLE
character and a reset command. The IFE boards will never issue more than
two (2) IDLE commands in immediate succession using the current data
rates and data format. It was decided that four (4) IDLE characters was
a reasonable number to denote a Buffer Board reset. Once the reset is
received, the Buffer Board resets its RAM write pointers and the next
data word for each IFE is placed in location 1. The Buffer Board
implements checking of the counters in the footer word to verify that
synchronization is achieved.