**Tevatron Ion Profile Monitor (IPM) Project**

**QIE for IPM Front End (QIFE)**

Kwame Bowie

August 6, 2003

Revised: August 14, 2003

**Introduction:**

This document will present the current state of the multi-channel QIE
board that will be designed for the Tevatron IPM project. The motivation
of this document is to present the current design ideas. This document
will discuss the basic architecture as well as the key parts that are
necessary for the design to materialize.

**System Architecture:**

The QIE for IPM front end (QIFE) board is the front end portion of the
IPM data acquisition (DAQ) system. The following diagram generated by A.
Jannson shows in block diagram form the basic architecture and data flow
of the DAQ system:

![](media/image1.wmf)

One element of the system design that has not been solidified yet is the
number of channels of QIE data that may be multiplexed into the fiber.
The number will most likely either be 8 or 10. This number is largely
dependent upon the availability of rad-hard serializers. The
serialization rate achievable using 8 QIEs per fiber without combining
CAPID bits is as follows:

9 bits/QIE \* 8 QIEs \* 17.6MHz = 1.267Gbps after 8B/10B 1.584 Gbps

If we decide to keep only 1 set of CAPID bits per board then the data
rate becomes:

(7 bits/QIE \* 8 QIEs + 2 CAPID bits) \* 17.6 MHz = 1.02 Gbps 1.276 Gbps

If we decide to try to serialize data from 10 QIEs our data rates jump
to:

9 bits/QIE \* 10 QIEs \* 17.6MHz = 1.584 Gbps after 8B/10B 1.98 Gbps

If we decide to try to serialize 10 QIEs and keep only 1 set of CAPIDs
we get:

(7 bits/QIE \*10 QIEs + 2 CAPIDs) \* 17.6MHz = 1.276 Gbps 1.584 Gbps

It is important to note these data rates so that the system level DAQ
specifications may be formalized.

**Specifications:**

The following set of specifications was developed by the author, and may
need to be refined to truly represent the needs of the project. However
the specifications listed have been developed as the result of several
meetings with various members of project group.

The specifications are shown below in list form:

-   Digitize the charge data of 8 or 10 QIEs operating in all possible
    modes of operation.

-   Accept accelerator timing information from a header board.

-   Multiplex data for serialization via an optical link

-   High speed serial data out using optical link (1.5, 1.6 or 2.5 Gbps)

-   Operation reliable up to 10 krad total ionizing dose (TID)

**Power Consumption:**

  ----------------- ----------------- ----------------- -----------------
                    Power per chip    Chips on board    Total power

  CMS QIE8          600 mW            8 or 10           4.8 or 6 Watts

  Serializer        360 mW            1                 0.360 Watts
  (TLK2501)                                             

  FPGA                                                  

  LVDS Receivers    50 mW             18                0.90 Watts

  PECL Clock fanout 1.254 mW          1                 1.254 Watts
  (MC100ES6111)                                         
  ----------------- ----------------- ----------------- -----------------

8.5 Watts per board

**CKM QIE Test Beam Board:**

The QIFE board is to be based upon the CKM QIE Test Beam Board (QTBB).
The QTBB board is a very simple front end board that simply multiplexes
and serializes QIE data out over an optical link to a remote PC. The
selling point of the

QTBB board is its simplicity, and flexibility. The simplicity was a
great benefit when debugging the QTBB board. The QTBB board requires no
initialization or programming; once the board is powered, it
continuously sends QIE data to the test stand.

The QTBB board is very flexible by design and has many possible modes of
operation that are set by jumpers, resistors, or firmware updates. The
block diagram below breaks the QTBB board down into its functional
units:

![](media/image2.wmf)

**Changes from CKM QTBB:**

The changes between the QIFE and the CKM QTBB essentially relate to
three goals: minimizing overhead, adding provisions for multiple boards
operating together, and improving radiation-hardness.

The CKM QTBB has the overhead of NIM level inputs and outputs for
control and a parallel data bus for testing. The removal of the extra
inputs and the NIM-level logic generation circuitry removes the
necessity for a --5V power supply and decreases board power consumption
by roughly 2 Watts. The final QIFE board will have more inputs for
controlling board parameters thans the QTBB board. However, the only
reason that the current QTBB does not need more inputs for mode control
is because all of the mode control is hardwired. In order to change
operational modes for the QTBB, resistors are removed, jumpers are
placed, or the PLDs are reprogrammed. For the QIFE boards, this is not
an option since the boards will reside in a radiation environment. The
QIFE cannot have mode control determined using resistors and jumpers
since attempts to replace resistors or jumpers will require that the
radiation dissipate from the PC board before any technician will be
allowed to change the configuration. Also the QIFE will use rad-hard
one-time-programmable devices for the logic section, so reprogramming
the programmable devices to switch modes is not an option. The other
unnecessary portion of the QTBB design was the parallel readout port
used for testing purposesThe removal of the parallel data bus for
testing also significantly reduces the power consumption and reduces
parts count.

The next key modification to the core design of the CKM QTBB is that the
QIFE board is not an R&D testing board and must interoperate in a system
with fifteen (15) other QIFE boards. The system block diagram in the
system architecture section shows that 16 QIFE boards share Tevatron
timing information and will reside in a single crate. To power these
boards and provide the necessary clocks, and header information, a
backplane interface must be developed and implemented on each QIFE
board. This backplane interface does not need to be complicated, but it
is a departure from the core QTBB design and adds some complexity. The
block diagram below shows that most of the functional elements are the
same as the QTBB, but the physical locations have changed due to the
need for a backplane.

![](media/image3.wmf)

The last key modification to the core design of the CKM QTBB is that all
of the parts must be qualified, tested, or designed to be tolerant of
moderately high radiation doses. The generally accepted figure is that
all parts must function at least as well as the QIE, which has a maximum
total ionizing dosage (TID) of 20 kilorads (krad). Fortunately, many of
the parts from the QTBB board are radiation tolerant; this good fortune
is a result of the QTBB design being derived from a CMS testing board.
The CMS project has put significant effort into testing and verifying
the radiation tolerance of all components of their front end boards. We
hope to take advantage of the research that the CMS project has
conducted and use parts that are either verified by the manufacturer, or
the CMS project as radiation tolerant. It is important to demonstrate
the reliability of each key component in a radiation environment. The
following is a list of the key integrated circuit (IC) components of the
QIFE board:

1)  8 QIE chips

2)  1 PECL 1:10 clock fanout driver

3)  1 Antifuse FPGA

4)  18 LVDS to TTL level converters.

5)  1 vertical cavity surface emitting laser (VCSEL)

6)  Linear regulator

All of the above key parts have been validated at high radiation levels
except the linear regulator. Fortunately linear regulators are general
purpose integrated circuits and are often used in high-radiation
environments (aerospace, etc.). As a result, there are numerous
commercially available solutions when searching for a rad-hard linear
regulator.

**Data Synchronization:**

As the QIFE board and DAQ system will be based upon two asynchronous
clocks, it is very important to understand the relationships between
these two clocks and its impact on the data. The two clocks that must be
present for the system to operate reliably are the system clock and the
serializer clock.

The system clock is based upon the tevatron RF cycle. The frequency of
the system clock is actually one third of the tevatron RF frequency or
17.6 MHz. Unfortunately the frequency of this clock varies slightly with
the Tevatron's mode of operation. This variation in the system clock
makes it unsuitable for use as the serializer clock due to its greater
than 100ppm frequency variation.

To enable reliable serialization, a separate high-precision crystal is
located on the QIFE. The frequency of this crystal depends upon the
bit-rate of the serializer and is normally a factor of 10, 20, or 40
times smaller than the actual data rate of serialization. The clock
multiplier internal to the serializer device is different for each
serializer device and depends on the serializaed data rate.
Synchronization will be discussed for two separate digitization rates:
1.6 Gbps and 2.5 Gbps.

Since this system is based on two asynchronous clocks that are not pure
harmonics of one another, there is a definite need to ensure that data
is not lost or re-transmitted. Either situation makes the data
acquisition a more complex problem. To make the receiving of the serial
data the simplest, only relevant data will be transmitted. Any stale
data will not be transmitted during its respective serializer clock
cycle. Instead an idle (K28.5) character will be sent, which will notify
the receiver that the previous word was not valid data. Over time the
two clocks will be related in a periodic way, and this relationship will
be discussed below:

**[Digitizing 8 QIEs @ 1.6 Gbps:]{.underline}**

For an 8-channel board with serialization at 1.6Gbps, the serializer
clock would be 80 MHz. The relationship between these two clocks is such
that it will take several cycles of the system (QIE) clock for there to
be an extra serializer clock cycle. The calculations based upon ideal
clocks (no frequency variation) would look as follows:

System clock period = 1 / 17.6 MHz = 56.818 ns

Serializer clock period = 1 / 80 MHz = 12.5 ns

Serializer clocks per system clock period = 4.5454

Skew between leading edges after 4 cycles: 6.818 ns

Number of system clock cycles before extra clock cycle: 12.5 / 6.818 = 2

So the number of serializer clocks per QIE clock will follow the
following pattern:

5-4-5-4-5-4...

**[Digitizing 10 QIEs @ 2.5Gbps:]{.underline}**

For a 10-channel board with serialization at 2.5 Gbps, the serializer
clock would be 125 MHz. The relationship between these two clocks is
such that will take several clock cycles of the QIE clock before the
serializer clock will produce an extra cycle.

The calculations based upon ideal clocks (no frequency variation) would
be as follows:

QIE clock period = 1 / 17.6 MHz = 56.818 ns

Serializer clock period = 1/125 MHz = 8 ns

Serializer clocks per QIE clock period = 7.10

Skew between leading edges after 7 cycles = 0.818 ns

Number of QIE clock cycles before extra clock cycle = 8/0.818 = 10

So the number of serializer clocks per QIE clock will follow the
pattern:

8-7-7-7-7-7-7-7-7-7-8-7...

**Logic Design:**

The last section discussed the need for a method of synchronization. To
comply with this need there must be at least a 3-deep FIFO to ensure
that no data is lost. The FIFO implements the pipelining that is
necessary to give time for the FPGA to make a decision on whether or not
the data should be transmitted. As the data enters the FIFO, it is
preserved until it is shipped to a multiplexing module. The multiplexing
module clocks data out at the serializer clock frequency. The diagram
below shows the basic architecture of the programmable logic. Although
the core of the FPGA functionality is the multiplexer implementation,
there is a need to provide testing and mode control capabilities within
the FPGA. As a result, there are four sections of the FPGA design:
master/mode control, qie control, serializer control, and multiplexer
module.

![](media/image4.wmf)

**[Master/Mode Control:]{.underline}**

The master control module receives the mode control information from the
header board and determines the mode of operation of the QIFE board. The
QIFE board has two operational modes: testing mode and run mode. The
testing mode sends out counter data instead of QIE data. This testing
mode will be very useful for determining system-level latencies. The run
mode transmits real QIE data, and actually has several sub-modes which
determine the operational mode of the QIE.

**[QIE Control:]{.underline}**

The QIE control module is responsible for relaying the operational mode
from the master control module to the actual QIE chips and must decode
the control signals as well as handle resets correctly.

> **[Serializer Control:]{.underline}**

The serializer control is responsible for setting up, initializing and
sending data to the serializer chips. Especially important is the
serializer initialization and reset control since for certain
serializers such as the GOL, initialization and resetting may be
non-trivial.

> **[Multiplexer Module:]{.underline}**

The multiplexer module is performs a very simple time division
multiplexing (TDM) operation. The multiplexer streams data out at the
serializer clock rate and determines when to disable the serializer so
that data is synchronized and transmitted correctly to the receiver
board.

**Header Data:**

The QIFE boards will embed accelerator timing in their data stream, so
that a precise and absolute time reference is known for each data point
with respect to a start acquisition signal. The accelerator timing
information will be based upon the proton and anti-proton revolution
timing. The amount of header data that will be transmitted depends on
serialization rate and the number of QIEs that must be present on each
QIFE board. The number of bits available for header information are
calculated below:

**[8 QIEs per board @ 1.6Gbps:]{.underline}**

header bits = (1.28 Gbps data payload -- ( 8 QIEs \* 7 bits / QIE + 2
CAPID bits) \* 17.6 MHz) / 17.6 MHz

header bits = 14 bits

**[10 QIEs per boards @ 2.5 Gbps:]{.underline}**

header bits = (2 Gpbs data payload -- (10 QIEs \* 7 bits / QIE + 2 CAPID
bits) \* 17.6 MHz) / 17.6 MHz

header bits = 41 bits

An example of the header data format will be presented now. There are 14
bits that are available for timing information using 8 QIEs at 1.6 Gbps.
Most likely there will need to be a few bits to relay the QIE
operational mode. The QIE has 3 main operational modes for each biasing
setting. The four modes are as follows: calibration mode, fixed-range
normal mode, and auto-ranging mode. These four operational modes may be
relayed in two bits of header information. After QIE mode information is
added to the header, there are 12 bits left for timing information. The
Tevatron timing information is to come from the VUCD board, which will
transmit the Tevatron RF clock as well as both the proton and
anti-proton revolution markers to the header board. The header board
that shares the crate with 16 QIFE boards will send a QIE clock and a
header data word to the 16 QIFEs via a backplane. The sample header word
shown below demonstrates how the header word will be composed of both
timing information and QIE control information.

![](media/image5.wmf)

The desired additional timing information for the QIFE boards is based
upon proton and antiproton (pbar) revolution markers and the accelerator
RF. The timing information is generated on the header card and is
shipped downstream to each QIFE card simultaneously over a data bus. The
information that is placed upon the data bus is actually one of four (5)
possible counters. The following counters reside within the header
board:

-   Proton revolution counter

-   Antiproton revolution

-   Proton OR antiproton revolution

-   QIE clock counter

-   RF counter

The transmitted header data word is determined by the mode control
inputs of the header card. The QIE clock counter and RF counter may be
useful for development and testing purposes whereas the proton and
antiproton revolution counters are used by the DAQ system for data
logging.

There are two perceived modes of operation for the DAQ system: soft
trigger and hard trigger mode. The hard trigger mode simply stores a
preset number of data words based upon some trigger condition. This
trigger condition may be based upon the integrated charge data, timing
counter information, or any more complex combination of the data. The
soft trigger mode forces a preset number of samples to be stored
immediately. This trigger functionality occurs at the readout card layer
of the DAQ system since the QIFE continuously digitizes and ships out
data regardless of its operating modes.

**Remaining Concerns:**

The system-level design of the Tevatron IPM DAQ system seems to be
relatively safe and robust as far as the technology to be used is
concerned. In fact, similar systems have been used in testbenches for
several projects here at Fermilab. However, a few specifics of exactly
what is needed for the IPM project must be ironed out. These specifics
are related to the tradeoffs that must be made between channel density
and information content. The sample header data format given in the
previous section of this document uses the example of 8 QIEs per board
serialized at 1.6 Gbps. This example data format works out very well
because the 12-bits allocated for the counter word allows for a
modulo-2048 counter. The current design assumptions are that data from
the QIFE boards will be stored on the readout board as 2000 data chunks
where each chunk is a block of data digitized at the same QIE clock
edge. As a result, a modulo-2048 counter will enable each data chunk to
have a unique time identifier without wasting very many binary
combinations. This example 8-channel system provides a very
channel-dense solution, however it lacks pertinent of timing and
debugging information. For example, it might be useful to keep CAPID
bits for all QIEs to easily determine when a channel dies (or multiple
channels die). It might also be useful to have information in the header
word that defines which information is displayed in the counter word.
One might even suggest that more than one counter are present in the
header data word. Any of these suggestions would require the channel
count per QIFE board to be decreased. The removal of each QIE adds 7
more available data bits for header information. A minimum header word
information spec would address all of the remaining concerns.