# Presentation: 279795

## Nevis FVTX FEE Concept 

- PHENIX FVTX Review
- 19 February 2007

## Outline

- Overview
- Detailed descriptions:
- Daughterboard
- Relay & Interface
- Receiver
- Examples
- Readout issues
- Test setup at Nevis
- Summary

- D. Winter: Nevis FVTX FEE Concept

- 19 February 2007

## Design Overview

- _Key Features:_
- Reduction of signal lines at the wedge
- Copper carries signals from wedges to nearby rack
- Two data paths provided between IR and backend
- Relay card: fast unidirectional fiber carrying all events to Lvl1
- Interface card: not-quite-as-fast duplex fiber for slow control (including timing signals) and data (potentially Lvl1 accepted at the front end)

- Relay
- Board

- Interface
- Board

- Lvl1 Trigger

- Receiver
-  Control, Timing
-  Level1 accept
-  Data

- Enclosure

- IR

- Rack Room

- 4 planes (x 48)

- 4 (1 per plane)

- 1 per endcap

- High BW Fiber

- (Lower BW) Fiber

- D. Winter: Nevis FVTX FEE Concept

- 19 February 2007

## Assumptions for signal/channel counts

- Each plane consists of 48 wedges
- Each wedge read out by 12 (1st plane) or 26 (planes 2-4) FPHX chips
- LVDS signals required by each FPHX
- 2 Data Output: 120 total (plane 1),
- 6 LVDS signals that can be shared by FPHX chips:
- Shift Data In, Shift Data Out, Shift Control, Beam Clock (BCO) In, Data Clock In, Reset
- 3 Required non-LVDS signals that can be bused:
- Pulse inject, Digital voltage in, Analog voltage in
- Total Number of connections per wedge:
- 2x26(or 12) + 9 = **61 (33 for plane 1)**
- These numbers do not include connections not directly related to readout, such as sensor bias voltages, etc
- Total number of strips per wedge:
- 128*26(or 12) = 3328 (or 1536 for plane 1)

- D. Winter: Nevis FVTX FEE Concept

- 19 February 2007

## Wedge Daughterboard

- 1:1 Wedge to DB ratio
- Could consider 2:1 or 4:1 if deemed necessary
- LGA-based daughterboard
- Nominally 3cm X 3cm
- Size dominated by connectors (LGA, to-wedge connection, routing on board)
- Wedge-LGA connection via kapton HDI(s)
- Requires rad-tolerant FPGA
- E.g.: 256-pin BGA Actel A3PE600
- FPGA provides calibration pulses
- Derived from output of logic pins
- Amplitude determined by small (~6) resistor array.
- Provides power to digital and analog portions of FPHX
- Duplex LVDS communication with Relay
- Eg. Twinax, Firewire, etc.
- 300 Mbps link to Relay
- No external serialization necessary (no serdes)
- 1+1 for Tx/Rx both control & data
- Headroom for additional data lines if necessary

- D. Winter: Nevis FVTX FEE Concept

- 19 February 2007

## Relay

- Outside enclosure but inside IR (~10m from FVTX)
- 48:1 ratio of DB to Relay (per endcap)
- Assumes 1 DB per wedge
- DB-Relay via duplex LVDS line (eg. Firewire)
- Provides high-bw unidirectional link to Lvl1 trigger hardware
- High-speed serdes + fiber
- Data/Control path with Interface via duplex LVDS pairs (eg. twinax)

- Board

- Lvl1 Trigger

- High BW Fiber

- D. Winter: Nevis FVTX FEE Concept

- 19 February 2007

## Interface

- Located outside enclosure but inside IR (~10m from FVTX)
- Co-located with relay boards
- 1 Interface gathers 4 relays
- Merges data from relays for single transmission to receiver
- Connected via duplex fiber to receiver (rack room)
- Receives and relays control & timing signals upstream

- Relay
- Board

- Board

- Lvl1 Trigger

- High BW Fiber

- (Lower BW) Fiber

- D. Winter: Nevis FVTX FEE Concept

- 19 February 2007

## Receiver

- Provides duplex fiber communication to front end for both control and data
- Receives standard PHENIX timing signals, reset, Lvl1 accept
- On-board memory buffer for data
- Events are pipelined, in time order
- Application of Lvl1 accept straightforward
- Option to allow BCO window around Lvl1 accept
- Handles passing data to the archiving backend

- Relay
- Board

- Interface
- Board

- Lvl1 Trigger

-  Control, Timing
-  Level1 accept
-  Data

- High BW Fiber

- (Lower BW) Fiber

- D. Winter: Nevis FVTX FEE Concept

- 19 February 2007

## DB Example: Test interface to FPIX

_**Some Key Features:**_
- Based around Altera Cyclone FPGA
- Option for time-ordering pixel data (relevant only for FPIX, not FPHX)
- Option for 4-line readout (default is 1-line)
- Includes 15-bit counter to implement timed commands and timestamps
- FIFO buffers non-immediate commands (128 deep)

- D. Winter: Nevis FVTX FEE Concept

- 19 February 2007

## Receiver Example

- Optical
- Tx/Rx

- Serdes

- PCI Register Logic

- Memory

- 32-bit 33/66 MHz PCI Bus

- DMA

- Beam
- Clock

- Lvl1
- Accept

- Reset

- D. Winter: Nevis FVTX FEE Concept

- 19 February 2007

## Communication between Interface and Receiver

- Develop(ed) packet protocol for fiber link between current interface and receiver
- Protocol not specific to fiber communications, though, could adapt to copper communications as well
- Control commands:
- Immediate commands to be executed by interface upon receipt
- Examples: Firefighter reset, interface reset, read timer, etc.
- Interface commands queued in FIFO, executed in order of receipt
- Examples: read/set interface CSR, pulse chip
- FPIX API commands (also queued) passed on to FPIX control input
- Pixel (FPIX) data:
- Packets correspond to hits with same BCO
- New packets started as new BCOs are detected in data stream

- D. Winter: Nevis FVTX FEE Concept

- 19 February 2007

## Level1 Accept

- The Level1 trigger rejection, dropping unwanted data, can take place at multiple points in the readout
- We count BCO and BCO-N (where N represents the Level1 latency) at the Receiver, the Interface and DBs in the detector
- Note: this implies that we extend the BCO counter to 12 or more bits (easy to do)
- When we get a Level1 accept, buffer the BCO of accepted events, which can be matched against the time-ordered data at each of the 3 levels
- In the Receiver, we would also count Level1 accepts and buffer the event number along with the desired BCO.

- D. Winter: Nevis FVTX FEE Concept

- 19 February 2007

## Power Issues

- Only place where power dissipation is a concern in within the enclosure
- Power needs of the wedge, common to any design
- Digital and Analog voltage to FPHX chips
- Single FPIX draws ~ 0.2 A
- FPHX has 1/22 channels, assume 26 chips draw 0.5A @ 2.5V = 1.25 W per wedge
- Regulation of voltage near chips: 0.4 W @ 3.3V input
- Additional power needs as a result of presence of DB
- PLD: ~ 0.1 – 0.75 W (depends on many factors)
- Basically a small addition to existing dissipation

- D. Winter: Nevis FVTX FEE Concept

- 19 February 2007

## Level1 Trigger

- Current Level1 typically takes 12 ± 1 clock ticks (1 tick = 106 ns) from interaction to arrival of Level1 accept at FEMs
- This latency could be extended to 40, though it involves retiming of many (all?) systems
- Taking into account time for Level1 decision tasks (LL1 calculation, GL1 decision, etc.) this leaves approximately 4 clock ticks to deliver FVTX data to Level1 hardware
- ~4 hits/chip (central AuAu?) translates to ~5000 hits/plane -> assuming 16 bits/hit: 79kbits/plane / 4 ticks -> 186 Gbps/plane bandwidth required
- Relaxing the requirement of 4 clock ticks *may be in order*...
- Furthermore: do all bits need to be sent?

- Relay
- Board

- Lvl1 Trigger

- High BW Fiber

- D. Winter: Nevis FVTX FEE Concept

- 19 February 2007

## Key Features

- Large degree of flexibility (both in topology and board count)
- Intelligence placed at the wedge
- Large reduction of signals
- Repackage hits from native format to slightly more compressed format
- Uses FPIX command interface (same as future FPHX)
- Adds small extra power dissipation to existing system
- Rest of intelligence near, but not immersed in rad environment (eg. Rack on magnet)
- Data needing to be shipped fast (LVL1) copied to high-speed link
- Space required ~< 6U crate + power bus
- Receiver writes accepted events directly to PC memory
- BUT: not a feature integral to this design
- Pipelining of events from endcaps
- Lvl1 accept consists of matching accepted event list with data event list
- Drop events with clock older than current Lvl1

- D. Winter: Nevis FVTX FEE Concept

- 19 February 2007

## FPIX-based Prototype Implementation

- _“__2__nd__ Gen” Interface board_
- Provides control in/out to FPIX
- Data out from FPIX
- Provides pulser signal
- Supplies regulated digital and analog power
- Option for 1- or 4-line readout
- Option for timeordering of hits (relevant for FPIX, not PHX)
- Includes 15-bit counter to implement timed commands and timestamps

- _PCI Receiver_
- 32-bit 33 MHz PCI card
- 256 kB SRAM on-board
- User interaction via PCI registers
- Provides 3 channels: control out, control in, data in
- Dataflow governed by ABC register architecture (one set of ABC per channel)
- A : physical RAM address (in PC) of next DMA transfer
- B : physical RAM address (in PC) of next user access
- C: info for address wrapping boundaries

- FPIX Test Board
- From FNAL

- Interface board

- PCI Receiver

- D. Winter: Nevis FVTX FEE Concept

- 19 February 2007

## Test system at work

- Able to perform tests without requiring full phenix timing system
- Threshold and pulse height scan of FPIX response (similar to noise measurements performed by Sergey)
- Measure gains and offsets of each pixel channel

- Turn-on curve
- As function of threshold
- (fixed pulse height)

- D. Winter: Nevis FVTX FEE Concept

- 19 February 2007

## Summary

- Concept consists of multiple boards
- Intelligence at the wedge (DBs)
- Relay & Interface to manage control/date from of DBs
- Remote Receiver simply inserts data into backend
- Level1 accept can take place at multiple points
- Concept emphasizes
- Reduction of signal lines at the wedge
- Simplicity, flexibility
- Toughest issue is Level1 trigger constraints
- How much data needed in what time?
- How to accomplish this?
- Simple prototypes in the spirit of this readout have been implemented and have been used to perform simple tests and studies

- D. Winter: Nevis FVTX FEE Concept

- 19 February 2007

## Backup Slides

- D. Winter: Nevis FVTX FEE Concept

- 19 February 2007

## Notes on bits, clocks, data rates, and readout lines

- Data format needs to take into account addressing of wedges, planes, BCOs, etc.
- From the DB:
- 16 bit header to contain extended BCO
- 16 bit hits: 5 bit chip id, 7 bit channel id, 3 bit ADC, 1 bit “reserved”
- From the Relay:
- 16 bit header: ~6 bit wedge id, 3 bit plane id
- Data clock out of FPHX: 200 MHz
- How many readout lines are needed?
- Need to truncate the readout for high-occupancy events ...
- Communication from DB to Relay = 300Mbps

- D. Winter: Nevis FVTX FEE Concept

- 19 February 2007

## Strawman Cost Estimate (DB/Relay)

| Component | Quantity | Price/Unit | Total |
| --- | --- | --- | --- |
| Altera Cyclone FPGA | 1 | $30.00 | $30.00 |
| Oscillator | 1 | $10.00 | $10.00 |
| resistors (assorted) | 20 | $0.05 | $1.00 |
| Voltage Regulator 1.5V (digital) | 1 | $5.00 | $5.00 |
| .01 uF Capicitors | 20 | $0.05 | $1.00 |
| .10 uF Capacitors | 10 | $0.10 | $1.00 |
| Wafer Connector-96 pair | 1 | $10.00 | $10.00 |
| twinax connector – 4  pair | 1 | $5.00 | $5.00 |
| eprom – EPCS14 | 1 | $8.00 | $8.00 |
| SERDES TI TK3101 |   | $28.00 | $0.00 |
| Optical Driver |   | $25.00 | $0.00 |
| JTAG Connector | 1 | $1.00 | $1.00 |
| PC Board:4 layer, tooling, testing | 1 | $125.00 | $125.00 |
| Board Assembly | 1 | $25.00 | $25.00 |
| Total |   |   | $222.00 |

| Component | Quantity | Price/Unit | Total |
| --- | --- | --- | --- |
| Actel FUSION PLD | 1 | $30.00 | $30.00 |
| Voltage Regulator 1.5V (digital) LT1963 | 1 | $5.00 | $5.00 |
| Voltage Regulator 2.5V (Silicon-Digital) LT1963 | 1 | $5.00 | $5.00 |
| Voltage Regulator 2.5V (Silicon-Analog) LT1963 | 1 | $5.00 | $5.00 |
| .01 uF Capicitors | 10 | $0.05 | $0.50 |
| .10 uF Capacitors | 5 | $0.10 | $0.50 |
| LGA Connector | 1 | $1.00 | $1.00 |
| Connector-Signal, Power | 1 | $2.00 | $2.00 |
| PC Board- double sided, tooling and testing | 1 | $2.50 | $2.50 |
| Board Assembly | 1 | $10.00 | $10.00 |
| Firewire cable w/ connectors 10m  | 1 | $40.00 | $40.00 |
| resistors, assorted | 10 | $0.05 | $0.50 |
| Total |   |   | $102.00 |

- D. Winter: Nevis FVTX FEE Concept

- 19 February 2007

## Strawman Cost Estimate (Interface/Receiver)

| Component | Quantity | Price/Unit | Total |
| --- | --- | --- | --- |
| Altera Cyclone FPGA | 1 | $30.00 | $30.00 |
| Oscillator | 1 | $10.00 | $10.00 |
| resistors (assorted) | 10 | $0.05 | $0.50 |
| Voltage Regulator 1.5V (digital) | 1 | $5.00 | $5.00 |
| .01 uF Capicitors | 10 | $0.05 | $0.50 |
| .10 uF Capacitors | 5 | $0.10 | $0.50 |
| eprom – EPCS14 | 1 | $7.50 | $7.50 |
| TI TLK3101 SERDES | 1 | $28.00 | $28.00 |
| optical transciever | 1 | $25.00 | $25.00 |
| 8MB Memory | 1 |   | $0.00 |
| GLINK  | 1 |   | $0.00 |
| RG45 Connector | 1 | 1.00 | $1.00 |
| Optical receiver | 1 |   | $0.00 |
| Header Connector- JTAG | 1 | 1.00 | $1.00 |
| PC Board:4 layer, tooling, testing | 1 | $125.00 | $125.00 |
| Board Assembly | 1 | $25.00 | $25.00 |
| Total |   |   | $259.00 |

| Component | Quantity | Price/Unit | Total |
| --- | --- | --- | --- |
| Altera Cyclone FPGA | 1 | $30.00 | $30.00 |
| Oscillator | 1 | $10.00 | $10.00 |
| resistors (assorted) | 20 | $0.05 | $1.00 |
| Voltage Regulator 1.5V (digital) | 1 | $5.00 | $5.00 |
| .01 uF Capicitors | 10 | $0.05 | $0.50 |
| .10 uF Capacitors | 5 | $0.10 | $0.50 |
| wafer connector – 4X4  pair | 1 | $5.00 | $5.00 |
| TI TLK3101 SERDES | 1 | $28.00 | $28.00 |
| optical transciever | 1 | $25.00 | $25.00 |
| Duplex Fiber optic cable | 1 | $50.00 | $50.00 |
| eprom – EPCS14 | 1 | $8.00 | $8.00 |
| JTAG Conector | 1 | $1.00 | $1.00 |
| PC Board-4 layer, tooling, testing | 1 | $125.00 | $125.00 |
| Board Assembly | 1 | $25.00 | $25.00 |
| Total |   |   | $314.00 |

- D. Winter: Nevis FVTX FEE Concept

- 19 February 2007

## DCM vs. DCM-less

- Current concept of the Receiver mates directly into PC (EvB) on PCI bus
- Implements PCI protocol in FPGA code, *but this could replaced (bridge chip, eg.) if absolutely necessary*
- Has provided a flexible platform in its current form for quick development of tools to interact with FPIX chip
- Future generations would format data into packets and frames, like any other PHENIX subsystem
*If it must be retargeted to deliver payload to standard DCMs, we do not anticipate any major challenges*

- D. Winter: Nevis FVTX FEE Concept

- 19 February 2007

## Fiber communications: Packet Protocol

- Packets consist of one or more 32-bit words
- Packets contain 16-bit header + one or more 16-bit data words
- Two flavors of packets: control and pixel data
- Control packets are all one 32-bit word long
- Pixel data packets with odd numbers of data words are padded with 16-bit 0 word (guaranteeing packets are multiples of 32-bit words)

- Upper 2 bits of header = 112
- Three types of control packets
- Interface commands
- Immediate commands
- FPIX API commands
- Format of header depends on which type
- Data also depends on type
- Interface: 16 bits of data
- FPIX API: 13-bit FPIX command

- Header
- Upper 2 bits of header = 102
- Bits 12-8: 5-bit chip id
- Bits 7-0: 8-bit BCO
- Data
- Upper bit of data word = 02
- Bits 14-3: 12-bit pixel addr
- Bits 2-0: pixel adc

- 1

- 1

- 1

- 0

**...**

- D. Winter: Nevis FVTX FEE Concept

- 19 February 2007

## Packets, PCI Card, & Physical RAM

- Receiver uses ABC registers to read or write packets between the physical RAM of the CPU and the onboard RAM
- ABC registers implement memory as a ringbuffer
- Transfers take place using DMA
- Control packets are echoed back to user
- Written out via channel 0
- Echoed back to channel 1, with data field replaced with timestamp of when command was executed (if command doesn’t request data to be returned)
- Useful when correlated with “Execute next command at specified time” immediate command

- D. Winter: Nevis FVTX FEE Concept

- 19 February 2007