# Presentation: 526264

## FEE Overview—LANL Perspective

- Forward Silicon Meeting
- August 28-29, 2006  at LANL
- Melynda Brooks
- Los Alamos National Laboratory

## Overview of Proposed Readout Scheme - LANL

- Specification Drivers/Interfaces to PHENIX
- Silicon Data Streams
- Requirements for getting data into PHENIX Data Stream
- Requirements for getting data to Lvl-1 Boards
- Ancillary functions required
- Possible locations
- Support and Cooling required

## Drivers for Specifications Document

- Specifications document :
- Geometry and occupancy:  FVTX is expected to be built so that it maintains or reduces per chip occupancy so that it will not introduce additional constraints on data stream
-  LDRD data rates have been used as baseline
- Simulated Central AuAu events with factor 2 contingency used as worst-case data stream that must be handled~10 hits/chip for FPIX
- Interfaces for Readout Cards assumed:
- Input from front end: FPIX, 4 data lines/chip, download, calibration lines
- Other Input:  PHENIX T&FC, slow-controls communication somewhat undetermined (PHENIX Ethernet/Arcnet...)
- Output to:  PHENIX DCM, Iowa Lvl-1 trigger board
- Baseline Assumption:  LDRD and FVTX will share the same functional readout plan; physical details to be worked out--open to modification

## I/O

- I/O Choices:
- FPIX, 4 data lines/chip – to give opportunity for Lvl-1 participation primarily
- PHENIX T&FC – PHENIX standard:  provides beam clock, 4*beam clock, Lvl-1 accept, mode-bits to tag each bunch crossing
- slow-controls communication – to allow download/readback to/from chips, download to FPGA, read-back of monitoring chips, control of calibration system...
- PHENIX DCM – PHENIX-wide standard for receiving data from FEMs and relaying to event builder.
- Iowa Lvl-1 trigger board – requires fast delivery of front-end data

- From Chi, on by-passing the DCM:
- The DCM design effort will be paid by PHENIX upgrade program. Hardware cost of the DCM port is about $500-600 per optical port. It is very cheap compare to EDIA cost for 6 month - 1 year design/testing/debugging. Your EDIA money should be thinking about FVTX electronics + test jig and not to design a "DCM+SEB".
- The DCM II backplane will run instantaneous rate of 6.4 Gbits/sec on the DCM backplane and use 2.5 Gbits/sec PCI expresses backplane on the PC. We are working on the design for next 6-10 years.
- AS far as PHENIX concern, we would like to have uniform designs at the DCM level. It minimizes cost of design/man power/trouble shooting/maintenance efforts. It always makes sense to the reviewers in any PHENIX programs. If your "DCM+SEB" has problem several years from now, who do you call? It is not essential to the FTVX effort. It is not just me. You will hard time convince any PHENIX upper level management, including Bill Zajc, to design a new "DCM+SEB" just for FVTX. Even we agree it is impossible to argue that to the DOE review committee. It is our job to kill some of "great" idea.

## DAQ Interface Proposal

- LV, HV

## Segmentation

- Uncertain
- 96 chipsone ROC/FEM starts to get close to a number of limits:
- I/O lines handled by 1 FPGA (if single FPGA),
- data size handled by 1 DCM (if 96 chipsFEM)
- probably too high to give comfortable failure-rate analysis result
- Should be balanced against:
- Number of fibers, FEMs, DCMs
- Cost
- ...
- ROCFEM mapping not necessarily 1:1

## Readout Stream Location – Somewhat Uncertain

- ROC Might be located at the “collar” (analogous to picture-frame location), the “endplate”, or co-located at both places
- Issues:  how to cool and support at either location, how soon do we need to reduce bundle sizes
- Baseline: LDRD should follow same functional divisions to keep R&D reasonably common between 2 systems, readdress this baseline as designs evolve

- “collar” locations

## Ongoing Works

- ROC
- Mark looking at component selections and overall design
- Anuj working on possible FPGA implementation of data stream handling to see if any issues with baseline proposal, handling of multiple chips and multiple data lines, etc.
- Expect to connect to real FPIX chips soon
- To address: do we handle data stream to Lvl-1 and FEM separately or not; for either, how do we do it
- Pat producing test card to try out chip pulsing scheme, working with Anuj/Mark
- FEM
- Sergey working on FPGA implementation for same reasons as Anuj
- Hope to connect ROC-FEM soon
- In Discussion or Progress
- More detailed specification of ROC-FEM interface?
- Clock distribution(s), speeds to run each piece...
- When to insert fiber drivers?
- Setting up DAQ to test compatibility with PHENIX environment
- Space/Power Consumption

## Critical Issues to Address

- Columbia-LANL currently following independent paths, must converge or some efforts will likely be wasted
- Agree to some component/language selections
- Specification of interfaces formed in more detail where necessary
- Division of labor must be determined
- Agreement to space options to explore for readout, pros/cons of each
- Agree on milestones for the near future

## Backup Slides

## Possible Segmentations – Upper limits

- 1⁄2 layer = 8*12 = 96 chips

- 1 ROC:
- 9 output fibers
- 1 I/O slow-control fiber
- LV, HV distribution (?)
- Calibration distribution

- ROC

- FEM

- 1 FEM:
- Output to DCM
- Input from T&FC
- I/O from Ethernet for Slow Controls

- Questions:  Does this segmentation make sense for Lvl-1?  How would 1⁄2 layer be split (front/back, left/right...)? Is number of chips maybe too many? (simulate)

## Possible Segmentations – Upper Limits

- 1 Wedge*4 layers ~99 chips

- 1 ROC:
- 9 output fibers
- 1 I/O slow-control fiber
- LV, HV distribution (?)
- Calibration distribution

- ROC

- FEM

- 1 FEM:
- Output to DCM
- Input from T&FC
- I/O from Ethernet for Slow Controls

## Some Channel Counts

## Some Occupancy Numbers

***LDRD***
- For the LDRD detector we calculate from GEANT simulations that there will be 0.14% track occupancy.
- We would expect the chips at the inner-most radii to have to deliver up to (0.0014)*(2)*(1.3 strips/track)*(2816 channels/chip) = 10 hits/chip in Central AuAu.
***FVTX***
- For the FVTX detector we calculate from GEANT simulations that there will be 2.8% track occupancy
- If we assume 256 channels per chip then the data to be delivered would be up to (0.028)*(2)*(256 channels/chip) = 14 hits/chip in Central AuAu.

## Front-end Chip ↔ ROC

- Memory

***Input***
- Data lines from PHX, FPIX chips
- One clock line (pair) per chip
- Serial download stream from FEM, to be passed to chips
- Serial readback from PHX, FPIX chips (?)
- FPGA JTAG data from FEM, ARCNet/Ethernet for reprogramming FPGA
- Calibration pulser to be passed to chips.  From FEM or direct?
- LV and HV to distribute to chips and sensors?

***Output***
- Serial data stream (fiber) out to FEM.  Stream is to contain info outlined in 6.3.  Included status word to indicate results of error checking?
- Data stream out should also have inserted into it slow-controls information as needed:  current, voltage, temperature monitoring, etc.
- Serial download stream, to chips
- Calibration pulser to chips, coming from on-board DAC
- LV and HV to chips and sensors?

## ROC Block Diagram

## Front-end Chip ↔ ROC

***ROC Function***
- The readout card must provide the following functions:
- Deserialize data coming from the N number of FPIX/PHX chips
- Strip off synch words that are in the data stream
- Arrange data words and pass them on to the FEM in such a way that the following information is maintained or added:
- Beam crossing number (at least 6 bits)
- Pixel/strip location within chip
- Chip ID
- ADC bits (3 bits)
- Pass download strings to FPIX/PHX (and do readback)
- Provide calibration pulses from on-board DAC.  Control coming from FEM, coming from ARCNet/Ethernet.
- Provide onboard memory for power-up chip download

- Memory

## Some Starting Points for ROC

- Clock will be provided per chip, so don’t need to immediately decide on clockless readout or not
- LDRD segmentation:  96 chips1 ROC (may be too many because of number of I/O required, but will determine from simulations)
- FVTX segmentation ?:  don’t need right now, but 4 wedges*4 layers seems reasonable (99 chips if 256 channels/chip)
- Fiber out
- Working on specifying data format out:  need relatively soon so both boards can move forward
- Want to do some space/power calculations soon to look for additional limitations
- Starting point:
- Define FEM interface
- Simulate input of N chips to FPGA and work on FPGA code
- DAC and other slow controls worked out separately?
- Download and readback of chip can be handled independently (?)

## How Many Serdes Channels/ROC?

**Depends on data format ROC********FEM.  If assume retain 24 bits/word:**
***LDRD***
- 10 hits/chip in Central AuAu.
- (10hits)*(24bits/hit)/(424x10-9) = 0.57 Gb/sec/chip
- Altera serdes at 6.375 GB/s6.375/0.57 = 11 chips/serdes channel96/11 = 9 serdes channels
***FVTX***
- 14 hits/chip in Central AuAu.
- Don’t know PHX format yet, so leave at this for now.  Will put in 9 serdes channels/FPGA as place-holder

## ROC ↔ FEM

***Input***
- Serial stream of data from ROC (fiber)  (might included slow controls information as well as data)
- Readback from FPIX/PHX chips (passed through ROC) (?)
- T+FC (beam clock, Lvl-1 accept)
- ARCNet/Ethernet to pass:
- Slow control info to FPGAs on FEM and ROC
- Initialization stream for readout chips
- Control information to calibration system onROC

***Output***
- Chip download stream, from ARCNet or Ethernet (to ROC)
- Packaged data to DCMs
- Readback from chips (coming from ROC and going to outside world)
- Split input data stream and send to Lvl-1 board(s)
- Strip off slow-controls info from ROC and send to slow-controls data stream

- ARCNet/Ethernet

- Data to DCMs

- Data to Lvl-1

- Slow Controls Data

## ROC ↔ FEM

- ARCNet/Ethernet

- Data to DCMs

- Data to Lvl-1

- Slow Controls Data

***FEM Function***
- Buffer data
- Grab data from event of interest upon a Lvl-1 accept
- Build packet for DCM and ship to DCM
- Pass download stream to readout chips
- Receive readback stream from chips (for debugging)
- Pass calibration control to ROC (coming to FEM via ARCNet/Ethernet)

## ROC ↔ FEM

## Some Starting Points for FEM

- One FEM/ROC
- Interface to DCM and other PHENIX subsystems well defined so just need ROC input defined
- Once ROC input is defined, can start building up system:
- Simulate input data packets from ROC
- Input Lvl-1 for event building (build up to PHENIX DAQ system)
- Pass packets to DCM
- Handling of slow controls (FPGA downloads, calibration, chip inits...) can almost be done separately
- ...

## Readout Format Options - LDRD

- Retain 24 bit-words (row, column, ADC, BCO):
- Data out = Chip ID (4 bits) + (10 hits/chip*24 bits)/(424 ns) = 0.57 Gb/s
- Chips/serdes = 2.5/0.57 = 4
- Chip ID (4 bits), BCO (6 bits) in header, data = 3 bits ADC + 12 bits channel ID
- Data out = (10 bits + 15*(10 hits/chip))/(424ns) = 0.38 bits/chip
- Chips/serdes = 6
- Problem:  doesn’t work if words not out in BCO order unless you introduce delay
- Compact channel ID??

## Also Needed

- Timing diagram, clock distribution...
- Details of calibration system
- Chip/Board monitoring
- Work out LV and HV distribution
- Work out power (how much will we use) and cooling requirements
- Investigate space issues

## Where to Put ROC?

- “Collars” at each layer? (would imply different segmentations)
- End-plate area as we tried to work out in December?

- I/O to chips
- Bundle size out (per arm):
- 480 fibers
- 48 LV cables (?)
- 24 HV cables (?)

- 2 FPGA/serdes (plus DAC, volt. reg., memory... )

- Fiber Drivers (need 20 channels/double-wedge)

## {

**Lvl-1 Timing, or NOT**

- Beam Clock, Lvl-1 Decision and Fly-Back

- 1

- 3

- 2

- 4

- 6

- 5

- 7

- 12

**Get Data on way to Lvl-1 Board**

- Fiber transmit

- 8

- 9

- 10

- 11

-  Lvl-1 Decision

-  GL1 Decision

- Fiber transmit

- Gives goal that we would like to deliver all FPIX/PHX data to FEM/Lvl-1 within 4 beam clocks
- But, PHENIX can accommodate Lvl-1 decision taking as long as 40 beam-clocks (could take up to 28 clocks more or 7* as much time?)
- IF there were no Lvl-1 participation requirement, requirement is that we get data to DCMs within 50 s:

-  Chip, Roc Propagation

- FEM Buffering, Propagation

- ~400 clocks later data delivered to DCM

- All data could be sent out serially on a single line, even for AuAu and a large number of chips, and get to DCM with no problem

**106 ns**

## What does meeting Lvl-1 Drive:

- Number of chips that can go to one Serdes Channel (currently 11 chips)
- Number of fiber channels per front-end board will be larger than if we didn’t
- Speed to drive Serdes probably needs to be higher
- If we have 96 chips/ROC, we would have the following Serdes:

| Serdes Speed
(Gbps) | Data Format | Number Serdes Channels | Bits per Serdes |
| --- | --- | --- | --- |
| 6.375 | full | 9 (8) | 44 (48) |
| 2.5 | full | 24 | 16 |
| 1.6 | full | 48 | 8 |
| 6.375 | Move BCO | 6 | 64 bits |
| 2.5 | Move BCO | 16 | 24 |
| 1.6 | Move BCO | 24 | 16 |

## Getting front-end chips in sync with RHIC (I think)

-  Beam Clock:
- 4*Beam Clock
- Lvl-1 Accept, 12 (40) clocks after event of interest plus some optional user-setable delay:
- Mode Bits per bunch crossing (groups of 120 bunches):
- Can label a particular bunch within a set as a “calibration” or “timing” event, for example
- GTM file for each subsystem allows for variable delay in delivery of Lvl-1 Accept to FEMs:
- LEVEL1DELAY:    6
- ROUGHDELAY:    1
- COURSEDELAY:  0
- LABEL:  B		#Modebits A
- FEM:
- 10X120		#120 values of 0x10
- ...
- Timing in:  1) guess at FEM-Lvl-1 time 2)scan a time window using LEVEL1DELAY taking data at each setting, 3)pick optimal setting based on peak amplitudes, most hits...

**Getting front-end chips in sync with RHIC (I think)**

- 12+LEVEL1DELAY

## Getting front-end chips in sync with RHIC (I think)

-  Beam Clock:
- 4*Beam Clock
- Lvl-1 Accept, 12 (40) clocks after event of interest plus some optional user-setable delay:
- Mode Bits per bunch crossing (groups of 120 bunches):
- Can label a particular bunch within a set as a “calibration” or “timing” event, for example
- GTM file for each subsystem allows for variable delay in delivery of Lvl-1 Accept to FEMs:
- LEVEL1DELAY:    6
- ROUGHDELAY:    1
- COURSEDELAY:  0
- LABEL:  B		#Modebits A
- FEM:
- 10X120		#120 values of 0x10
- ...
- Timing in ? :  1) Set a particular bunch number with mode-bits that indicate to be used for AqBCO 2) Based on what we know about various times, would guestimate how this BCO counter aligns with Lvl-1 accept counter.  3) would fine-tune using LEVEL1DELAY and data taking with each setting

**Getting front-end chips in sync with RHIC (I think)**

- 12+LEVEL1DELAY

## Front-End Chip downloads

- two clocks (Beam Clock and Readout Clock (?) )
- control data in
- control data readback
- programming enable
- reset.
- Command Format:
- 5 bit chip address, 5 bit register #, 3 bit instruction + data if WRITE
- Info to Download:
- Thresholds (8 per chip)
- Inject and Kill Masks
- AqBCO (start BCO)
- “firefighter”, “operation” RESETs

**Front-End Chip downloads**

## Technical Questions to Answer

- Chip Specification:
- Radiation Environment:
- Calculation of SEUs
- Rad-Hard/Tolerant or configuration checking?
- ...
- Space/Cooling Issues
- Need to spec some possible chips and try layout, power calculations, cost...
- Iterate
- Is 96 chipsone ROC the segmentation we want?
- I/O issues
- Failure analysis
- Chip analysis: cost, power, complexity, space
- Format of Data ROCFEM:  pick different options and give pros/cons
- What speed to run SerDes?
- Fulfilling Lvl-1 requirements as much as possible or not?
- Limiting number of fibers out of chip or not?
- ...
- Should ROC functionality be split between two locations?

**Technical Questions to Answer**

## Technical Questions to Answer

- Clock Distribution
- General handling of slow controls:  data from chips, data from monitors (?), download data to FPGAs (memory), chips,...
- Work out details of calibration system
- Error checking/handling:  SEUs, chip overload/data integrity, hardware debug options...

**Technical Questions to Answer**

## Where to Put ROC?

- “Collars” at each layer? (would imply different segmentations)
- End-plate area as we tried to work out in December?

- I/O to chips
- Bundle size out (per arm):
- 480 fibers
- 48 LV cables (?)
- 24 HV cables (?)

- 2 FPGA/serdes (plus DAC, volt. reg., memory... )

- Fiber Drivers (need 20 channels/double-wedge)

## ROC Components and Power – Stratix Design (One option)

- Altera Stratix II GX Design:

| Component | Quantity | Size (mm) |
| --- | --- | --- |
| FPGA | 1 | 40x40 |
| LVDS-CMOS | 120 | 5.5x5 |
| CMOS-LVDS | 36 | 5.5x5 |
| DAC | 1 | 9.8x6.4 |
| Fiber Driver | 2 | 18.8x35.8x15.1 |
| Voltage Regulator | 2 | 10x10 |
| Config Memory | 1 | 5.5x5 |
| Data Memory | 1 | 5.5x5 |

## ROC Components and Power – Stratix Design (One option)

- Altera Stratix II GX Design:

| Component | Quantity | Power ? (Watts) |
| --- | --- | --- |
| FPGA | 1 | 5 |
| LVDS-CMOS | 120 |  |
| CMOS-LVDS | 36 |  |
| DAC | 1 | 0.135 |
| Fiber Driver | 2 | 1.3*2 |
| Voltage Regulator | 2 | 5? |
| Config Memory | 1 |  |
| Data Memory | 1 |  |
| Total |  | 12.7 |

- Total System Power:
-  LDRD = 8*12.7 = 102 Watts
- FVTX = 24*12.7*(2 arms) = 610 Watts

| Voltage |
| --- |
| 1.2 |
|  |
|  |
| 3.3 |
| 3.3 |
|  |
|  |
|  |
|  |