# Presentation: 516356

## XFT Monitoring + Error Rates

- Alison Lister
- Robin Erbacher, Rob Forrest, Andrew Ivanov, Aron Soha

- Alison Lister, UC Davis

- L2 Upgrade review 19th June 2007

**Notes:**
Capitals in bullets

## Monitoring: TrigMon

- Timeline of TrigMon releases for XFT L2
- 8th January: First official XFT L2 version of TrigMon running in Control room
- Basic bank comparisons between data and data or simulation as well as data integrity checks (e.g. bunch counters) at each stage: Finders, Pulsars
- 3rd March: Updated monitoring
- Better checks for errors in bank comparisons
- 16th April: Latest version running
- Includes timing information for various components of the system
- Includes better diagnostics for the “out-of-sync” problems
- Includes plot for CO (but not in checklist yet)
- Further update will probably be required to finalise the checks, add canvas colour when bad, solve minor bugs
- To be done once all hardware is finalised

- Alison Lister, UC Davis

- L2 Upgrade review 19th June 2007

**Notes:**
TrigMon:
* Bank comparisions
Latency + Timing

## Monitoring: Hardware

- TrigMon code looks at bank comparisons at different stages
- XSFD (L1 path in Finders)
- T2FD (L2 path in Finders)
- TP2D (Pulsar bank)
- XSFD Sim where the simulation starts from the XTCD bank
- T2FD Sim where the simulation starts from the XTCD bank

- TP2D (data) vs T2FD Sim

- CO plot:
- Summary of all errors in whole system
- Other plots used to diagnose what the problem is

- Finder-space

- Pulsar-space

- Slot

- Crate

- Board

- Input Fiber

- Alison Lister, UC Davis

- L2 Upgrade review 19th June 2007

**Notes:**
TrigMon:
* Bank comparisions
Latency + Timing

## Monitoring: Hardware

- TP2D vs T2FD Sim

- TP2D vs T2FD

- XSFD vs XSFD Sim: SL7

- Run 243521
- Lumi 240-95e30
- Error rate:
- Max 3 errors / finder / 20’000 events

- 2 event with XSFD error

- 1 event with T2FD error
- Bunch Counter mismatches

- 3 events where TP2D bank structure is corrupt (!= 12 finders in XFT01)

- Expert canvases used to diagnose problem

- Alison Lister, UC Davis

- L2 Upgrade review 19th June 2007

**Notes:**
TrigMon:
* Bank comparisions
Latency + Timing

## Monitoring: Timing

- Timing information needed for commissioning and debugging of system (“expert only” plots)
- Timing information is available for most components of the system
- Arrival time for each finder packet to the Pulsar
- Available only when DATA I/O enabled
- CPU timing
- Main one used for tests
- L2TS Signal out to TS
- Signal back from TS
- Unpacking of different inputs
- TL2D: Algorithm timing

- Alison Lister, UC Davis

- L2 Upgrade review 19th June 2007

**Notes:**
TrigMon:
* Bank comparisions
Latency + Timing

## Monitoring: Timing

- Lots of work put in by L2XFT-team to reduce timing delays in new PC
- CPU Timing for parasitic PC with XFT (no algos) vs “old” PC

- Most events new PC ~1us faster than old one
- For SVT fast abort ~6us slower
- Findings independent of which inputs are being read out
- i.e. same for L2CAL only and XFT+L2CAL

- Alison Lister, UC Davis

- L2 Upgrade review 19th June 2007

**Notes:**
TrigMon:
* Bank comparisions
Latency + Timing

## Error Rates

- When no obvious hardware problems are present (“SEU”) rate
- ~ 3 10-4  to have an error somewhere in the system
- ~ 8 10-6 to have an error in a given “finder”package
- ~ 1/4 of them are related to problems upstream (also seen in L1)
- These problems don’t cause “operational” problems with L2 PC (e.g. data corrupt for 3 errors in a row that causes an HRR)
- Could affect outcome of trigger decision for single events
- Additional ~1-2 errors per 10’000 events where the bunch counters for the first finders in each crate have a bunch counter mismatch (off by 1)
- Data agrees with simulation so not an operational problem
- Hardware failures
- “out-of-sync” problem from March- April
- Finder problem with L2 (data corrupt)

- Alison Lister, UC Davis

- L2 Upgrade review 19th June 2007

**Notes:**
TrigMon:
* Bank comparisions
Latency + Timing

## “Out of Sync” Problem

- March-April problem:
- Rate ~once per store on average
- 1 chip in a Pulsar board gets out of sync wrt other chip and rest of event
- Stays out of sync until an HRR is issued
- HRR now issued from parasitic PC after 3 consecutive problems
- Not related to pulsar board

-  Problem solved after swapping input fibers
-  Think that 1 of the SL3 fibers might be slightly flakey
-  In current configuration the problem hasn’t returned for ~2 months

- Alison Lister, UC Davis

- L2 Upgrade review 19th June 2007

**Notes:**
TrigMon:
* Bank comparisions
Latency + Timing

## Finder L2 Problem

- Problem occurs when 1 finder contains corrupted data
- L1 path agrees with simulation
- L2 path does not agree with L1
- Causes operational problems (L2TO) as data gets out of sync (i.e. corrupted data in PC) and stays out of sync until problem is solved or XTC is masked on
- Seen in TrigMon plots (e.g T2FD vs XSFD)
- Happened once before in December when not running parasitically
- Related to the finder board
- Masking on XTC stopped the problem
- Swapped out the boards but was never investigated further
- Happened again end of May (?)
- Masking on XTCs allowed us to keep taking data
- Unmasked XTCs to narrow down problem and problem did not re-appear
- “Flakey” problem :-)
- Rate seems to be once every ~5 months

- Alison Lister, UC Davis

- L2 Upgrade review 19th June 2007

**Notes:**
TrigMon:
* Bank comparisions
Latency + Timing

## Conclusions

- Monitoring of all stages has been running in TrigMon for almost 6 months
- Still minor code changes to be implemented for “final” version
- Error rate for “normal” data taking is manageable
- Diagnostic tools for hardware failures have shown to be useful in quickly finding and solving problems

- Alison Lister, UC Davis

- L2 Upgrade review 19th June 2007

## Backup

- Alison Lister, UC Davis

- L2 Upgrade review 19th June 2007

## Finder L2 Problem

- My personal suggestion for “what to do if this problem happens again”
- Diagnostics
- Many L2TO causing data to no longer be taken
- L2 will be paged
- TrigMon plots should show some sign of which Finder is cause
- Not necessarily the case as TrigMon only runs on selected trigger path and if no data is being taken won’t appear recorded data..
- L2 PC logfile should show which XFT Pulsar is corrupted
- Dump Pulsar banks to narrow it down to 1 finder
- XFT should be paged
- Solving the problem
- Trial and error of masking on XTCs (as can’t mask all of them at once) until problem goes away
- Need to understand where the problem is
- Problem needs to happen again before can investigate further
- If finder hardware problem we will need help diagnosing
- Current statistics tell us this might happen every ~ 5 months

- Alison Lister, UC Davis

- L2 Upgrade review 19th June 2007

**Notes:**
TrigMon:
* Bank comparisions
Latency + Timing