# Presentation: 009410

## L3 DAQ

- Doug Chapin
- for the L3DAQ group

- DAQShifters Meeting 10 Sep 2002

- Overview of L3 DAQ
- uMon
- l3xqt
- l3xmon

## The L3DAQ System

- ROC
- SBC

- ROC
- SBC

- ROC
- SBC

- ROC
- SBC

- CISCO 6509
- Ethernet
- Switch

- trigger
- disable

- Farm
- CPU

- 48 farm
- nodes

- All ethernet, except for TFW communication

- Routing
- Master
- CPU

- TFW info

- Farm
- CPU

- Supervisor
- CPU

- COOR

- 63
- readout
- crates

## Communication Flow

- ROC
- SBC

- ROC
- SBC

- trigger
- disable

- Routing
- Master

- TFW

- Supervisor

- COOR

- Run information:
- crate and node list by trigger bit

- EVB

- Filter-shell

- ScriptRunnertriggerprogramming

- Farm node

- Crate listby event

- Bits firedby event

- “node ready”

- Routing info:
- destinationfarmnodeby event

- SCL: accept

## ReadOutCrate Example

- SBC

- VME block
- transfer(s)

- J3

- Slave Ready

- Done

- Controller
- Card

- SCL

- DataCard

- DataCard

- SBC
- intiated

- ctrl reg
- programming

- Ethernet

- CPUCard

## SBCs

- Intel ~1GHz, 128MB RAM, dual 100Mb ethernet, 128MB flash, VME Universe2
- VME slave interface
- VBD control register emulation
- Data buffer access
- Component front-end debugging
- J3 Control (SlaveReady,Done)
- DIO PMC add-on card
- Also drives status LEDs
- Software connections
- Actively connect to RM
- Recieve up to 2 connections from each farmnode

## SBC Operation

- Event Fragment
- Queue (23)

- Route Queue
- (100)

- Event fragments
- from VME

- SBC
- Process

- Event Tags
- From RM

- Match head event tag to head fragment via event number
- Fragments have one sec timeout
- If no tags, throw away head event after 1s
- Route queue circular, non-blocking
- Tags can be overwritten if no fragments are available
- Event number mismatch
- If frag ev# > tag ev#, throw away tag
- If frag ev# < tag ev#, drop fragment

- To node

## Supervisor Process

- Connections to
- RM
- Receives crate and node list per bit
- RM complains if needed nodes/sbcs not connected
- Scriptrunner processes
- Receive complete trigger programming
- Passed on from COOR
- When trigger downloaded
- Node processes may get stuck
- Big trigger lists take several minutes to download
- *N0* connection to SBC or EVB processes

## Farm Nodes

- Dual 1GHz Intel, 1GB RAM
- 48 nodes so far
- Runs Event Builder (EVB) and Filtershell processes
- Filtershell Processes (usually 2)
- This is ScriptRunner (L3 filters)
- Recieves trigger programming info from supervisor
- Recieves full event from EVB process
- EVB Process Operation
- Connects to all SBCs (incl RM), all the time
- Receive crate list by event# from RM
- Sends no. of free buffers to RM (“node ready”)
- Builds event from received fragments
- 1 - 5 second timeout

## Routing Master (RM)

- Recieves “run” information from supervisor
- Farm node list and crate list per bit
- Gets bits fired per event# from TFW
- Recieves no. of free buffers from each farm node
- Decides which nodes receive which events
- Sends routing info by event# to SBCs
- Sends crate list by event# to farm nodes
- Disables triggers when necessary

## RM Operation

- Process generates event tags and crate list
- based on bits fired and run info
- One sender thread and event tag queue per SBC
- Wait for 10 tags or 250ms, then send
- Minimize ethernet overhead
- Crate list sent to target node on every event

- sender
- thread

- To
- d0sbc001b

- sender
- thread

- To
- d0sbcXXXb

- Event Tag
- Queue

- Main RM
- Process

- TFW bit mask
- and event#

- Crate list to target node

## L3 Disable

- RM process running on TFW crate’s SBC (d0sbc001b)
- Spy on the TFW data block
- Extract L1 fired bits and L3 transfer number
- When L3-disable is needed
- Update RM monitoring information
- RM tells SBC to stop reading out the TFW crate
- TFW crate goes 100% front-end-busy and stops triggers
- Acts as global disable
- Cons
- Actual L3 Disable line is not used
- Cannot easily separate out l3daq backup (accounting issue)
- Shifter must check L3DAQ monitoring every time TFW crate is 100% front-end-busy

## Monitoring

- All processes (SBC, RM, super, EVB) produce monitoring info
- Monitor clients
- l3xmon
- ScriptRunner status (processing,flattening, sending,...)
- üMon is your friend
- SBC, RM, and some Supervisor info
- EVB info (missing crates, input rate)
- Send comments to _[sean@fnal.gov](mailto:sean@fnal.gov)_
- l3xqt
- Mostly expert info
- Daqmon info (L1/L2 busy fractions) useful to shifters

## Slide 13

## Slide 14

## Slide 15

## Slide 16

## Slide 17

## Scripts

- Online scripts
- l3xreset
- can restart farmnode processes\
- EVB and ScriptRunner simultaneously
- can restart Supervisor process (node d0lxmast)
- l3xdaq_reset
- Restarts sbc processes (incl RM)
- SBC scripts
- is_crate_requesting_readout.sh (SlaveReady asserted?)
- getInfo.sh  (sbc driver stats and status)
- reset_all.sh (restarts processes)

## L3DAQ Shifter Webpage

- _[http://www-d0ol/www/groups/l3daq/shifter](http://www-d0ol/www/groups/l3daq/shifter)_
- “What To Do When”
- Extremely useful
- Frequently updated???
- On-call schedule
- uMon documentation!!!
- Logfile access

## Typical Problems

- Prescales set for < 10Hz
- Unpredictable results (all crates missing)
- Not enough nodes in run
- L3 disables (TFW 100% FEB)
- SMT/CFT/CAL in full-readout mode?
- Farmnode loses connection to SBC
- Crate is missing (reset farmnode to fix)
- Crate 100% missing
- Almost always a component problem
- Check Route and Event queue state in uMon

## Improvements to Come...

- Phase out l3xmon
- Currently uses a separate monitor server
- Move features into uMon
- Trigger Crate
- Set L3 disables correctly