# Presentation: 009400

## L3DAQ

- An Introduction

- Michael Clements 5/5/03

## A Quick Glance 

- The DZero DAQ
- L3DAQ components
- L3DAQ communication
- An in depth look at monitoring

- Michael Clements 5/5/03

## Dzero DAQ

**63 readout crates in DAQ**
**0-20 kb/crate/event**
**Maximum events size ~250kb**

## The Players.... L3DAQ Components

- SBC
- 2948G Ethernet Switches
- 6509 Ethernet Switch
- Farm Nodes

- Routing Master
- Supervisor
- Node Processes
- SBC Processes
- Monitoring

- Michael Clements 5/5/03

**L3 is comprised of both hardware and software components**

**Hardware**

**Software**

## L3DAQ - Overview

## L3DAQ – Communication Flow

## SBC – Single Board Computer	

- 2 main functions
- Readout controller for event data (many instances)
- Routing master (1 instance)

- PIII 933Mhz, 128Mb RAM, 128Mb flash, 2 100 Mb/s                 Ethernet ports

- Tundra Universe II chip as VME-PCI interface
- Programmable DMA controller

- Michael Clements 5/5/03

- PMC expansion slot
- Holds BVM DIO module responsible for VME         readout over J3 backplane

## SBC – Logic & Operation 

- Event fragment thrown out after 1s if still waiting for route tag

- Route tags thrown out after several seconds if no event fragment

- If evt# in event fragment > evt# in route tag, throw out route tag

- If evt# in event fragment < evt# in route tag, throw out fragment

- Michael Clements 5/5/03

## SBC – How a Crate Reads Out

- Michael Clements 5/5/03

## SBC – Routing Master Basics

- Receives crate list per bit and list of farm nodes from Supervisor
- Receives bits fired, event number from TFW
- Chooses which nodes to send event for filtering
- Sends routing info to SBCs based on event #
- Sends crate info to nodes based on event#
- Receives number of available buffers from nodes
- If number of available buffers in the farm becomes to low, RM
-  can disable triggers allowing farm can catch up

- Michael Clements 5/5/03

## SBC – Routing Master Comm. Flow

- Michael Clements 5/5/03

## Ethernet Switches	

- 2984G switch
- Concentrates signal from up to 10 100Mb/s connections to 1Gb/s optical line
- Limits number of 100Mb/s connections to prevent network congestion
- 6509 switch
- Receives the Gigabit links and connects to the L3 nodes

- Michael Clements 5/5/03

## L3 Farm Nodes

- Michael Clements 5/5/03

- 48 Dual PIII 1GHz, 1GB RAM
- 34 Dual AMD Athlon 2GHz, 1GB RAM
- Dual Ethernet ports
- One dedicated to online system
- One dedicated to L3DAQ
- Runs several processes
- Event builder (EVB)
- Filtershell process (actually 2 running instances)

## Node – Event Builder

- Michael Clements 5/5/03

- Always connected to all SBCs (including RM)
- In-progress has a 1s timeout, resulting in incomplete event
- Max of 3 buffers advertised to RM to limit data flow into 6509

## Supervisor

- Primary function is L3 interface to COOR
- Communicates with ScriptRunner and RM
- No communication with SBCs or EVB
- Sends RM crate list per bit and node info
- RM complains if nodes/SBCs not connected
- Sends trigger programming to ScriptRunner when a trigger download is performed

- Michael Clements 5/5/03

## uMon 

## uMon – Front End Crates

- uMon monitors from the Routing Masters perspective!
- When crate color is....
- Red:  At least one event fragment from the crate did not reach the farm.  Included in a run.
- Yellow: Monitoring information is not available for the crate.  Included in a run.
- White: Included in a run and experiencing no problems .
- Grey: Crate not included in a run.

## uMon – Front End Crates, Part 2

- “Inc%: XX” monitors the percentage of events from a crate that have been missing.
- This is a running total.  Hitting the “Reset Inc Counters” button will reset this total to zero.  To go back to the running total simply hit the “Unreset Inc Counters”
- Live SBC connections to the farm nodes are shown as beige boxes, otherwise black.  The uppermost left box is node 0 (which does not exist and is therefore always black), next is node 1 and so forth.  The lowermost right box is the SBCs connection to the Routing Master (beige connected, black not).
- The strip chart is the rate which the SBC is reading from crate electronics.

## uMon – Front End Crates, Part 3

- Yellow block on the strip chart indicates no monitoring info from SBC
- Route Q bar chart:
- Fractional use of route Q
- Event Q bar chart:
- Fractional use of event Q
- Both charts utilize yellow on black background
- Route Q should be nearly empty and event Q should be < 2/3

## uMon – L3 Input/Inc. Event Rate

- Strip chart shows the rate at which the RM is receiving/sending event tags
- The “L3 Input Rate” is just the instantaneous rate
- Pull down list allows the vertical scale of the chart to be changed
- A yellow bar at the top of the strip chart indicates that there is not monitoring
-  information available.

## uMon – L3 Input Rate By Bit

- Blocks, going from left to right, correspond to L1/L2 trigger bits 0 to 127
- The grayscale of each block represents the rate of the particular trigger
-  with black being 0 Hz and white maximum rate.
- Clicking on a block will display the corresponding bit and its current rate
- Scale may be selected from drop down menu

## uMon – Route Fifo Depth	

- “Global L3 Disable” gives the fraction of time since the last update
-   that the Routing Master has disabled triggers.
- “Route FIFO Depth” histogram shows the number of events in the TFW FIFO as determined by polls from RM conducted every 10ms.
- “RM-Farm conn” shows the RM connections to the farm nodes.

## uMon – EVB Rates

- “EVB Rates” shows the
-  EVB input rate into
-  each individual node.
-  Missing event rate as
-  fraction of the total for a
-  node will show as a red
-  bar chart on top of the
-  bin.
- “Node” strip chart
-  shows the EVB rate of
-  the chosen node over a
-  period of ~20 minutes

- “Total EVB Rate” shows the input rate into all EVB processes on L3 farm.

## uMon – Routing Groups

- Selecting a routing group will highlight all
-  crates associated with that group.
- Input is given for each group
- Disable fraction is given for each group

## fuMon

## fuMon – Output/Pass Rate 

- fuMon monitors from the nodes perspective!
- Main strip chart shows total output rate of the farm
- Smaller chart shows pass fraction as grayscale bars
- Scale may be changed by drop down menu
- Reject is the ratio of input_events/sent_events over entire farm

## fuMon - Nodes

- When the color is...
- Orange: 10 or more complete EVB buffers waiting for processing by FS
- Red:  One or more events were incomplete and dropped by EVB
- White: In a run and experiencing no problems
- Grey: Not in a run.

- Three black dots represent # of advertised free buffers to RM
- First bar chart represents in process EVB buffers
- Second bar chart represents complete EVB buffers

## fuMon – Nodes, Part 2

- 2 beige boxes in the middle represent the state of the two filtershell processes
-  running on the node.
- 1st (highest) position: WAIT
- 2nd position: FILTER
- 3rd position: FLATTEN
- 4th position: SEND
- Top strip chart shows the event rate into the filtershell processes
- Bottom strip chart shows the pass fraction for the filtershell processes
- A yellow bar on the top of the strip chart indicates there is no monitoring info

## fuMon – Node Details	

- Clicking on a particular node will give you detailed instantaneous
-   information about that node.

## fuMon – Node CPU Usage

- “CPU Utilization” shows what fraction of the nodes total processing power is
-  being consumed.

## DAQ Dialog

## Typical Problems 

**Not enough nodes in run**
**Symptom: L3 disables, nodes in FILTERING**
**SMT/CFT/CAL in full-readout mode?**
**Collector Router backed up**
**Farmnodes orange (SENDING) in fuMon**
**L3 output rate too high or event size too large**
**Farmnodes wont respond to Super when SENDING**
**Supervisor appears hung**
**Must PAUSE and wait OR restart farmnodes and Super **
**Crate 100% missing**
**Almost always a component problem**
**Check Route and Event queue state in uMon**

## Some Useful Scripts

**Supervisor**
**start_daq l3supervisor **
**Nodes**
**l3xreset  <node>, or l3xreset to reset entire farm**
**Monitor server**
**l3xdaq_monitor_server**
**SBC**
**l3xdaq_reset <sbc>**
**SBC scripts**
**is_crate_requesting_readout.sh**** (SlaveReady asserted?)**
**getInfo.sh****  (sbc driver stats and status)**
**sbc_config_rdwr**** (VME targets and modes)**
**reset_all.sh (restarts processes)**

## Website

- _[http://www-d0online.fnal.gov/www/groups/l3daq/default.html](http://www-d0online.fnal.gov/www/groups/l3daq/default.html)_
- Documentation for all monitoring packages
- This and previous Level 3 talks and papers
- “What to do when” page for troubleshooting L3 problems
- Log files for the SBCs, Supervisor, Monitor Server, & daqAI
- On call expert list

## Slide 35

## Missing Node Info

**Missing Node Info**