# Presentation: 382166

## DAQ Back End Tutorial

**The players**
**Data and flow control diagrams**
**Brief description of each application**
**Configuration and log files**
**Starting and stopping applications**
**Monitoring**
**Status web page**
**How to reach the experts**
**Summary**

## The Players

** ****This**
**tutorial**
**    ****is**
** ****about**
**   ****the**
** ****green**
** ****guys**

## C/R and DD

**C/R = Collector/Router**
**Receives events from L3 nodes and SDAQ DMs**
**Routes an event according to the event’s **_**stream(s)**_**:**
- Forwards calibration events to the CM
- Forwards normal events to their DL(s) and as many as possible to the DD as well
- Events w/o valid streams are dropped (with a warning)
**Log file contains their event header dumps**
**Recently observed probability O(10****-4****), clustered in a few spikes**
**DD = (Data) Distributor**
**Receives events from the C/R(s)**
**Serves them to connected Examines according to each Examine’s stream and trigger wish lists**

## The DL

**DL = DataLogger**
**Receives events from C/R(s)**
**Saves them in files based on stream, run and partition (“{stream}_{run}_{partition}.raw”):**
- Each covers an integral number of luminosity blocks
- Each has an associated event catalog file
- Late events go to partition 000
- Events that arrive after their run has ended are dropped (with a warning)
**Log file contains their event header dumps**
**Recently observed probability O(10****-6****), clustered in a few spikes**
**Is told by DLSAM when to move to the next disk**

## SDAQ and DM

**SDAQ = Secondary DAQ Supervisor**
**Controls IOC processes (“crates”) participating in SDAQ runs (calibration, monitoring, tests)**
**Starts and controls corresponding DM processes**
**DM = DataMerger**
**Receives SDAQ event fragments from IOC processes, merges them into complete events and sends those to a CR**
**Receives calibration and/or monitoring summaries from IOC processes and sends them to a CR, which will forward them to the CM**
- Currently this is the main use of the SDAQ

## The DSM

**DSM = Data **(Flow)** State Manager**
**Controls all CR, DD and DL instances (“slaves”) on behalf of COOR**
- DD is non-vital  its replies are ignored
- Puts any restarted “slave” into current DAQ state
**Instructs CRs to which DL or CM to route a stream**
**Balances load over DLs**
- Currently there is only 1 DL
**Supplies L3 and SDAQ Supervisors with weighted list of CRs**
- 2nd CR needed when > ~80 L3 nodes

## DLSAM and DLCAT

**DLSAM = DL-SAM interface**
**Tells DL when to move to the next buffer disk**
- When occupancy or time limit is reached
- Currently 2 TB available (30 disks)
**Enough for 2 days of continuous (!) data taking at 12.5 MB/s**
**Uses SAM FSS to transfer completed files to Enstore**
- Checks PNFS before deleting any file
- Lets DLCAT fix meta-data missing in the SAM DB
**When the FSS lost its connection to the DB server**
**DLCAT = DL ****(event)**** Cataloguer**
**Checks and fixes meta-data missing in SAM DB**
**Stores and checks event catalogs in SAM DB**
- SAM DB (Oracle) will detect duplicate events, if any

## Configuration and Logs

**/online/data/d0online/d0online_names.py**
**Shows on which host(s) an application runs**
**/online/data/datapath/****$host****/****$app****.setup**
**Configuration file for application ****$app**** ****on ****$host**
- $app can be collector, datalogger, distributor, dlcat, dlsam, dsm, sdaq
**/online/log/****$app****/****$year****/****$month**
**Holds formatted logfiles**
**/online/log/****$app**
**Holds stdout/stderr files (****out.***** and ****stop.*****)**

## Starting and Stopping

**Start an application**
- setup d0online
- start_daq $app
**Stop an application**
**First stop **_**all**_** ongoing runs!**
**Not needed for DLCAT (nor DLSAM, but see below)**
- setup d0online
- stop_daq $app
**DLSAM must run when the DL is recording a run**
- It instructs the DL when to move to the next disk

## Monitoring

**Back end status web page**
**http://www-d0online.fnal.gov/www/daq/**
**   ****operations/status/onlstatus-auto.html**
**Updated every minute**
**Frequently checked by the experts**
**Usually sufficient to verify the status or to discover problems**

## Example Status Page Top

**Here the DSM reports all being OK for the ongoing run:**

**DLSAM file **_**state**_** numbers are approximate.**

**Buffer disk subdirectory contents.**

## Example Status Page Bottom

**Here DLCAT is busy storing 3 event catalogs, another is waiting to be checked against the DB, and another has just arrived:**

**The last ****100**** and ****1000+**** lines in an application’s log file, per host.**
**Statistics for the last N files transferred.  Trouble shooting hints.**

## DSM Status Snapshot

**Usually shows messages pertaining to the most recent COOR commands**
**Click on the DSM log file snapshot or episode link to see more details**
**Shows if the DSM is “ok” or “bad”**
**DSM is “ok” when all its required “slaves” are “ok”**
- The DD is not required
**Click on the DSM status dump link to see more details**
- Compare the lists of required and connected slaves
- Absent/misbehaving slaves can be (stopped and) restarted
**First stop all ongoing runs**

## DLSAM Status Snapshot

**Shows status of buffer disks and files to be transferred**
**“****timeout” ******** DL should move to another disk**
**Symptoms of SAM or Enstore problems:**
**Files staying “Queued” or “Submitted” > 1 hour**
- DLSAM may recover automatically
- Stopping and restarting DLSAM rarely helps
**The same applies to DLCAT**
**Files in “Alarm” state**
- Expert intervention needed

## How to reach the experts

**The ODS D0 Online Support group has a primary:**
- Send e-mail to helpdesk@fnal.gov
- Call the HelpDesk at x2345, or the Computing Services Operators at x2746 after hours, when the problem is _urgent_
**In all cases mention the application that has problems (e.g. “dsm”) and its host (“d0olc” will do)**
- The HelpDesk/Operator needs that information to figure out whom to contact/page
**Questions and comments about the DAQ Back End can be sent to:**
**d0online-support@fnal.gov**

## Summary

**The D0 DAQ back end is an integrated system comprising:**
**CR, DD, DL, DLCAT, DLSAM, DM, DSM, SDAQ**
- Their interactions/dependencies are not trivial
- Contact the experts for configuration changes
**DAQ shifters can stop and (re)start the applications**
**Following guidelines e.g. found in this tutorial**
**Please inform the experts of any problems**
**The status web page is handy for monitoring and trouble shooting**