# Presentation: 250333

## Online DAQ System: From Detector to Tape

- T. Yasuda
- Fermilab

- Overview
- Hardware
- Control System
- Primary/Secondary Data Path
- DAQ Applications
- DAQ in action
- Conclusions

## Overview

- The DØ DAQ system is divided into two components:
- Trigger system
- Level 1 hardware trigger components
- Level 2 specialized processors
- Level 3 crate readout and software trigger components
- Online or Host system
- Detector controls
- Data Logging
- Monitoring
- Control room applications

## Overview

- DAQ Architecture
- Event data rate and operational redundancy achieved by a high degree of parallelism
- Level 3
- Host
- Capability for multi-user, multi-stream operation
- With central resource configuration manager
- Network-centric Host design

## DAQ Components

**Detector**

**FCC**

***UNIX***
***Servers***

***NT Level 3***

***Linux***
***PCs***

***Control***
***Room***
***PCs***

***Controls***

***Trigger and Readout***

## Hardware Description

- 3 Compaq/Digital Alpha Servers
- d0ola: Alpha Server 4000, 1 processor, 466MHz, 500 MB memory
- d0olb: Alpha Server 4000, 2 processors, 600 MHz, 500 MB memory
- d0olc: Spec out by Aug 3. Probably Alpha Server ES40, 4 processors, 667 MHz
- Clustered / redundant
- 500 GB shared RAID disks for online apps and database (Mirrored)
- 500+ GB local ‘data buffer’ disks, fiber channel based (40M/sec)

## Hardware Description

- Linux/NT nodes
- Buying 6 nodes with dual PIII, 600 MHz, 500MB memory, 2 graphics cards
- 3 Linux nodes and 3 NT nodes exist
- Will run Vmware on Linux nodes
- Control system embedded 68Ks and PowerPCs (VxWorks)
- Network
- Cisco 6509 GB Ethernet switch for all FCH nodes
- Satellite 100 MB switches in MCH
- GB fiber to FCC
- Security
- Access control filter to online machines
- Kerberos authenticated ssh session only

## Control & Monitoring

**Detector**

**Readout Crate**

**Controls Crate**

*Trigger and Readout*

*UNIX Servers*

*Control Room PCs*

**EPICS Clients:**
**  ****Low Voltage**
**  ****High Voltage**
**  ****Rack Monitor**
**  ****1553 Devices**
**  ****SMT Monitor**
**  ****FT Monitor**
**  ****etc**

*Controls*

*Ethernet*

*1553*

**ORACLE**

**EPICS DB Generator**

*Vertical Interconnect*

## Control System

- Built upon EPICS control system
- A ‘standard’ toolkit upon which we’ve built DØ extensions
- Lots of user-community supplied tools
- ORACLE Hardware Database
- Extract EPICS db from ORACLE
- Web-based and batch interfaces
- Hardware Control
- Low Voltage, High Voltage, etc dedicated GUI applications
- Downloading
- Registers, Pedestals, etc
- Significant Event (alarm) System
- Interface to Accelerator and Cryogenics controls systems

## Status of Control System

- Calorimeter
- Preamp PS, BLS PS, ADC PS control exist
- Pulser control (in progress)
- SMT
- EIPCS records for Sequencer, Sequencer control, VRB, VRBC, Emulator (in progress) exist and used in the test stands
- Muon
- Tested communication using 1553 for PDT, MDT, SRC cards
- FPD
- Used RM support to control motors

## Status of Control System

- Luminosity
- Scalers and FE processing results communicated to Accelerator via ACNET
- Cryo
- Communicated with DMAX system
- Common Tools
- generic 1553 support
- generic VME support
- HV used in SMT, Muon, Lum
- V1 running for months, V2 work starting
- Diagnostic support for busses
- Standard operator interface (GUI)

## Hardware Database

- Describes Control aspects of electronics
- Based on ORACLE
- 2 instances of the database ( dev, user testing)
- Web based interface for entering, modifying, deleting records
- Python script for batch entries exists
- Calorimeter records in the database

## Hardware Database

## Significant Event (Alarm) System

- System to detect alarm conditions and state changes in the DAQ system
- Server with DAQ components as clients
- COOR sends alarm and run control messages
- CR, DL, DD send alarm messages
- Version 1 Display exists
- Working on version 2 Display (Summer student)
- Need to integrate EPICS alarms into the system (Fall 2000)
- on IOC EPICS alarms -> ITC client
- ITC client sends alarms to Server on host
- EPICS Alarm Handler can be used for now

## Significant Event System

**Significant Event**
** ****Server**

**Fault Watcher**

**Archiver**

**HV Control**

**Heartbeat**

**Display**

**Run Control**
**(COOR)**

**Process**

**Periodic**
**Heartbeat**

**Run**
**Suspend**

**F**

**F**

**F**

**F**

**F**

**F**

**Filter**

**SE Message**

**Filtered Message**

## Configuration & Run Control

**Detector**

***L1, L2, TCC***

***L3 Supervisor***

**L3 VRC**

**Readout Crate**

**L3 Filter**

**FCC**

*Data Cable*

*Data Cable*

**Controls Crate**

*UNIX Servers*

*NT Level 3*

*Ethernet*

*Ethernet*

*Linux PCs*

*Control Room PCs*

*Controls*

*Trigger and Readout*

**Run Control**
**Client**

**Comics**

**DSM**

**COOR**

**Collector / Router**

**Data Logger**

**Data Distributor**

**EXAMINE**

**RIP**

*1553*

*Vertical Interconnect*

## Software Description

- Configuration Management and   Run Control
- Coordination (COOR)
- User interface (TAKER)
- Download manager (COMICS)
- Primary event path
- DAQ State Manager (DSM)
- Collector / Router
- Data Logger
- Event metadata manager (SAM)
- Event data manager (enstore)
- Secondary event path
- Secondary DAQ Supervisor
- Data Merger

## Software Description

- Event monitoring
- Data Distributor
- Analysis applications (EXAMINE)
- DAQ Monitoring
- Client/Server access to DAQ flow statistics, trigger rates, etc
- Detector Monitoring
- Front End active & parasitic monitors
- Calibration
- Client/Server interface to database
- Infrastructure
- Databases (ORACLE)
- Task-to-task communication (ITC)

## Secondary DAQ Data Flow

*Controls/Readout*
*         **Crates*

***1553***** *****Bus***

*VME* *Bus*

**FCC**

*Linux PCs*
*UNIX Server*

*UNIX Server*

*Control Room PCs*

*ITC*

**Detector**

*CA Link*

## Secondary DAQ System

- Alternative data path
- Mainly used for monitoring and calibration
- Takes advantage of powerful front-end processors
- Uses the same data path as the primary path after Data Merger

## DAQ Monitor

- Monitors the status of DAQ subsystems (L1/2, CR, DL, DD)
- Collects statistics information from the subsystems
- C++ itc Server with python Display clients

## DAQ Monitor

## Event Monitoring:EXAMINE

- Samples and reconstructs event based on Stream IDs and trigger IDs
- Clients of Distributor
- Network and file event transfer modes work
- Calorimeter EXAMINE
- used for preamp testing
- CFT EXAMINE
- Getting ready for raw data unpacking
- MC packed data?
- SMT EXAMINE
- used for SiDet data
- Muon EXAMINE
- used for commissioning

## Event Monitoring: EXAMINE

**Detector**

***L1, L2, TCC***

***L3 Supervisor***

**L3 VRC**

**Readout Crate**

**L3 Filter**

**Collector / Router**

**Data Logger**

*Data Cable*

*Data Cable*

*UNIX Servers*

*NT Level 3*

*Ethernet*

*Linux PCs*

*Control Room PCs*

**Data Distributor**

**EXAMINE**

**Express Line**

**ROOT Client**

**ROOT Client**

*Trigger and Readout*

## Event Monitoring:EXAMINE

- Need:
- L3 EXAMINE
- Vertex EXAMINE
- Preshower EXAMINE
- Planned improvements
- Histoscope -> Root after NIU workshop
- on-the-fly histogram
- e-browser
- Framework improvement
- name server for accessing only the histograms

## Online Event Display

## Online Calibration

- Perform electronics calibration of sub-detectors and insert results to ORACLE database
- COOR controlled via Taker
- Common server and database interface for all sub-detectors
- Calibration results transmitted as special event messages through DAQ paths
- Current status
- Successfully ran SMT calibration at 1% test stand and NW test stand

## Calibration Manager

- Taker

- COOR

- COMICS

- Crates

- Calibration
- Manager

- Calibration
- Database

- Calib. Data
- Processor

- Database
- Interface

- Configure

- Request start run

- Request
- download

- Download

- Start run

- End run

- Data

- Pedestals
- Gains

- Pedestals
- Gains

- Comparison
- Results

- Database
- Access

- End calib.

- Start run,
- End run

- Validator

- Calib Manager
- Display

## Online DAQ in Action

- Electronics/DAQ Commissioning
- 2 VRB crates with 10 cards each, synchronized with SCL from TFW to L3
- L1 muon crate
- 1 Muon Scint crate with 2 MRCs
- 1 Muon PDT crate with 1 MRC
- 1 Calorimeter crate
- Combinations of 2 systems done
- but not with MCH2+MCH3
- 2 simultaneous runs done
- 3 simultaneous runs require one more L3 node or script runner

## Online DAQ in Action

- SMT Test Stands
- 1% and NW test stands
- 1 HDI, 1 Sequencer, 1 VRB, 1 VRBC, 1 VBD
- Download done with COMICS and database
- 10% test stand
- 3 HDIs, a few Interface Boards, a few Sequencers, a few VRBs, 1 VRBC, 1 1553 controler, 1 VBD + L3
- Download done by spread sheet for now
- Databases (Electronics) exist for all three stands
- Calibration run performed at 1% and NW test stands

## Online DAQ in Action

- Commissioning Run
- Two detectors installed for the upcoming Commissioning Run
- Run I Luminosity scintillation counters
- Forward Proton Detector
- Both detectors will be read out using the Run II Online system.
- Data will be transferred to and from the Accelerator Controls System via the EPICS/ACNET Gateway.

## Conclusions

- All of the DAQ components exist and function.
- Improvements are implemented daily following user suggestions.
- We have been intimately involved in daily commissioning activities for the past few months.
- Bring in your sub-detectors!!