# Presentation: 553936

## The PHENIX Event Builder

- David Winter
- Columbia University
- for the PHENIX Collaboration
- DNP 2004
- Chicago, IL

## Overview

- Introduction
- Quarks & Gluons at the extreme: Heavy Ion Collisions at RHIC
- The challenge: The PHENIX experiment and its DAQ
- The Event Builder
- Software & Hardware
- System Design
- Monitoring & Performance
- Present and Future Development
- Summary

- David Winter - The PHENIX Event Builder

- 29 Oct 2004

## Torturing the Nucleus: Heavy Ion Collisions

- Entropy Density

- Temperature

- David Winter - The PHENIX Event Builder

- 29 Oct 2004

## PHENIX @ RHIC

- Two independent rings
- 3.83 km circumference
- Capable of colliding ~ any nuclear species on ~ any other species
- Center of Mass Energy:
- 500 GeV for p-p
- 200 GeV for Au-Au (per N-N collision)
- Luminosity
- Au-Au: 2 x 1026 cm-2 s-1
- p-p  : 2 x 1032 cm-2 s-1 (polarized)

- David Winter - The PHENIX Event Builder

- 29 Oct 2004

## Data Collection: The Challenge

- Blue and Yellow Beams cross at ~10MHz

- Collisions occur at ~10kHz

- hadrons

- photons

- leptons

- High rates
- Large event sizes (Run-4: >200 kb/event)
- Interest in rare physics processes
- 	=> ***Big Headache***
- How do we address these challenges?
- Level-1 triggering
- Buffering & pipelining: “deadtime-less” DAQ
- High Bandwidth (Run-4: ~400 MB/s archiving)
- Fast processing (eg. Level-2 triggering)

- David Winter - The PHENIX Event Builder

- 29 Oct 2004

## Event Builder Overview

- Data stream from granule

- SEB

- GBit
- SWITCH

- ATP

- ATP

- ATP

- ATP

- STORAGE

- SEB

- SEB

- ATP

- EBC

- Data stream from granule

- Data stream from granule

- Control provided by Run Control

- Three functionally distinct programs derived from the same basic object
- SubEvent Buffer (SEB): Collects data for a single subsystem – a “subevent”
- Event Builder Controller (EBC): Receives event notification, assigns events, flushes system
- Assembly Trigger Processor (ATP): Assembles events by requesting data from each SEB, writes assembled events to short-term storage, can also provide Level-2 trigger environment

**Partition 1**

**Partition 2**

- (x1)

- (x52)

- (x39)

- David Winter - The PHENIX Event Builder

- 29 Oct 2004

## Software & Hardware

- Software environment: Run-4 to Run-5 paradigm shift
- New platform: Windows NT/2k Linux 2.4.x (FNAL’s SL3.0.2)
- New compiler: Visual C++ 6.0 GCC 3.2.3
- Same: Iona Orbix (CORBA), Boost template library
- 105 1U Rack-mounted dual CPU x86 servers
- 1.0 GHz PIII & 2.4 GHz P4 Xeon
- Gigabit NIC (Intel PRO/1000 MT Server)
- Foundry FastIron 1500 Gigabit Switch
- 480 Gbps total switching capacity
- 15 Slots, 10 in use (includes 96 Gigabit ports)

- JSEB: custom-designed PCI card
- Interface between EvB and incoming data stream
- Dual 1 MB memory banks (allows simultaneous r/w)
- Programmable FPGA
- Latest firmware enables DMA Burst – up to 100 MB/s I/O

- David Winter - The PHENIX Event Builder

- 29 Oct 2004

## Basic Component Design

- Booted

- Configured

- Initialized

- Configuration Data from Run Control

- State Machine whose transitions are controlled by CORBA methods

- Running

- Monitor Data to client(s)

- Specific EvB components are derived from this model

- Event-driven
- Multithreaded

- UDP sockets for data TCP sockets for control

- David Winter - The PHENIX Event Builder

- 29 Oct 2004

## Performance Monitoring

- Each component keeps track of various statistics
- Data served via CORBA calls
- Java client displays stats in “real time”
- Strip charts display data as function of time
- Histograms display data as function of component

- David Winter - The PHENIX Event Builder

- 29 Oct 2004

## Run Control

- David Winter - The PHENIX Event Builder

- 29 Oct 2004

## Where does the future lie?

- How do we break the 2.5 kHz boundary?
- _The most important improvement we can make: Port to Linux_
- Win32 ***was*** the right platform when using ATM
- ATM (completely) replaced by Gigabit in Run-4
- At the limit of what Win32 can provide us
- Growing pains while porting
- Thread-safety: Replacing Interlocked operations
- Who said writing atomic operations in assembly isn’t fun?
- Replacing Overlapped socket I/O with synchronous I/O
- Linux and AIO? Maybe in our lifetime...
- Event synchronization: Events vs. Condition variables
- Timeout mechanisms (eg. Dropped packets/events)

- David Winter - The PHENIX Event Builder

- 29 Oct 2004

## The Impact of a Linux Port

**Win32 Asynch Socket**

**Win32 Synch Socket**

**Linux Synchronous Socket**

- Linux beats Win32 hands-down in simple socket tests

**Win32 Completion Socket**

- A picture is worth a thousand words

- Write Bandwidth (Bytes/s)

- David Winter - The PHENIX Event Builder

- 29 Oct 2004

## The Payoff

- Tests with multiple granule partitions have been performed
- Fake Data Mode with 1 & 2 granules
- 26 kHz (0.240 kB/event) to 10 kHz (~150 kB/event)
- Clock Triggers with 1, 2, & 4 granule partitions
- 4.5 - 5 kHz (little to no dependence on event size)

- David Winter - The PHENIX Event Builder

- 29 Oct 2004

## Summary

- Ideal laboratory for the study of hot, dense quark matter: Heavy Ion Collisions at RHIC.
- The PHENIX experiment is designed to make high statistics measurements of a variety of physics processes, esp. rare signatures
- The PHENIX Event Builder lies at the heart of a parallel pipelined DAQ, enabling high rates of archiving.
- Three multithreaded programs originally implemented on Win32
- Win32 EvB has done a respectable job so far, but we need more
- Linux is the future of the PHENIX EvB
- Synchronous I/O superior to even Win32’s overlapped I/O
- OS overheads much lower (in general)
- Various issues when porting from Win32 to Linux
- I/O, timers, threading
- Bottom line: Run-5 will have a Linux Event Builder that early tests show will improve performance by as much as a factor of 10.  The goal of archiving up to 1 GB/s at 5 kHz is well within reach.

- David Winter - The PHENIX Event Builder

- 29 Oct 2004

## Backup Slides

- David Winter - The PHENIX Event Builder

- 29 Oct 2004

## The PHENIX DAQ

- A **granule** is a detector subsystem, including readout electronics

- A **partition** is one or more granules that receives the same triggers & busies

- David Winter - The PHENIX Event Builder

- 29 Oct 2004

## JSEB Interface Card

- PLX 9080 PCI controller

- Input for JSEB cable from DAQ

- PCI 2.1 interface:
- 32-bit 33 MHz with
- 3.3v or 5v signaling

- 2 banks of 1 MB static RAM

- Altera FLEX10k FPGA

- JTAG port for
- programming FPGA

- Interfaces the DAQ to the SEB
- Data transmitted via 50-pair 32-bit parallel cable
- (Pseudo) Driver provided by Jungo
- Latest firmware provides DMA burst mode

- David Winter - The PHENIX Event Builder

- 29 Oct 2004

## CORBA

- _C_ommon _O_bject _R_equest _B_roker _A_rchitecture
- The networking protocol by which the run control software components talk to each other.
- Based on a client/server architecture through a heterogeneous computing environment.
- VxWorks, Linux, Windows NT/2000
- Servers: implement “CORBA Objects” that execute the functionality described in the member functions of the object
- Clients: invoke local CORBA Object’s methods. This causes the server to execute the code of its corresponding member function

- David Winter - The PHENIX Event Builder

- 29 Oct 2004

## Characterizing PCI bus interactions

**JSEB contention read test**
**Without network writing (top curves)**
**With network writing (bottom curves)**

- David Winter - The PHENIX Event Builder

- 29 Oct 2004