# Presentation: 553930

## The PHENIX Event Builder

- David Winter
- Columbia University
- for the PHENIX Collaboration
- CHEP 2004
- Interlaken, Switzerland

## Overview

- Introduction
- Heavy Ion Collisions
- The Relativistic Heavy-Ion Collider
- The PHENIX experiment and its DAQ
- The Event Builder
- Hardware
- Software
- Monitoring & Performance
- Present and Future Development
- Summary

- David Winter - The PHENIX Event Builder

- 28 Sept 2004

## Begin at the Beginning

- Big Bang:
- Formation of the universe approximately 1010 years ago
- Condensation of “normal” matter (protons and neutrons) 10-4 s later
- Water: steam  Water  Ice
- Hadronic matter: free quarks and gluons  protons, neutrons, etc.
- $64,000 Question: Can we recreate the hot and dense matter that existed at the beginning of the universe?

- David Winter - The PHENIX Event Builder

- 28 Sept 2004

## Torturing the Nucleus: Heavy Ion Collisions

- “Cartoon” of what we imagine to be phase diagram of hadronic matter
- (Temp vs. baryon density)

- Lattice QCD calculations have long indicated existence of phase transition

- David Winter - The PHENIX Event Builder

- 28 Sept 2004

## Relativistic Heavy Ion Collider

- 3.83 km circumference
- Two independent rings
- Capable of colliding ~any nuclear species on ~any other species
- Energy:
- 500 GeV for p-p
- 200 GeV for Au-Au(per N-N collision)
- Luminosity
- Au-Au: 2 x 1026 cm-2 s-1
- p-p  : 2 x 1032 cm-2 s-1 (polarized)

- As if that weren’t enough, it’s visible from space!

- David Winter - The PHENIX Event Builder

- 28 Sept 2004

## PHENIX

- The Pioneering High-Energy Nuclear Interaction Experiment

- David Winter - The PHENIX Event Builder

- 28 Sept 2004

## Data Collection: The Challenge

- High rates
- (Au-Au) Collisions  occur at about 10kHz, while the beam crossing rate occurs at 9.6MHz.
- Interest in rare physics processes
- J/ ( and e channels), high pt Spectra, others
- Large event sizes (Run-4: >200 kb/event)
- How do we address these problems?
- Triggering
- Most crossings don’t interact: need to select those that do
- Selection based on “interesting” physics: Event has muon tracks, high energy or momentum charged or neutral particles.
- Buffering & pipelining: Need a “deadtime-less” DAQ
- High Bandwidth (Run-4: up to 400 MB/s archiving)
- Fast processing (Level-2 triggering, for example)
- Granules and Partitions
- A Granule is a subsystem unit
- A Partition consists of one or more granules.  All granules in the partition receive the same triggers
- Multiple partitions can run in parallel, independent of each other

- David Winter - The PHENIX Event Builder

- 28 Sept 2004

## The PHENIX DAQ

- FEM

- G
- T
- M

- LL1

- GL1

- DCM

- DCM

- DCM

- SEB

- GBit
- SWITCH

- ATP

- ATP

- ATP

- ATP

- EVENT
- TEMP
- STORAGE

- FEM

- SEB

- SEB

- ATP

***Event Builder***

- EBC

**Some systems are used to make triggers**

**Some aren’t**

- David Winter - The PHENIX Event Builder

- 28 Sept 2004

## Slide 9

- David Winter - The PHENIX Event Builder

- 28 Sept 2004

## The Event Builder at-a-glance

- Run Control

- Front-End Electronics

- Data Collection Modules

- Event Builder Controller

- Sub-Event Buffer

- Assembly Trigger Processor

- Event Storage

- David Winter - The PHENIX Event Builder

- 28 Sept 2004

## Event Builder Overview

- Multithreaded & Distributed
- Three functionally distinct programs derived from the same basic object
- SubEvent Buffer (SEB)
- One per granule
- Physically connected to DCMs
- Collects data for a single subsystem – a “subevent”
- Event Builder Controller (EBC)
- Receives event notification from GL1 SEB
- Assigns events to ATPs
- Flushes system when event is archived
- Assembly Trigger Processor (ATP)
- Assigned task of assembling events
- Requests subevent data from each SEBs
- Writes assembled events to short-term storage
- Notifies EBC of event completion

- David Winter - The PHENIX Event Builder

- 28 Sept 2004

## CPUs

- 105 1U rack-mounted dual CPU servers
- 28 Dell PowerEdge 1000
- 1.0 GHz Dual PIII
- 256 MB ECC RAM
- 8 GB Seagate U160 SCSI HDD
- 77 Microway machines
- 2.4 GHz Dual P4 Xeon
- 1 GB ECC RAM
- 37 GB Seagate ATA100 HDD
- 45 Tyan MB with 400 MHz FSB
- 32 Supermicro MB with 533 MHz FSB

- David Winter - The PHENIX Event Builder

- 28 Sept 2004

## Network Interfaces

- Dell PowerEdge servers
- On-board dual Intel Fast Ethernet (10/100 Mbps) controllers
- 17 retrofitted with Gigabit adapter add-on cards
- 16 Intel PRO/1000 MT Server Adapters
- 1 Syskonnect SK-98xx Gigabit Adapter
- Tyan MB based Microways
- On-board combination Intel Fast Ethernet and Gigabit network controllers
- Supermicro MB based Microways
- On-board dual Intel PRO/1000 MT Server Adapter controllers

- David Winter - The PHENIX Event Builder

- 28 Sept 2004

## Gigabit Switch

- Foundry FastIron 1500 Gigabit Switch
- Total switching capacity = 480 Gbps
- Jumbo Frame capable (used 9014 MTU in Run-4, Run-5 configuration under study)

- 15 Slots (10 in use)
- 2 control modules - each includes 24 100 Mbps ports (module spans two slots)
- 6 16-port Gigabit modules (95 ports in use)
- 39 SEBs
- 52 ATPs
- 1 EBC, phnxbox2, phnxbox3, corba

- David Winter - The PHENIX Event Builder

- 28 Sept 2004

## JSEB Interface Card

- PLX 9080 PCI controller

- Input for JSEB cable from Partitioner Module

- PCI 2.1 interface:
- 32-bit 33 MHz with
- 3.3v or 5v signaling

- 2 banks of 1 MB static RAM

- Altera FLEX10k
- FPGA

- JTAG port for
- programming FPGA

- David Winter - The PHENIX Event Builder

- 28 Sept 2004

## JSEB Cont’d

- Interfaces the Partitioner Module (PM) to the SEB
- Data from DCMs transmitted via 50-pair 32-bit parallel cable
- Firmware contained in FPGA
- Versions 0x11 and 0x13 used in Run 4
- Version 0x27 written for Run 5
- (Pseudo) Driver provided by Jungo
- DMA burst mode not available in Run 4
- Version 0x27 fixes this problem

- David Winter - The PHENIX Event Builder

- 28 Sept 2004

## Software

- _Run 4_
- Windows NT/2000
- Visual C++ 6.0
- Iona Orbix ASP 6.0
- Java 1.[34].x
- Run Control
- Monitoring
- Boost 1.30.x
- Threads, Mutexes, etc.
- Code developed on dedicated machine, distributed to cluster via FTP

- _Run 5_
- Linux 2.4.21 (SL 3.0.2)
- GCC/G++ 3.2.3
- Iona Orbix ASP 6.1
- Java 1.[34].x
- Boost 1.30.x
- Code developed on any machine with proper versions of CORBA and GCC
- Common file systems NFS mounted, use make install

- David Winter - The PHENIX Event Builder

- 28 Sept 2004

## CORBA

- _C_ommon _O_bject _R_equest _B_roker _A_rchitecture
- The networking protocol by which the run control software components talk to each other.
- Based on a client/server architecture through a heterogeneous computing environment.
- VxWorks, Linux, Windows NT/2000
- Servers: implement “CORBA Objects” that execute the functionality described in the member functions of the object
- Clients: invoke local CORBA Object’s methods. This causes the server to execute the code of its corresponding member function

- David Winter - The PHENIX Event Builder

- 28 Sept 2004

## Basic Component Design

- Booted

- Configured

- Initialized

- Configuration Data from Run Control

- State Machine whose transitions are controlled by CORBA methods

- Running

- Monitor Data to client(s)

- Specific EvB components are derived from this model

- Event-driven
- Multithreaded

- David Winter - The PHENIX Event Builder

- 28 Sept 2004

## Level-2

- Logical place to run Level-2: ATP (contains whole event)
**Primitives:** perform calculations, record self, available to all algorithms
**Algorithms:** high level decisions based on primitive calculations
- Primitives/Algorithms saved in ATP data stream
- Database primitives can read from db or ASCII files
- Can be run during event building or as later pass over data stream

- David Winter - The PHENIX Event Builder

- 28 Sept 2004

## Performance Monitoring

- Each component keeps track of various statistics
- Data served via CORBA calls
- Java client displays stats in “real time”
- Strip charts display data as function of time
- Histograms display data as function of component

- David Winter - The PHENIX Event Builder

- 28 Sept 2004

## Run Control

- David Winter - The PHENIX Event Builder

- 28 Sept 2004

## Where does the future lie?

- _The most important improvement we can make: Port to Linux system_
- Linux is a “sane” platform (saner than Win32)
- Win32 WAS the right platform when using ATM (ATM-free in ’03!)
- Each piece performed well, but when put together suffered degraded performance
- At the limit of what Win32 can provide us (Peak stable performance: ~2.2kHz, DAQ is spec’d to at least 8kHz)
- Exploit the vast body of Linux knowledge that already exists in PHENIX
- Common development platform eases development and maintenance
- Re-evaluate architecture and cut away the “fat”
- Integrate the improvements made in Run-4 but never used
- ATPs requesting multiple events at a time
- SEB Load balancing algorithm

- David Winter - The PHENIX Event Builder

- 28 Sept 2004

## Transition to Linux

- Architecture Issues
- Fundamentally the architecture is sound, the primary issue being excessive overhead in the OS. Therefore, we are taking a direct-porting approach, wherever possible
- Use boost template library to enhance portability and performance
- Use of ACE is under study, though not planned for the start of Run-5
- Chosen platform is Fermilab’s Scientific Linux (currently using version 3.0.1, moving to 3.0.2)
- Based on Redhat’s Enterprise Linux
- GCC 3.2.3
- Linux’s new Native POSIX Thread Library (NPTL)
- Linux and asynchronous I/O
- Win32 implementation relied heavily on overlapped I/O
- Asynch (socket) I/O basically not usable in Linux
- (Pseudo-)Real-time event timers not available

- David Winter - The PHENIX Event Builder

- 28 Sept 2004

## The Impact of a Linux Port

**Win32 Asynch Socket**

**Win32 Synch Socket**

**Linux Synchronous Socket**

- Linux beats Win32 hands-down in simple socket tests

**JSEB read test**
**Without network writing (top curves)**
**With network writing (bottom curves)**

**Win32 Completion Socket**

- A picture is worth a thousand words

- David Winter - The PHENIX Event Builder

- 28 Sept 2004

## Implementing Under Linux

- All I/O is now synchronous
- At first this was a major concern, *BUT*
- It greatly simplifies the architecture
- Linux synchronous performance much better than anticipated
- Timeouts (Event Timers)
- In Win32 version, all sockets were UDP
- Unreliability of UDP requires some form of timeout
- Level-1 messages from GL1 to EBC now use TCP
- Communication between ATPs and EBC now uses TCP
- TCP now greatly simplifies code
- Data messages are still UDP, therefore a preliminary timeout mechanism has been developed

- David Winter - The PHENIX Event Builder

- 28 Sept 2004

## The Payoff

- To date, all components have been ported
- Approximately 35% of the servers converted
- Distcc is a god-send: 90k lines of code in <3 min
- The timeout mechanism is in the works
- Preliminary tests with 1 & 2 granule partitions have been performed

- David Winter - The PHENIX Event Builder

- 28 Sept 2004

## Summary

- David Winter - The PHENIX Event Builder

- 28 Sept 2004