# Presentation: 383287

## The STAR Databases: From Objectivity to ROOT+MySQL

- Torre Wenaus
- BNL
- ATLAS Computing Week
- CERN
- August 31, 1999

## Content

- ATLAS meeting 8/99

- Torre Wenaus, BNL

## My Crude View of the STAR 8/99 Picture

- ATLAS meeting 8/99

- Torre Wenaus, BNL

## STAR at RHIC

- RHIC: Relativistic Heavy Ion Collider at Brookhaven National Laboratory
- Colliding Au - Au nuclei at 200GeV/nucleon
- Collision species from p to Au and energies 30-200 AGeV
- Principal objective: Discovery and characterization of the Quark Gluon Plasma
- Additional spin physics program in polarized p - p collisions
- Engineering run 6-8/99; first year physics run 1/00
- Nuclear physics experiments at a large scale characteristic of HEP
- Two large experiments, STAR and PHENIX, representing ~80% of the experimental program; >400 collaborators each
- Two smaller, PHOBOS and BRAHMS
- STAR experiment
- Heart of experiment is a Time Projection Chamber (TPC) drift chamber (operational) together with Si tracker (year 2) and electromagnetic calorimeter (staged over years 1-3)
- Hadrons, jets, electrons and photons over large solid angle

- ATLAS meeting 8/99

- Torre Wenaus, BNL

## The STAR Computing Task

- STAR trigger system reduces 10MHz collision rate to ~1Hz recorded to tape
- Data recording rate of 20MB/sec; ~12MB raw data size per event
- ~4000+ tracks/event recorded in tracking detectors (factor of 2 uncertainty in physics generators)
- High statistics per event permit event by event measurement and correlation of QGP signals such as strangeness enhancement, J/psi attenuation, high Pt parton energy loss modifications in jets, global thermodynamic variables (eg. Pt slope correlated with temperature)
- 17M Au-Au events (equivalent) recorded in nominal year
- Relatively few but highly complex events requiring large processing power
- Wide range of physics studies: ~100 concurrent analyses in ~7 physics working groups

- ATLAS meeting 8/99

- Torre Wenaus, BNL

## STAR Computing Requirements

- Nominal year processing and data volume requirements:
- Raw data volume: 200TB
- Reconstruction: 2800 Si95 total CPU, 30TB DST data
- 10x event size reduction from raw to reco
- 1.5 reconstruction passes/event assumed
- Analysis: 4000 Si95 total analysis CPU, 15TB micro-DST data
- 1-1000 Si95-sec/event per MB of DST depending on analysis
- Wide range, from CPU-limited to I/O limited
- ~100 active analyses, 5 passes per analysis
- micro-DST volumes from .1 to several TB
- Simulation: 3300 Si95 total including reconstruction, 24TB
**Total nominal year data volume: 270TB**
**Total nominal year CPU: 10,000 Si95**

- ATLAS meeting 8/99

- Torre Wenaus, BNL

## STAR Computing Facilities

- Dedicated RHIC computing center at BNL, the RHIC Computing Facility
- Data archiving and processing for reconstruction and analysis (not simulation; done offsite)
- Three production components: Reconstruction and analysis services (CRS, CAS) and managed data store (MDS)
- 10,000 (CRS) + 7,500 (CAS) Si95 CPU, balanced between CPU-intensive farm processing (reconstruction, some analysis) and I/O intensive SMP processing (I/O intensive analysis)
- ~50TB disk, 270TB robotic tape, 200MB/s, managed by HPSS
- Current scale: ~2500 Si95 CPU, 3TB disk for STAR
- Limited resources require the most cost-effective computing possible
- Commodity Intel farms (running Linux) for all but I/O intensive analysis (Sun SMPs)
- Support for (a subset of) physics analysis computing at home institutions

- ATLAS meeting 8/99

- Torre Wenaus, BNL

## STAR Software Environment

- Current software base a mix of Fortran (55%) and C++ (45%)
- from ~80%/20% (~95%/5% in non-infrastructure code) in 9/98
- New development, and all post-reco analysis, in C++
- Framework built over ROOT adopted 11/98
- Supports legacy Fortran codes, table (IDL) based data structures developed in previous STaF framework without change
- Deployed in offline production and analysis in our ‘Mock Data Challenge 2’, 2-3/99
- Post-reco analysis: C++/OO data model (‘StEvent’), all C++
- Requirement: StEvent interface does not ‘express’ ROOT, and analysis codes are unconstrained by ROOT (eg. CINT) and need not (but may) use it
- Next step: migrate the OO data model upstream to reco
- Infrastructure (including databases) development team: ~7 people
- Two less than planned due to budget constraints
- Database/event store development ~ 2.5 FTE-years in last 1.5 years
- Reliance on leveraging community, commercial, open software

- ATLAS meeting 8/99

- Torre Wenaus, BNL

**Notes:**
Yuri, Victor, Valery, Pavel, Jeff, Herb, TW+Sasha

DB/event store, FTE-mo:
98: Jeff/Nathan 3, TW 2, Akio/Zimm 1, Valery/Victor 6: 12
99: Jeff 5, TW 2, V/V 6, Yuri 2, Sasha/Pavel 3: 18

## Data Management in the RHIC Experiments

- RHIC Event Store Task Force fall ‘97 addressed data management alternatives
- Requirements formulated by the four experiments
- Objectivity and ROOT were the ‘contenders’ put forward
- STAR and PHENIX selected Objectivity as the basis for data management
- Concluded that only Objectivity met the requirements of the STAR, PHENIX event stores
- ROOT selected by the smaller experiments and seen by all as analysis tool with great potential
- Issue for the two larger experiments:
- Where to draw a dividing line between Objectivity and ROOT in the data model and data processing

- ATLAS meeting 8/99

- Torre Wenaus, BNL

## Data Management Requirements

- Support C++ with a well documented API.
- Must scale to handle data set sizes at RHIC.
- Ability to operate in range of the aggregate I/O activity required.
- Currently support or plan to support an interface with HPSS
- Provide adequate levels of integrity and availability.
- Ability to recover from permanently lost data.
- Object versioning and schema evolution.
- Maintainable and upgradeable. Long-term availability.
- Support for read/write access control at the level of individuals and groups.
- Administration tools to manage the database.
- Backup and recovery of subsets of the data.
- Support for copying, moving data; data distribution over WAN
- Control over data locality

- ATLAS meeting 8/99

- Torre Wenaus, BNL

## Objectivity in STAR

- Objectivity selected for all database needs
- Event store
- Conditions and calibrations
- Online database and detector configurations
- Decision to use Objectivity would have been inconceivable were it not for BaBar leading the way with an intensive development effort to deploy Objectivity on the same timescale as RHIC and at a similar scale
- From Nov 1 STAR will have *one* dedicated FTE on database development (event store, conditions and configuration)
- STAR has imported the full BaBar software distribution and deployed it as an ‘external library’ providing the foundation for STAR event store and conditions database
- BaBar’s software release tool and site customization features used to adapt their code without direct modification to the STAR environment

- ATLAS meeting 8/99

- Torre Wenaus, BNL

## Objectivity-based Event Store

- BaBar event store software adapted to support STAR event store down to the level of event components
- Event collection dictionary with collection organization in Unix directory tree style
- System, group, user level authorization controls
- Federated database management: developer-level federations, schema evolution
- Event collection implementation
- Event components are purely STAR implementation and map directly to IDL-defined data structures
- STAR raw data will **not** be stored in Objectivity
- In-house format will be used to insulate STAR against long-term viability of Objectivity-based storage
- Event store will provide file pointers to the raw data, and an Objectivity-wrapped version of the in-house format for small-scale convenience use

- ATLAS meeting 8/99

- Torre Wenaus, BNL

## Conditions and Configuration Databases

- Prototype conditions database supporting time-dependent calibrations implemented based on BaBar’s conditions database
- Like the event store, builds on existing IDL-based data model
- Prototype configuration database supporting STAR’s online system developed standalone, for later integration into BaBar framework

- ATLAS meeting 8/99

- Torre Wenaus, BNL

## Data Access and Data Mining

- Crucial issue in STAR is rapid, efficient data availability for physics analysis
- Full DST and post-DST volume far in excess of disk capacity
- A DOE Grand Challenge initiative addresses managed access to and ‘mining’ of the HPSS-resident Objectivity-based DST data
- Next talk
- Focused presently on RHIC and with heavy STAR involvement
- Very practically addressed to immediate RHIC needs
- Already a production-capable tool being integrated into current STAR data management regulating access to HPSS-resident DSTs

- ATLAS meeting 8/99

- Torre Wenaus, BNL

## Mock Data Challenge (MDC)

- RHIC-wide exercise in stress-testing RCF and experiment hardware and software in full production environment from (simulated) raw data through physics analysis
- MDC1 just concluded, with objective of processing (at least) 100k events per experiment on 10%-nominal RCF facilities
- A success for RHIC and STAR
- In STAR,
- 217k Au-Au events (1.7TB) simulated on the T3Es with network transport to RCF HPSS
- 168k events reconstructed (600GB of DSTs generated)
- 50GB Objectivity DST database built (disk space limited) and used at a small scale with STAR analysis codes
- Full 600GB HPSS-resident DST database will be built in ~2 weeks using Grand Challenge software to manage access from STAR analysis codes

- ATLAS meeting 8/99

- Torre Wenaus, BNL

## Event Store Implementation and Status

- STAF-standard XDF format (based on XDR) for IDL-defined data is today the production standard and will remain an Objectivity fall-back
- IDL-defined persistent data model ensures XDF compatibility
- IDL-standard restricts how we use Objectivity: don’t even pretend it’s C++
- But, STAR (taking a lesson from BaBar) is completely decoupling persistent and transient data models, bearing the cost of translation
- Design of C++ transient model is decoupled from persistent implemenation
- Transient representation is built in the translator
- No direct exposure of application code to Objectivity
- Objectivity/BaBar based event store deployed in current Mock Data Challenge and now a production fixture of STAR data processing
- Linux support is a crucial issue; currently limited to Sun/Solaris
- With Objectivity port available, BaBar/STAR software porting will proceed by end of year

- ATLAS meeting 8/99

- Torre Wenaus, BNL

## A Hybrid Event Store for STAR?

- Requirements driving STAR to Objectivity are grounded in the very large scale data management problem
- We’re comfortable, so far, with Objectivity in the global data management role
- Requirements and priorities for select components of the data model differ and can drive different choices
- Non-Objectivity raw data already addressed
- Post-DST physics analysis data (micro-DSTs) is such an area
- High value in close coupling to data analysis tool (ROOT, in all probability): direct access to data model during analysis, presentation
- Great physicist-level flexibility essential in defining object model (schema), and Objectivity (currently) presents severe problems in secure, flexible schema management
- Premium on storage optimization for compact, rapid-access data (compression, N-tuple like storage)

- ATLAS meeting 8/99

- Torre Wenaus, BNL

## Hybrid ROOT/Objectivity Event Store

- These considerations motivate the use of ROOT for micro-DST level persistent data
- Particularly given the immaturity of the analysis tools being developed for LHC to work in conjunction with Objectivity-based data
- STAR is currently investigating ROOT-based micro-DSTs integrated into the Objectivity event store
- ROOT-based micro-DSTs associated with an event collection incorporated via ROOT file references at the collection level

- ATLAS meeting 8/99

- Torre Wenaus, BNL

## Event Store and Data Management

- Success of ROOT-based event data storage from MDC2 on relegated Objectivity to metadata management role, if any
- ROOT provides storage for the data itself
- We can use a simpler, safer tool in metadata role without compromising our data model, and avoid complexities and risks of Objectivity
- MySQL adopted to try this (relational DB, open software, widely used, very fast, but not a full-featured heavyweight like ORACLE)
- Wonderful experience so far. Excellent tools, very robust, extremely fast
- Scalability must be tested but multiple servers can be used as needed to address scalability needs
- Not taxing the tool because metadata, not large volume data, is stored
- Metadata management (multi-component file catalog) implemented for simulation data and extended to real data
- Basis of tools managing data locality, retrieval in development
- Event-level tags implemented for real data to test scalability and event tag function (but probably not full TagDB, better suited to ROOT files)
- Will integrate with Grand Challenge in September

- ATLAS meeting 8/99

- Torre Wenaus, BNL

## Persistent StEvent

- StEvent OO/C++ DST data model and usage tools, sample analysis code released before MDC2; now the standard DST level analysis tool
- Persistent version of StEvent evolved since Jan and deployed in May; migration in the last month
- Provides
- Capability to store StEvent as ROOT file(s)
- Option to base micro-DSTs on StEvent; post-DST analysis using familiar, standard data model
- We need to provide an example ASAP
- ROOT-aware data model
- StEvent browser GUI, direct StEvent visualization
- Updating of DST content and organization underway, which will be reflected in an updated StEvent

- ATLAS meeting 8/99

- Torre Wenaus, BNL

## Conclusions

- The circumstances of STAR
- Startup in 1999
- Slow start in addressing event store implementation, C++ migration
- Large base of legacy software
- Extremely limited manpower and computing resources
- drive us to extremely practical and pragmatic data management choices
- Beg, steal and borrow from the community
- Deploy community and industry standard technologies to leverage future as well as current work elsewhere
- Isolate implementation choices behind standard interfaces, to revisit and re-optimize in the future
- which leverage existing STAR strengths
- Component and standards-based software greatly eases integration of new technologies
- preserving compatibility with existing tools for selective and fall-back use
- while efficiently migrating legacy software and legacy physicists
- By today’s judgement, data management is on track to have a capable system by startup that scales to STAR’s data volume and 10-15yr lifespan.

- ATLAS meeting 8/99

- Torre Wenaus, BNL