# Presentation: 080353

## Software and Computing Readiness

- Torre Wenaus
- BNL
- STAR Collaboration Meeting Plenary
- BNL
- August 2, 1999

## Outline

- Manpower  and organization
- Progress and status
- Real data processing
- Infrastructure issues
- Databases
- QA
- Documentation and tutorials
- Computing facilities
- Post engineering run priorities
- Other talks:
- Online readiness, Claude Pruneau
- RCF readiness, Bruce Gibbard
- Offline Production Readiness, Yuri Fisyak
- Simulations Readiness, Pavel Nevski
- Analysis Software Readiness, Thomas Ullrich
- Offline Computing Capabilities at LBNL/NERSC, Doug Olson

- STAR Collaboration Meeting 8/99

- Torre Wenaus, BNL

## Manpower and Organization

- New Software Executive Group (Jacobs, Pruneau, Ullrich, Wenaus) to address and reach consensus on software issues, particularly directly related to physics analysis
- Weekly meetings, web site linked from comp home page
- Effective in addressing and/or diffusing contentious issues and improving communication and coupling between computing and user needs
- Very important development: Big new influx of postdocs, students into computing and related activities! Bravo and thank you!
- Inevitably incomplete list on next page
- Increased participation in QA, online, production, calib/DB, detector software, ... is really helping
- Keep it up! Activities such as QA, production, non-TPC DB/calib, online are still or soon will be ramping
- Many carry major responsibilities in the software, and we look forward to more
- Still missing online/general computing systems support person
- Opening of position is working its way through the BNL system

- STAR Collaboration Meeting 8/99

- Torre Wenaus, BNL

## Some of our Youthful Manpower

- Dave Alvarez, Wayne, SVT
- Lee Barnby, Kent, QA and production
- Jerome Baudot, Strasbourg, SSD
- Selemon Bekele, OSU, SVT
- Marguerite Belt Tonjes, Michigan, EMC
- Helen Caines, Ohio State, SVT
- Manuel Calderon, Yale, StMcEvent
- Gary Cheung, UT, QA
- Laurent Conin, Nantes, database
- Wensheng Deng, Kent, production
- Jamie Dunlop, Yale, RICH
- Patricia Fachini, Sao Paolo/Wayne, SVT
- Dominik Flierl, Frankfurt, L3 DST
- Marcelo Gameiro, Sao Paolo, SVT
- Jon Gangs, Yale, online
- Dave Hardtke, LBNL, Calibrations, DB
- Mike Heffner, Davis, FTPC
- Eric Hjort, Purdue, TPC

- Amy Hummel, Creighton, TPC, production
- Holm Hummler, MPG, FTPC
- Matt Horsley, Yale, RICH
- Jennifer Klay, Davis, PID
- Matt Lamont, Birmingham, QA
- Curtis Lansdell, UT, QA
- Brian Lasiuk, Yale, TPC, RICH
- Frank Laue, OSU, online
- Lilian Martin, Subatch, SSD
- Marcelo Munhoz, Sao Paolo/Wayne, online
- Aya Ishihara, UT, QA
- Adam Kisiel, Warsaw, online, Linux
- Frank Laue, OSU, calibration
- Hui Long, UCLA, TPC
- Vladimir Morozov, LBNL, simulation
- Alex Nevski, RICH
- Sergei Panitkin, Kent, online
- Caroline Peter, Geneva, RICH

- Li Qun, LBNL, TPC
- Jeff Reid, UW, QA
- Fabrice Retiere, calibrations
- Christelle Roy, Subatech, SSD
- Dan Russ, CMU, trigger, production
- Raimond Snellings, LBNL, TPC, QA
- Jun Takahashi, Sao Paolo, SVT
- Aihong Tang, Kent
- Greg Thompson, Wayne, SVT
- Fuquian Wang, LBNL, calibrations
- Robert Willson, OSU, SVT
- Richard Witt, Kent
- Gene Van Buren, BNL, documentation, tools, QA
- Eugene Yamamoto, UCLA, calibrations, cosmics
- David Zimmerman, LBNL, Grand Challenge

- A **partial** list of young students and postdocs now active in aspects of software (apologies for omissions and errors, and corrections are welcome!)...

- STAR Collaboration Meeting 8/99

- Torre Wenaus, BNL

## Status Relative to Nov Workshop Decisions

- ROOT-based production software by MDC2 
- Legacy codes will continue to be supported 
- C++/OO data model for DST, design unconstrained by ROOT 
- Analysis codes in C++; unconstrained by ROOT (templates etc.) 
- ROOT-based analysis framework and tools 
- Choice between Objectivity and ROOT for data store from MDC2 
- Objectivity for event store metadata, “all being well” 
- All was not well! Concerns over manpower, viability, outside experience (BaBar) ... and most importantly success of MySQL and ROOT I/O provided good alternative which does not compromise data model
- Will use ROOT for micro-DSTs in year 1
- Presumed extension of data model to persistent was overambitious; it was, but our hyperindustrious core team (largely Yuri, Victor) did it anyway. Persistent StEvent as basis for micro-DSTs is now an option.

- STAR Collaboration Meeting 8/99

- Torre Wenaus, BNL

## From MDC2 to the May Workshop

- MDC2:
- ROOT based production chain and I/O, StEvent (OO/C++ data model) in place and in use
- Statistics suffered from software and hardware problems and the short MDC2 duration
- Software objectives were met and the objective of an active physics analysis and QA program was met; 5pm QA meetings instituted
- Post-MDC2: round of infrastructure modifications in light of MDC2 experience
- ‘Maker’ scheme incorporating packages in framework and interfacing to I/O revised
- Robustness of multi-branch I/O (multiple file streams) improved
- XDF based I/O maintained as stably functional alternative
- Rapid change brought a typical user response of Aaaargh!
- Development completed by the late May computing workshop

- STAR Collaboration Meeting 8/99

- Torre Wenaus, BNL

## Status Relative to Engineering Run Goals established in May

- Stabilize offline environment as seen by users
- End of rapid infrastructure development in May and subsequent debugging have improved stability, but the issue hasn’t gone away
- Shift of focus to consolidation: debugging, usability improvements, documentation, user-driven enhancements, developing and responding to QA
- Support for DAQ data in offline from raw files through analysis. Done.
- Provide transparent I/O supporting all file types. Done
- Universal interface’ to I/O, StIOMaker, developed to provide transparent, backwards compatible I/O for all formats
- Future (from Sept) extension to Grand Challenge
- Stably functional data storage
- ROOT I/O debugging proceeded into June/July; now appears stably functional. XDF output continues to provide backup.
- Deploy and migrate to persistent StEvent
- Deployed; migration to persistent StEvent in progress since the meeting. Period of coexistence to allow people to migrate on their timescale.

- STAR Collaboration Meeting 8/99

- Torre Wenaus, BNL

## Persistent StEvent

- StEvent OO/C++ DST data model and usage tools, sample analysis code released before MDC2; now the standard DST level analysis tool
- Persistent version of StEvent evolved since Jan and deployed in May; migration in the last month
- Provides
- Capability to store StEvent as ROOT file(s)
- Option to base micro-DSTs on StEvent; post-DST analysis using familiar, standard data model
- We need to provide an example ASAP
- ROOT-aware data model
- StEvent browser GUI, direct StEvent visualization
- Updating of DST content and organization underway, which will be reflected in an updated StEvent

- STAR Collaboration Meeting 8/99

- Torre Wenaus, BNL

## Status Relative to May Goals (2)

- Populated calibration/parameter database in use via stable interface
- Not there! Still a work in progress!
- Resume (in persistent form!) the productive QA/analysis operation seen in MDC2
- Big success; thanks to Trainor, Ullrich, Turner and the many participants
- Extend production DB to real data logging, multi-component events and file catalog needs of Grand Challenge
- Done, but much work still to come.

- STAR Collaboration Meeting 8/99

- Torre Wenaus, BNL

## Real Data Processing

- StDaqLib from DAQ group provides real data decoding
- StIOMaker provides real data access in offline framework
- St_tpcdaq_Maker interfaces to TPC codes
- Current status from Iwona:
- Can read and analyze zero suppressed data all the way to DST
- DST from real data (360 events) read into StEvent analysis
- Bad channel suppression implemented and tested.
- First order alignment was worked out (~1mm), the rest to come from residuals analysis
- For monitoring purposes we also read and analyze unsuppressed data
- 75% of  TPC read out.
- 10 000 cosmics with no field and  several runs with field on
- working towards regular production and QA reconstruction of real data

- STAR Collaboration Meeting 8/99

- Torre Wenaus, BNL

## Release Management and Stability

- Separate infrastructure development release .dev introduced to insulate users
- .dev migrated to dev weekly (Fri), standard test suite run over weekend
- QA evaluation by core group, librarian, QA team Monday
- dev to new migration decision taken at Tuesday computing meeting
- migration happens weekly unless QA results delay it
- Better defined criteria for dev->new migration  to be instituted soon
- Objective:
- A stable, current **new **version for broad use
- Currently too heavy a reliance on the unstable dev version
- Version usage to date: (pro artifically high; default environment)
- Level .dev    2729
- Level  dev  18503
- Level  new   7641
- Level  old      662
- Level  pro  57697

- STAR Collaboration Meeting 8/99

- Torre Wenaus, BNL

## Event Store and Data Management

- Success of ROOT-based event data storage from MDC2 on relegated Objectivity to metadata management role, if any
- ROOT provides storage for the data itself
- We can use a simpler, safer tool in metadata role without compromising our data model, and avoid complexities and risks of Objectivity
- MySQL adopted to try this (relational DB, open software, widely used, very fast, but not a full-featured heavyweight like ORACLE)
- Wonderful experience so far. Excellent tools, very robust, extremely fast
- Scalability must be tested but multiple servers can be used as needed to address scalability needs
- Not taxing the tool because metadata, not large volume data, is stored
- Metadata management (multi-component file catalog) implemented for simulation data and extended to real data
- Basis of tools managing data locality, retrieval in development
- Event-level tags implemented for real data to test scalability and event tag function (but probably not full TagDB, better suited to ROOT files)
- Will integrate with Grand Challenge in September

- STAR Collaboration Meeting 8/99

- Torre Wenaus, BNL

## ROOT Status

- ROOT is with us to stay!
- No major deficiencies, obstacles found; no post-ROOT visions contemplated
- ROOT community growing: Fermilab Run II, ALICE, MINOS
- We are leveraging community developments
- First US ROOT workshop at FNAL in March
- Broad participation, >50 from all major US labs, experiments
- ROOT team present; heeded STAR priority requests
- I/O improvements: robust multi-stream I/O and schema evolution
- Standard Template Library support
- Both emerging in subsequent ROOT releases
- FNAL participation in development, documentation
- ROOT guide and training materials just released
- The framework is based on ROOT; application codes need not depend on ROOT (neither is it forbidden to use ROOT in application codes).

- STAR Collaboration Meeting 8/99

- Torre Wenaus, BNL

## Visualization

- Herb Ward’s STARGL (still SGI-specific) now supplemented by new (to current STAR framework) visualization tools
- DSV (UW, Duncan Prindle) familiar to RD45; standalone GUI/visualization tool now usable with XDF based reconstruction output
- Refined over long period as physics analysis tool
- ROOT based tool (Valery Fine) integrating geometry, ROOT-aware data model into visualization capability within framework
- Online monitoring, standard displays
- Capability to apply user-developed, StEvent based filtering algorithms to define custom visualizations
- Example needed to allow users to assess value

- STAR Collaboration Meeting 8/99

- Torre Wenaus, BNL

## QA

- Peter Jacobs appointed QA Czar by John Harris
- With Kathy Turner, Tom Trainor and many others, enormous strides have been made in QA with much more to come
- QA process in the semi-weekly 3pm 902 meetings very effective
- Suite of histograms and other QA measures in continuous development
- Automated tools managing production and extraction of QA measures from test and production running being developed
- QA process as it evolves will be integrated into release procedures, very soon for dev->new and with new->pro to come
- Acts as a very effective driver for debugging and development of the software, engaging a lot of people

- STAR Collaboration Meeting 8/99

- Torre Wenaus, BNL

## Documentation and Tutorials

- Greater attention to documentation needed and getting underway
- Documentation and code navigation tools developed
- Infrastructure documentation, ease of use a priority as part of ‘consolidation’ effort
- Many holes to be filled in application code doc as well
- Entry level, user guides and detailed (reference manuals) needed
- Standards in place; encouragement (enforcement!) needed to get developers to document their code and users to participate in entry level material
- Gene Van Buren appointed Software Documentation Coordinator -- thank you Gene!
- Oversees documentation activities
- Provides prioritized list of needs
- Makes sure the doc gets written and checked
- Let Gene know what exists
- Implements easy web-based access; contributes to entry level doc
- We must all support this effort!
- Monthly tutorial program ongoing, thanks to Kathy Turner for organizing and other presenters for participating

- STAR Collaboration Meeting 8/99

- Torre Wenaus, BNL

## Computing Facilities

- Broad local and remote usage of STAR software
- RCF        35532	BNL Sun    10944	BNL Linux   6422	Desktop     3255
- HP              422	LBL              9500	Rice              3116	Indiana    17760
- Enabled by AFS based environment; AFS an indispensable tool
- But stressed when heavily used for data access
- Agreement reached with RCF to allow read-only access to RCF NFS data disks from STAR BNL computers
- Urgently needed to avoid total dependence on AFS for distributed access to (limited volumes of) data
- Trial period during August, followed by evaluation of any problems
- Linux
- New RCF farm deployments soon; cf. Bruce’s talk
- New in STAR: 6 dual 500MHz/18GB (2 arrived), 120GB disk (arrived)
- For development, testing, online monitoring, services (web, DB,...)
- STAR Linux manageable to date thanks largely to Pavel Nevski’s self-education as a Linux expert
- STAR environment provided for Linux desktop boxes

- STAR Collaboration Meeting 8/99

- Torre Wenaus, BNL

## Post Engineering Run Priorities

- Extending data management tools (MySQL DB + disk file management
-   + HPSS file management + multi-component ROOT files)
- Complete schema evolution, in collaboration with ROOT team
- Management of CAS processing and data distribution both for mining
-   and individual physicist level analysis
- Integration and deployment of Grand Challenge
- Completion of the DB: integration of slow control as data source,
-   completion of online integration, extending beyond TPC
- Extend and apply OO data model (StEvent) to reconstruction
- Continued QA development
- Improving and better integrating visualization tools
- Reconstruction and analysis code development directed at addressing QA results and year 1 code completeness and quality
- responding to the real-data outcomes of the engineering run

- STAR Collaboration Meeting 8/99

- Torre Wenaus, BNL