# Presentation: 425546

## Software Project Status

**Torre Wenaus**
**BNL**
- U.S. ATLAS Physics and Computing Advisory Panel Review
- Brookhaven National Laboratory
- May 21, 2001

## Outline

- Brief software project overview
- Objectives, scope, organization
- Project planning status
- WBS, schedule, relation to International ATLAS
- Technical progress; current and near-term activities
- Core software (Framework/Architecture, Databases, Event Model)
- Software support and QA/QC
- Training & collaborative tools
- Grid software
- Status relative to Nov PCAP comments, recommendations
- Budget/Personnel: Priorities, status

- Torre Wenaus, BNL               PCAP Review

- May 21, 2001

## U.S. ATLAS Software Project Overview

- Major roles in key core software domains which leverage U.S. capability and are central to U.S. physics analysis
- Control framework and architecture
- Chief Architect, Athena framework development, event model
- Databases and data management
- Database co-Leader, major development roles
- Software support: for developers, users and physics analysis
- Software librarian, quality control, software development tools, training...
- Subsystem software roles complementing hardware responsibilities
- Closely coupled to core development: tight feedback loop
- Leadership roles commensurate with our activities
- Scope commensurate with U.S. in ATLAS: ~20% of overall effort

- Torre Wenaus, BNL               PCAP Review

- May 21, 2001

## U.S. ATLAS Software Organization

- Torre Wenaus, BNL               PCAP Review

- May 21, 2001

## Project Planning Status

- U.S. ATLAS WBS based on XProject essentially complete
- XProject extensions to support International ATLAS complete; integration in ATLAS by Helge Meinhard in progress
- More detailed U.S. WBS being integrated into ATLAS PBS
- U.S. and ATLAS versions will mostly coincide (wherever possible)
- ATLAS and U.S. ATLAS schedules fully integrated
- U.S. and ATLAS project management cooperating and in synch: working from same work breakdown and schedule sources
- Common sources (data and code) in ATLAS CVS
- New ‘WBS projections’ capability of XProject also used for grid planning: ‘US Grid Computing WBS’ projection developed

- Torre Wenaus, BNL               PCAP Review

- May 21, 2001

## Schedule

- Integrated (U.S. software + U.S. grid + ATLAS), comprehensive schedule developed
- Linked to U.S.ATLAS, ATLAS, U.S. Grid WBS’s throughout
- Supports, but does not yet show most linkages between tasks/milestones
- Reasonable detail for next 1-2yrs; sketchier beyond that
- WBS and schedule are input to the U.S. ATLAS project management accounting and tracking system
- Microsoft Project version (e.g. for Gantt charts) autogenerated (but not used to date)

- Torre Wenaus, BNL               PCAP Review

- May 21, 2001

## XProject Project Planning Tool

- WBS, schedule, personnel info implemented as simple text files
- Maintained in the ATLAS CVS repository
- WBS and schedule ATLAS-wide; personnel is U.S.-only
- Text sources processed by home-grown software ‘XProject’
- Text source converted to XML conforming to an XProject DTD by perl script
- XML is parsed and processed by a Java program
- Builds object collections for WBS, sched, personnel
- These are used to generate needed outputs
- Static and dynamic (Java servlet) web pages showing project info, filtering on dates, tags, WBS nos., etc.
- Input (CSV) files for U.S. ATLAS Project Office:
- Access databases for manpower, WBS
- Microsoft Project

- Torre Wenaus, BNL               PCAP Review

- May 21, 2001

## U.S. ATLAS - ATLAS Coordination

- Torre Wenaus, BNL               PCAP Review

- May 21, 2001

## ATLAS Detector/Task Matrix

|  | Offline Coordinator | Reconstruction | Simulation | Database |
| --- | --- | --- | --- | --- |
| Chair | N. McCubbin | D. Rousseau | K. Amako | D. Malon/
 R.D. Schaffer |
| Inner Detector | D. Barberis | D. Rousseau | F. Luehring | S. Bentvelsen |
| Liquid Argon | J. Collot | S. Rajagopalan | M. Leltchouk | S. Simion/
R. Sobie |
| Tile Calorimeter | A. Solodkov | F. Merritt | A. Solodkov | T. LeCompte |
| Muon | G. Poulard | J.F. Laporte | A. Rimoldi | S. Goldfarb |
| LVL 2 Trigger/ Trigger DAQ | S. George | S. Tapprogge | T. Hansl-Kosenecki | H. P. Beck |
| Event Filter | V. Vercesi | F. Touchard |  |  |

**New since last time: Pavel Nevski (BNL), Geant3 simulation coordinator**

- Torre Wenaus, BNL               PCAP Review

- May 21, 2001

## Technical Progress Since November

- Control Framework and Architecture
- Databases and Data Management
- Software Support and QA/QC
- Training and collaborative tools
- Grid Software
- Subsystem software -- no time to cover this

- Torre Wenaus, BNL               PCAP Review

- May 21, 2001

## Control Framework and Architecture

- Major Athena release in Jan (‘December’ release)
- StoreGate data model deployed
- Interactive scripting based on Python deployed
- Athena Users Guide written
- Common Gaudi core code base established with LHCb
- Event filtering into multiple output streams
- Data dictionary prototype developed
- Architecture Review Committee final meeting 5 Mar
- Final reported awaited. Bottom line: Athena will be endorsed.
- Near term focus: Physics analysis release supporting physics workshop in Sep; expected in June.
- Generator and simulation support; Objy and ROOT I/O support

- Torre Wenaus, BNL               PCAP Review

- May 21, 2001

## Databases and Data Management

- ANL staffing ramp completed in Jan: 3.5 FTEs (.5 supported by ANL) plus 0.5 at Chicago
- Database support for Athena released in Jan
- Objy baseline, and ROOT I/O (based on ROOT ‘blobs’)
- Supporting Atlfast fast simu, and user-written I/O
- MySQL deployed in LAr for conditions data; also used for bookkeeping, file & metadata management
- Being integrated into overall DB effort
- DB technology evaluation plan in development
- Technology choice probably delayed to mid ‘01, allowing time for adequate evaluations of Objy, ROOT I/O, (object) relational DBs
- Integration with early distributed file management tools soon
- Output collection registration

- Torre Wenaus, BNL               PCAP Review

- May 21, 2001

## Software Support, QA/QC

- U.S. based support of releases, other software
- Provided in AFS for use throughout the U.S.
- Nightly build facility developed and deployed
- Full software build based on most recent tags; email to developers
- Ease integration of ATLAS software into timely releases
- ATLAS Coding Standards document edited and released
- HyperNews web-based discussion tool deployed
- U.S. based CVS repository (+browsers, etc.) established
- Preliminary and U.S.-local projects; personal CVS areas

- Torre Wenaus, BNL               PCAP Review

- May 21, 2001

## Training, Collaborative Tools

- New cycle of training courses initiated
- Solicitation to gather interest: C++, Java, G4, code management tools,...
- 1-week Geant4 course in Michigan in Feb
- With careful arrangements to record it using the Syncomat web-based lecture tool
- Video, audio, synchronized slides
- High-quality result should be released in a few weeks
- LAr reconstruction training in Dec
- Athena based software in action; use of StoreGate event model
- New Athena user manual very valuable

- Torre Wenaus, BNL               PCAP Review

- May 21, 2001

## Grid Software

- Much attention since last PCAP (part political, part technical)
- Objective is to integrate grid software activities tightly into ongoing core software program, for maximal relevance and return
- Grid WBS developed, clarifying relative roles of Software and Facilities
- Software workplan developed (expressed in XProject schedule/WBS)
- Grid tool (Globus) deployment and testing getting underway
- In-house distributed data and processing infrastructure and prototype planning/development underway; utilizing Globus, Condor, ...
- Integration of early distributed data, ‘virtual data’ tools into Athena to start soon
- Output dataset registration in distributed catalog
- ‘Generate on demand’ virtual data trial in Atlfast fast simu planned

- Torre Wenaus, BNL               PCAP Review

- May 21, 2001

## Subsystem Software

- Activity continuing...
- Silicon Tracker
- Transition Radiation Tracker (TRT)
- Liquid Argon Calorimeter
- Tile Calorimeter
- Muon Spectrometer

- Torre Wenaus, BNL               PCAP Review

- May 21, 2001

## Subsystem Software Activities

- Performance/design studies – All
- G3 based simulation – All
- G3 simulation coordination for ATLAS
- Test beam – Si, LAr, Tile
- Athena integration – Si, LAr, Tile, Muon
- Reconstruction development in C++ - Si, LAr, Tile, Muon
- G4 based simulation development – Si, TRT, LAr, Muon
- G4 physics validation – Si, LAr
- XML based detector description – TRT, Muon
- Database – LAr, Tile, Muon

- Torre Wenaus, BNL               PCAP Review

- May 21, 2001

## Summary Milestone Schedule

- Software in blue

**Done!**

**Done!**

**Probably mid 2002**

**Probably delayed**

- Torre Wenaus, BNL               PCAP Review

- May 21, 2001

## Budget (= Personnel) Priorities

- Status of pre-FY01 priorities:
- Control/Framework:  LBNL build-up supporting A-Team achieved
- Databases: ANL build-up supporting DB role finally achieved
- Support: Software librarian at BNL in place
- Subsystem support: software professionals at core/subsystem interface unfunded -- out of project scope set by agencies
- FY01(+) priorities:
- Sustain LBNL and ANL efforts
- Begin the delayed BNL ramp: Add first sw pro developer
- New hire in progress now. Increment by 1 person/yr through FY04
- Info on BNL program follows
- Establish sustained presence at CERN
- Unfunded; 1 person @ CERN via existing funds (LBNL relocation)

- Torre Wenaus, BNL               PCAP Review

- May 21, 2001

## FY01 Budget Guidelines

- Original agency profile of monies in software FY01-06 was an *impossible* one for software (not too strong a word)
- Extremely sharp and late peak looks like the profile of a failed project:
- Program fitting the profile makes critical mass at any US site impossible
- Dismissals of valuable HEP experts would have been necessary
- We developed a ‘compromise profile’ well below our Jan 2000 proposal  which provides the needed flatter profile

- Torre Wenaus, BNL               PCAP Review

- May 21, 2001

## Funding Profile Comparisons

**Agency guideline**

**Jan ‘00 PMP**

**Compromise profile**

- Torre Wenaus, BNL               PCAP Review

- May 21, 2001

## Planned & Required Effort Levels

| FTEs by FY | FY01 | FY02 | FY03 | FY04 | FY05 |  |
| --- | --- | --- | --- | --- | --- | --- |
| Arch/Frame | 7.0 | 6.3 | 6.2 | 6.1 | 6.4 | Needed |
|  | 6.0 | 5.9 | 6.4 | 6.4 | 6.4 | U.S. provided (LBNL + BNL) |
| DB/Data mgmt | 14.6 | 15.8 | 18.3 | 16.5 | 18.5 | Needed |
|  | 5.5 | 6.9 | 7.4 | 8.3 | 8.3 | U.S. provided (ANL + BNL) |

- Needs based on bottom-up estimate of Int’l ATLAS needs from WBS level 5. Developed by U.S. software managers based on experience (developed by one of us, reviewed by other two; revisions were small). Broadly consistent with International ATLAS estimates.

- Torre Wenaus, BNL               PCAP Review

- May 21, 2001

## BNL Software Program

- Program complements BNL roles: Tier 1; physics analysis center; subsystem software development
- Principal core activity areas:
- Databases and data management
- Event model
- Physics analysis infrastructure
- Closely coupled to both of these: grid software development
- Software support and QA/QC
- Project management
- All areas leverage local expertise and experience (STAR/RHIC, D0)
- Subsystem software development activities closely couple to hardware expertise (Lar, muons) and core activity

- Torre Wenaus, BNL               PCAP Review

- May 21, 2001

## Current & Near Term BNL Software Activity

- Software support and quality control
- Databases and data management
- Relational (MySQL) database development for LAr conditions data and distributed data management
- ROOT based I/O, applying STAR experience: Currently LAr applications, leading to role in ATLAS event store evaluation
- Event model development: StoreGate
- Grid software
- Distributed data services prototype in development
- Software project management
- Also ‘user liaison’ with Facilities
- LAr, muon simu/reco; G3 coordination for ATLAS

- Torre Wenaus, BNL               PCAP Review

- May 21, 2001

## ATLAS Software Organization at BNL

- Physics Applications Software (PAS) Group established 10/2000
- “To provide a physics software development capability and expertise base, particularly to the BNL HEP program and the ATLAS experiment.”
- Fully focused on ATLAS software
- Physicists and professionals migrating from STAR, and new professional hires. Currently:
- Yuri Fisyak, Pavel Nevski, Torre Wenaus, Valery Fine, Alex Undrus
- Close collaboration with ATLAS physicists in Omega group
- Currently H.Ma, S.Rajagopalan (LAr group) in particular
- Ramp: one project-supported developer today; 4 in FY04

**http://www.usatlas.bnl.gov/AFSWEB/bnl/pas**

- Torre Wenaus, BNL               PCAP Review

- May 21, 2001

## Nov PCAP - Comments Arising

- Hire 2 DB persons *done*. Increase CERN presence. *Unfunded. One person from LBNL resident at CERN on existing funds. Fulfilling U.S. staffing has priority if funds are short.*
- Collaborative data mgmt reassessment encouraged. *In the works, in collaboration with CMS in the U.S. and LHC experiments/CERN internationally. U.S. ATLAS role in Objy, ROOT I/O, relational DBs expected.*
- Formal decision on Athena needed. *Will be soon and positive.*
- QC should be strengthened with Athena the priority application. *BNL provided increased manpower. SW test/validation improved with new nightly build scheme, with Athena/D.Quarrie a principal client, motivator and source of design and functionality input.*

- Torre Wenaus, BNL               PCAP Review

- May 21, 2001

## Nov PCAP comments (2)

- Negotiate with DOE and NSF the required funding profile for software development. *Boy, how we try!*
- Complete and debug the WBS. *Done, modulo ongoing refinements and Int’l ATLAS-driven changes. *Unify U.S. WBS with ATLAS PBS. *Complete from the U.S. side, and in progress on the ATLAS side. *Some areas need to be developed in more detail. *Areas without U.S. activity are probably still weak and await International ATLAS input.*

- Torre Wenaus, BNL               PCAP Review

- May 21, 2001

## Summary

- Core teams in place at LBNL, ANL; focus now on sustaining these and building up at BNL according to plan.
- Framework/Architecture progress and acceptance good. Architecture review should (finally) be coming out with an endorsement of Athena and many useful and substantive recommendations
- Database progress increased but still short on resources both in U.S. and internationally. Technology choice delayed to mid-2002 (?) to allow time for realistic evaluations. U.S. should play a major role -- in Objy, ROOT, RDBMS (options and evaluation plan not settled) -- at ANL and BNL
- Useful progress and clarification in grid software planning
- Geant3 coordination for ATLAS a welcome addition which strengthens our role in near term Data Challenges and ‘applied’ distributed processing
- Funding situation is perilous: both project and base program, and we still await substantive return on large time investments in grid and other proposals.

- Torre Wenaus, BNL               PCAP Review

- May 21, 2001