# Presentation: 810134

## Overview of Run II Computing

- Vicky White,
- Head, Computing Division
- September 13, 2004

## Run II Computing – continuing to evolve

- Run II Software and Computing is in “Operations and Maintenance” - we continue to enhance, replace, and evolve the systems each year to meet the needs of the collaboration
- Technology does not stand still
- Computational and storage needs of CDF and D0 evolve
- The computing models and use of off-site computing resources are evolving
- The LHC moves ever closer and the question is not whether to do Grid Computing, but how to do things in the most effective and intelligent way to optimize use of human and hardware resources and to ensure long-term support for Run II

- Vicky White - Run II Computing Review

- September 13, 2004

## CD Theme for Run II for the past year

- In the 2003 review my “theme” was how to work towards ever more common approaches between D0 and CDF
- More progress this year as you will hear
- Still some slow spots
- This year my “theme” is how to work towards common support and common Grid facilities for all of our experiments supported at Fermilab
- FermiGrid is where we are heading – all of our Farm and Storage resources grid accessible and jobs from both Run II experiments able to run on FermiGrid (and thereby on Open Science Grid in the US and LCG resources worldwide – as permitted of course)
- We are working hard to achieve this but we are not there yet
- Whenever there is work to be done – e.g. to remove bottlenecks in Farm processing, or to make job submission at remote sites work smoothly - we have been encouraging common projects and common approaches to move us towards this goal

- Vicky White - Run II Computing Review

- September 13, 2004

## Merge of CDF and D0 departments

- Run II department to better support both CDF and D0
- Amber will talk more about the recent formation of this department
- If CDF and D0 can work together on operational support and on data handling systems and analysis systems they will both be better off
- Gives depth of support
- Enhances communication on technical matters
- Supports goal of heading towards common Grid approaches

- Vicky White - Run II Computing Review

- September 13, 2004

## Recommendations from last review (2)

- D0 should further review its reconstruction code including profiling and comparison with L3 code
- D0 should resolve the system administration problem for the clueD0 cluster
- We recommend aggressive removal of the D0 SGI system
- CDF should clarify the projections for their computing needs which depend on a model that underestimates true usage by a factor of 2
- CDF should take into account the international contributions to CAF and remote computing.
- CDF should find remote facilities to help meet the demands of the proposed data rate increase

- Vicky White - Run II Computing Review

- September 13, 2004

## Recommendations from last review 

- CDF should more aggressively pursue a full SAM implementation plan
- CD should consider making SAM more of a division-level project, and aim for appropriate balance of effort from the two experiments
- CD should consolidate the administration of the Enstore systems
- CD should work on a risk assessment and benefit analysis for the proposed data dispersal plan
- CDF should invest effort in a “thumbnail” data format

- Vicky White - Run II Computing Review

- September 13, 2004

## Recommendations from last review (3)

- The experiments should recognize that database technology is now a key technology and consequently should ensure that they provide sufficient support
- There should be more coordination and coherence of database related developments
- CD and the experiments should seriously investigate forming a joint operations group for system administration of the farms.

- Vicky White - Run II Computing Review

- September 13, 2004

## Run II Computing - a collaborative endeavor more than ever

- Run II Computing and Software is, of course, an endeavor that involves many parties
- CDF collaboration and its Computing and Offline Analysis organizations
- D0 collaboration and its Computing and Core Software plus Analysis Software organizations
- Computing Division –Run II department and 3 large departments
- CEPA (Computing and Engineering for Physics Analysis)
- CCF (Computation and Communications Fabric)
- CSS (Core Support Services)
- Grid Projects efforts in US and Europe
- Common projects with US-CMS
- CERN support for some software (e.g. GEANT, ROOT)
- Increasingly – Computing Centers and institutions with significant computing resources outside Fermilab

- New this
- year

- Vicky White - Run II Computing Review

- September 13, 2004

## Computing Division Support for Run II Expts

- ~$2.7M of capital equipment for Run II Computing in FY03
- $300K used to buy UPS for High Density Computing Facility
- Great pressure on operating budget
- Maintenance+tapes      versus      people
- Will be looking to trim costs in this area over FY04 costs
- ~ 58 FTEs  of Run II effort in our 2004 WPAS budget plan  (~ $6M)
- Approximate dedicated effort for Run II is 36 FTEs (including scientists doing research part of their time)
- Common Services – will be covered in talks
- Data Movement and Storage Services
- Network Services
- Physics Applications, libraries, projects
- System and Database administration for Run II Farms
- Grid Projects
- + helpdesk, security, _facility operations_ and more

**Still very Large **
**Fraction **
**For **
**Run II**

- Vicky White - Run II Computing Review

- September 13, 2004

## Facility Support 

- Feynman Computing Center already 1 year ago was maxxed out for power and cooling
- Extremely aggressive program at the lab to deal with this situation
- New facility under construction at the old Wideband Lab – the High Density Computing Facility (HDCF) – in stages
- For FY04/FY05 - 1000 KVA   with UPS (no generator until 2007)  with room for 66 racks of 40 nodes
- Moving in this week
- Meanwhile had to add infrastructure out at the New Muon Lab
- Moved FY04 CDF early buys + 5 racks of CDF nodes and 15 racks of D0 Farm nodes out to this facility to deal with summer power load
- No UPS, No Generator
- Sharing with Lattice Gauge Facility

- Vicky White - Run II Computing Review

- September 13, 2004

## The HDCF  and New Muon Lab

- _[Wideband](http://www-cdserver.fnal.gov/cd_fnal/cdo/shared/WidebandToHDCF/2004-09-09/page_01.htm)_ in final stages of conversion to the first stage of a High Density Computing Facility
- _[Wideband](http://www-cdserver.fnal.gov/cd_fnal/cdo/shared/WidebandToHDCF/2004-05-03earlier/page_01.htm)_ in early 2004 before the conversion started
- _[New ](http://www-cdserver.fnal.gov/cd_fnal/cdo/shared/New%20Muon%20Lab/2004-06-17/page_01.htm)__[Muon](http://www-cdserver.fnal.gov/cd_fnal/cdo/shared/New%20Muon%20Lab/2004-06-17/page_01.htm)__[ Lab ](http://www-cdserver.fnal.gov/cd_fnal/cdo/shared/New%20Muon%20Lab/2004-06-17/page_01.htm)_computing facility – June 2004

- Vicky White - Run II Computing Review

- September 13, 2004

## Other Facility-related developments

- Our program of work to fully automate all of our monitoring of building controls, systems and services is just about complete
- Remedy helpdesk S/W at core of a notification and escalation system
- NGOP and other monitoring systems provide notifications and alarms
- Metasys building system (FCC and HDCF) + additional sensors
- Off-site call center part of backup human process
- Pager rotation systems for all core 24X7 services and systems
- KTeV and SDSS DLT tapes moved to robotic tape storage
- Legacy tape staging system for occasional use remains (service times not guaranteed)  e.g. for Run I data
- Enterprise backup system installed – automated all backups (except Business Services systems)
- We are without operators at Feynman from midnight to 8.0 am now (and in New Muon all of the time)
- We are ready to go to fully automated operations of all of our systems and services in all three of our Computing Centers

- Vicky White - Run II Computing Review

- September 13, 2004

## Computing Division Organization – Sept. 2003

- CDF

- D0

- CMS

- Exp.
- Astrophysics

- Exp.
- Support

- Division Infrastructure

- Core
- Support
- Services

- Computing
- and
- Communications
- Fabric

- Computing
- and
- Engineering
- For Physics
- Applications

**9**

**15**

**7 **

**16 **

**18 **

**65**

**55 **

**56 **

**26 **

**Scientists of all sorts, Engineers, Technical, Computing, Admin**** **** = 267 + 5 openings****	**

- Vicky White - Run II Computing Review

- September 13, 2004

**Notes:**
Kpro Comments:  Count taken from August 9, 2000 Organization Chart

Information should be retrieved from specific dated chart which is consistent with all other stats throughout the report.  Slide should give date or report information was extracted from.    For example.  Computing Division Organization as of September 30, 1998.

File FTE_99Chart.opx in Cd_mgt\Budget\Budget_Planning_Memos\Directors_CD_Bud_Review_FY99\FY99_DirBudRev_Support_Files

## Computing Division Organization – Sept. 2004

- RunII

- CMS

- Exp.
- Astrophysics

- Exp.
- Support

- Division Infrastructure

- Core
- Support
- Services

- Computing
- and
- Communications
- Fabric

- Computing
- and
- Engineering
- For Physics
- Applications

**15 +0+5**

**16**

**6 **

**35+0+1**

**61+3+1**

**52 +2 + 3**

**52 +5 +2**

**30 **

**Scientists of all sorts, Engineers, Technical, Computing, Admin**** **** = 267+10 +12**

- Posted openings

- Requested openings

- Vicky White - Run II Computing Review

- September 13, 2004

**Notes:**
Kpro Comments:  Count taken from August 9, 2000 Organization Chart

Information should be retrieved from specific dated chart which is consistent with all other stats throughout the report.  Slide should give date or report information was extracted from.    For example.  Computing Division Organization as of September 30, 1998.

File FTE_99Chart.opx in Cd_mgt\Budget\Budget_Planning_Memos\Directors_CD_Bud_Review_FY99\FY99_DirBudRev_Support_Files

## CD Budget and Manpower assumptions for Run II for the future

- Planning on no more than $3M of capital equipment per year – may need to deduct “tax” for facility infrastructure
- Expect to trim operating costs compared to FY04
- Savings from SGI maintenance but Network, Oracle, DB machines all increasing
- Not planning for a tape budget matched to CDF’s most ambitious needs or matched to our desire to replicate data
- technology may save us
- Make CDF and D0 reconstruction (and analysis) compatible with FermiGrid
- FY05 is the time to execute the plans and evolutions that put Run II in good shape for common services
- FY06 is the time to start reducing effort on Run II developments and operations

- Vicky White - Run II Computing Review

- September 13, 2004

## Run II Computing is working well 

- Both experiments have computing facility hardware, services and support that enables them to
- Reliably record data from the detector
- Create and store large amounts of MC data
- Reconstruct the data rapidly
- Provide good access to the data onsite and offsite
- Make new releases
- Get physics results out
- The questions at this stage are
- Are the future requirements well understood?
- Can they be met within the resource constraints?
- Are we all working (together) intelligently and optimally to provide support and evolution for Run II Computing and Software?
- Do we have a workable strategy to deal with the gradual transition from the Run II era to the LHC era?

- Vicky White - Run II Computing Review

- September 13, 2004

## Thanks

- For spending the time to learn about our progress, problems and plans for Run II
- We look forward to your comments and suggestions – they will be immensely helpful to us

- Vicky White - Run II Computing Review

- September 13, 2004