# Presentation: 127661

## Status of Computing and Software

- Amber Boehnlein

- DØCA - Operations Report

## Focus on Operations

- Most systems are in an operational mode
- Problems remain
- There are some conspicuous exceptions
- Aggressively working to improve operations
- Operational support in the Computing Division is phenomenal
- SAM shifters making a difference
- Working to develop metrics, reported weekly at the ADM
- Users typically focus on farm production, SAM, and central systems performance
- Part of a chain, so provide good measures but
- That doesn’t mean that everything is SAM’s fault!

- DØCA - Operations Report

## Local Farms Status

-     DØ Farms Processing: 13.7 M evts processed last week  rate over 2.5 M events per day, 2.7M peak (goal 3 M)  outstanding issues:                                     Not enough tape drives
-                                         Higher cpu time for RECO at higher IntLum                                   Load/problems on D0bbin                                 Current reco vers - p13.06.01

**NOV**

**Feb 10**

**Three month CPU utilization on the farms**

- DØCA - Operations Report

## MC Farms Status 11 Feb 2003

- Mcp13 running smoothly
- 4.6 M events generated and stored in SAM so far this January for Moriond!
- Most Moriond Samples generated and stored
- All high priority samples generated
- Pausing for reflection
- To Come
- p13.08.00 certification samples running
- trigsim now part of standard processing
- D0gstar packing bug fixed
- mcp14 next phase
- Tutorial – see MC Production Web Site

- DØCA - Operations Report

## Updated Thu Jan 16 07:38:31 CST 2003 http://www-d0.fnal.gov/computing/mcprod/request_details/Request.html

**The Next Request to be processed is Request ID = 3558 **
*Updated Thu Jan 16 07:38:31 CST 2003* _[http://www-d0.fnal.gov/computing/](http://www-d0.fnal.gov/computing/mcprod/request_details/Request.html)__[mcprod](http://www-d0.fnal.gov/computing/mcprod/request_details/Request.html)__[/request_details/Request.html](http://www-d0.fnal.gov/computing/mcprod/request_details/Request.html)_

- DØCA - Operations Report

## Central Systems Status

- Downtimes - unscheduled:  D0lomite
- Downtimes - scheduled: Feb 4, hardware fixes, rebuild file system on D02ka
- Next Scheduled Downtime: March. 4 th
- D0mino will be reduced to 128 processors—save on maintenance costs
- Do we need IRIX builds after p14?

**Jan-Feb CPU usage on CAB**

**Jan CPU usage on D0mino**

- DØCA - Operations Report

## Tape usage Library      Tot	 Blank   Used   Stored            Effmezsilo 3225        162   3063  178.6TB    0.97  samlto   1099        291     808    71.2TB    0.88Writing reco output to 9940bs (180 GB/tape, 30 mb/sec)	test jobs for reading are starting now.	CDF has copied ~800 tapes, 1 read error/450 tapes
There is crud in our STK robot—investigations continue

- There is crud in our STK robot—investigations continue

**Robotic Storage**

**Data transfer rate/day**

**7TB**

**Past week**

**TB/day**

**Number of STK drives in use**

**Nov**

**Feb**

**15**

- DØCA - Operations Report

## Data Handling

**30 day data consumption on CAB**
**integrated**

**35TB**

- Activity on FNAL analysis SAM stations 02/02/03-02/09/03
- CLuED0:350 projects, 800 GB consumed, 300 GB/50 GB transferred in/out
- CAB: 350 projects, 10.0 TB consumed, 0.8 TB/0 TB transferred in/out
- D0mino: 2015 projects 10 TB consumed (2.0/0.5 TB in/out), 80%-90% cache hits
*D0Karlsruhe- 98 projects!  See talk on Friday*

**Data Handling**

- DØCA - Operations Report

## CAB transfers: in (+) out (-)

**ClueD0 transfers: in (+) out (-)**

**11/2/02**

**12/2/02**

**16TB**

**1 TB**

**CLuED0 transfers in/out**

**MC stores?**

- DØCA - Operations Report

## Consumption on CLuED0

- DØCA - Operations Report

## Analysis Feedback

- Recent GCAS focused on analysis experiences
- Different groups have different approaches
- All groups seem to run or use standard skim sets
- Some groups use of thumbnails/framework directly for analysis
- Some groups use private tuple formats or derived from old reco_analyze format
- Some groups using TMBTree
- Some groups inserting skimmed data sets in SAM—in general, groups doing this want MORE functionality, either with extended metadata or with metadata generated for Root jobs
- Several groups rely heavily on pick_events and re-reco—evaluate other mechanisms for long term
- Focus standardizing tools, integrating runjob, rte, d0tools would be a good step
*Additional meeting Thursday at 4:30, Ninth Circle*

- DØCA - Operations Report

## On the Mail Lists

- Cab policy regarding non-SAM jobs
- Want to track legitimate analysis use of system
- System can not be used MC production farm—if someone sees a group running large scale production, we will kill the jobs
- What is small scale MC generation or production?  That which does not attract the attention of the other uses, either by flooding the queues or by hosing the network.
- Set up a medium queue on CAB for better control and monitoring
- Helpdesk is not 24/7.  Call x2345 for after hours instructions for really big problems— like no home areas. _[helpdesk@fnal.gov](mailto:helpdesk@fnal.gov)__  now returns message after hours and that web page give after hours instructions_
- DAB networking—working to address
- Report of corrupt files on tapes—investigating possible tape faults

- DØCA - Operations Report

## News

- EDM/root project-Marc Paterno reported good progress
- While showing interface to D0, was reminded of a constraint necessary for interactive use of “draw” command in Root.  Highly constrained interface (not OO).  Three possible solutions for consideration
- Store “drawable” objects on separate branches
- Adapt/extend root draw
- Make new “draw” with interface that can handle objects.
- Production Release strategy proposal
- P14 based on p13, cut Feb 4, and all changes will be tightly controlled (KCC compiler)
- Cut p15 based on the test releases in March (gcc compiler).
- See Harry’s talk on Friday

- DØCA - Operations Report

## News, Continued

- Job submission and monitoring using Grid based tools are scheduled for commissioning deployment in April
- D0 institutions has submitted an ITR GRID proposal (Umich, Michigan State, UC Riverside, UTA and Fermilab).
- Congratulations and good luck.
- Active work at Umich on farm scripts for re-reconstruction, access to National Partnership for Advanced Computing Infrastructure Resources.

- DØCA - Operations Report

## Databases

- Testing of use of calibration Dbservers for Farm production
- New machines for dbservers—Moving SAM dbserver
- “Killer Queries” to SAM dbservers are frustrating to everyone. Use the Metadata, avoid wildcarding, use of filenames*See SAM tips and tricks tutorial*
- Monitoring connections on SAM web server node. SamDbServers are currently running on this node and run-away network connections can cause serious global performance degradation
- Luminosity database/dbserver considered as part of the on-going luminosity review

- DØCA - Operations Report

## Computing System Planning

- Yearly review process started 2002
- Continue process by updating planning document
- Get input to refine the computing estimates and evaluate needs
- Provide standard cost estimates for FNAL systems and remote systems, financial estimates
- Increased budget pressure at FNAL likely to continue
- Target for internal review ~ April

- DØCA - Operations Report

## 2003 Budget

***Budget guidance $1.5M + $0.2M carry-over, still no budget***

- DØCA - Operations Report

## Costing Assumptions

- Analysis cost was estimated by modeling possible user scenarios—Demonstrated need for collaboration resources beyond FNAL for computing, but does not reflect full system costs
- Disk and servers
- Networking
- Infrastructure such as gateway machines, code servers, and db server machines
- Mass storage
- Cache machines and drives to support extensive data export
- Wide Area Networking
- Disk estimate only for FNAL central SAM cache on IRIX machine, project disk not included
- Monte Carlo estimates not included
- Reprocessing not included

- DØCA - Operations Report

## A RAC/IAC example

- Assume 10% dsts
- Assume 0.1% RAW and 0.1% RAW/DST
- Assume 2 copies data thumbnails
- Assume 20% each PMCS and MC thumbnails
- Assume disk purchased as needed
- Assume 25 Medium jobs and 25 short jobs

- DØCA - Operations Report

## Slide 20

- DØCA - Operations Report

## Conclusions

- Most systems working reasonably well, but with some operational issues
- SAM stations on CAB and CLuED0 are working
- Current hardware and functionality in computing model is meeting basic needs.  Time to evaluate next steps for development
- Analysis is occurring at remote sites, all TMBs at Lyon
- Next stage in planning underway

- DØCA - Operations Report

## Some Special Thanks	

- Reorganization in progress—General thanks to all members of the Computing Planning Board, past and present.
- 	Special thanks to Chip Brock, spreadsheet wizard, stepping down from DB/DH.
- 	Very special thanks to Dave Ritchie—He came to D0 to do NT builds and stayed on to develop RTE and to provide the minutes of the Friday meeting.  Taking on new responsibilities at CD
- Computing and Software Co-leaders
- Wyatt Merritt and Bruce Gibbard (pre-history)
- Wyatt Merritt and Nick Hadley (1996-2000)
- Wyatt Merritt and Heidi Schellman (2000-2001)
- Wyatt Merritt and ASB (2001-six months)
- Nick Hadley and ASB (2002-six months)
- Jianming Qian and ASB (2002-2003)
- Best wishes and thanks to all C&S past co-leaders!

- DØCA - Operations Report