# Presentation: 863632

## D0 Grid Data Production Initiative:Coordination Mtg

- Version 1.1 (post-meeting edition with notes)
- 26 February 2009
- Rob Kennedy and Adam Lyon
- Attending: RDK, AL, MEV, KC, LH, RI, QL

## Overview

- News and Summary
- System mostly OK
- Expanded to ~1800 slots w/ retired CDF nodes.
- Tues drop: operator hand-off error
- CAB2 Job Dequeuing issue seems resolved
- Cabsrv2 memory capacity increased – helped, but not a fix
- Maui partitioning introduced – seems to be a fix.
- Backlog: 178 MEvts (Raw – Unmerged)
- Up a little bit from 175 MEvts last week.
- Is 5.8 MEvts/day is good day or not? Need L value for data processed to decide.
- Are we getting the increased output we expected by adding the (older, retired) CDF nodes?
- Presenting to CPB this Friday
- Brief summary of Phase 1 progress, Phase 2 Work list, and recent progress
- Agenda (very short meeting this week)
- Expanded Use of CAB2: March 10 – April
- Phase 2 Overall Planning – prioritized work list
- Just covered some CAB2-related ideas that would part of a Phase 2 review.

## Phase 2 Near-Term Tasks

- Increasing Capacity

## Capacity Timeline Skeleton Drafthelps drive short-term tasks

- Mid-February 2009  Early-March: better, but not Keep-Up Level yet.
- MEV initiated: Add 115 retired CDF worker nodes... 4 years old... about 230 (slower) slots.
- Create Model to Determine Keep-Up Level  = f(L).
- Identify and strengthen anticipated bottlenecks to expansion: Looks Viable As Is for 25-50% increase. More is speculation.
- Early-March  April 2009: Keep-Up Level + Work-through-Backlog Level
- Upgrade PBS head nodes (FEF) during March 3 downtime  This is now scheduled for March 10 (QL)
- Add ~all CAB2 analysis nodes to d0farm queue for use by Data Production _**(QL: Proposal Accepted... next slide)**_
- Not only to scale up to “keep-up” CPU level, but to work down the backlog.
- Scale-up in steps to avoid a major roll-over: 18002500 slots, run 3 days+weekend, sign-off Mon, 25003200 slots Mar 17 if OK.
- Setup steering committee to track end conditions (1 Initiative, 1 D0, 1 CD) and someone to report to them on daily basis?
- RDK to arrange Mon March 2 meeting to discuss the details of plan
- D0 analysis on the Grid to help avoid triggering end of CAB2 expanded use: not likely to taken up by users in this time scale if at all.
- Push more opportunistic use of CPU, eg. GPFarms. CMS nodes were available for a time, for instance.
- Consider opportunistic use of “Other VO” cpu during this same time period
- Purchase Req out in late March for more WN + ?                                         ALSO: END of April = END of Initiative
- May  July 2009: not Keep-Up Level, could be close w/opportunistic use. GAP TO BE FILLED in this scenario.
- Downsize system as analysis CPU returned, less opportunistic CPU available... how to maintain enough CPU to keep up?
- Develop an estimate of FY10 needs for budget planning, working with CD and D0
- New CPU may arrive in July, but will have to be burned in, infrastructure tested, etc
- August  December 2009: Keep-Up Level (+ headroom?)
- Add CPU and infrastructure (from procurement) to support a “keep-up” system.
- Refine FY10 budget request as more experience gained with newly expanded system. Receive budget allocation, plan FY10.

## CAB2 Temp Expanded Use (create a name for this activity?)

- Early-March  April 2009: Keep-Up Level + Work-through-Backlog Level
- Temp Expanded CAB2 Use by Data Production: 2/20/2009 via Email:
- 	Regarding temporarily using the whole CAB2 for the production, D0 management has made a decision that from March 10, we will temporarily expand the d0 farm queue to be the whole CAB2. The purpose is to catch up the backlog in data production for the summer conference. This configuration is temporary. We will change it back to the current configuration when one of the following condition happens:
-    - when the backlog has been reduced to be less than one week of data; or    - May 1, 2009, or    - when there is an analysis need for more CPUs than CAB1 can provide. Although the configuration change will be done by FEF (thanks to FEF!), the SamGrid team may need to plan to adjust related parameters to handle a much larger production farm. The current d0 farm queue has 1800 job slots. The new d0 farm queue will have 1800+1400 job slots, temporarily. Thank you, Qizhong
- (meeting ended here, with a quick trip to slide 12)

## Near-Term Work: Action Items

- Phase 1 Follow-up – Done during early Feb downtime?
- Enable auto-update of gridmap files on Queuing nodes: AL: 2 done. 1 more complicated, not done... does not require a downtime.
- Enable monitoring on Queuing nodes – AL: not all done yet.
- Understand PBS Scheduling (QL, JA)
- Memory increase in cabsrv2 – improved dequeuing performance?
- JA: Best estimate: hardware performance limitation on cabsrv2. PBS head node upgrade March 3 is next step.
- MEV: alternative date like March 17 to avoid last minute work for Moriond? Follow-up with QL. (... GCC-B power outage April 7th)
- CDF Node deployment (FEF)
- Remainder: LH – 10 to 12 being used to test new PBS head nodes.
- Test Data Production with Opportunistic Usage (MD)
- MD: Not sure how production role will be treated at other sites. KC: Depends on GUMS server setup.
- MD: Success on GPFarm up to storage of TMB. KC: Please use site gateway fermigridosg1. Avoid jobs going to CAB2, etc. Mike will test larger-scale use (of reconstruction stage only, not the merge stage) over the next week.
- Data Flow Diagram and the “next” Bottleneck (RDK)
- MEV: Discussed with Simon Kwan about creating simulation of data flow and queuing. Determine impact of changes beforehand.
- MD: Observed long waits for transfer of unmerged TMB to SAM cache (in merging step). Discussion. Consider workflow latency, network or hardware contention when next occurs. LH: Which nodes involved?
- RDK: Evaluated space and rates on cache and durable storage nodes.
- Output 25% > input on d0srv071, 72? = Tarballs.		Also, d0srv063 swapping June-Oct ’08, but memory 2x overkill on d0srv065?
- Avg event size: unmerged = 180-240 kB/evt		merged =80-200 kB/evt
**Early conclusion**: dataflow OK for 25% increase. May want more storage nodes for bandwidth, contention relief at 100% scale increase.

## Data Flow for Data Production

_**Enstore**_
- LTO4-G

_**Enstore**_
- LTO4-F

_**SAM Cache**_
- d0srv071
- d0srv072

_**Durable Store**__**,**_
_**Stager Space**_
- d0srv063
- d0srv065

_**Worker Nodes**_
- Scratch space

- Raw Data

- Merged TMB

- Tarballs

- Unmerged
- TMB

- Shared w/Analysis Users

- No automated failover between ‘63, ‘65

- Durable Storage and Stager Space are on separate partitions

- In2p3 remote uploads

- Also on cache nodes: 0-bias skim, LCG cache

- Initiated by Merge Job, Via gridftp

***1***

***0***

***2***

***3***

***4***

***5***

***6***

***7***

- Initiated by Reco Job

- Other data destined for Tape Storage

## Phase 2 Task Priorities

- Fit Tasks into Schedule with Limited Resources:
- “Identify someone to do task(s) in
- February – April 2009 or defer task”

## Phase 2 Work List Outline

- 2.1 Capacity Management: Data Prod is not keeping up with data logging.
_**Capacity Planning:**_ Model nEvents per Day – forecast CPU needed
_**Capacity Deployment:**_ Procure, acquire, borrow CPU. Infrastructure capable?
_**Resource Utilization**_: Use what we have as much as possible.
- 2.2 Availability & Continuity Management: Expanded system needs higher reliability
_**Decoupling:**_ @ SAM station, @ Durable Storage, Durable Storage capacity, FWD4a/b
_**Stability, Reduced Effort:**_ Deeper Queues, Scripted workflow vs PBS behavior, fcp conf.
_**Resilience:**_ Eliminate SPOF, add/improve redundancy
_**Configuration Recovery:**_ Capture configuration and artefacts in CVS consistently.
- 2.3 Operations-Driven Projects
_**Monitoring:**_ Can we “see” above in real-time? Can we trace jobs “up” as well as down?
_**Issues:**_ Address “stuck state” issue affecting both Data and MC Production
_**Features:**_ Add state at queuing node (from Phase 1). Distribute jobs “evenly” across FWD.
_**Processes:**_ Enable REX/Ops to deploy new Condor. Revisit capacity/config 2/year?
_**Phase 1 Follow-up: **_auto-update of gridmap files, FWD3 reboot – DONE?
- Deferred Work List

## 2.1 Capacity Management

- Highest Priority: will treat in Phase 2. Refine work list to be more detailed.
- Mostly covered in “short-term work”...
- 2.1.1 Capacity Planning: Model nEvents per Day – improve to handle different Tevatron store profiles, etc.
**Goal: What is required to keep up with data logging by 01 April 2009?**
**Goal: What is required to reduce backlog to 1 week’s worth by 01 **_***June ***_**2009? ... Use all CAB2 to catch up Mar/Apr**
- Follow-up: Planning meeting Monday?
**Goal: What infrastructure is required to handle the CPU capacity determined above?**
- Latencies impacted CPU utilization?  MD, Adam looking into (consider oversubscription if so)
- Ling Ho: 8 core servers (2008) have scratch/data on system disk (only 1 disk). Have observed contention, slowing jobs down.
- Data Handling tuning (AL, RI): tuning SAM station, investigate if CPU waiting on data.
- 2.1.2 Capacity Deployment: **Goals to be determined by capacity planning**
- Added CDF retired nodes (FEF, REX): 17 Feb 2009
- Upgrade PBS Head nodes (FEF): 3 March 2009
- Plan and execute temporary expansion of “d0farm” on CAB2 to work through backlog in March and into April
- Plan and execute some level of opportunistic use of CPU for data production.
- Infrastructure capacity: appears sufficient for 25% CPU increase, probably OK for larger increase.
- Fcp config: achieved optimal config to max out network bandwidth? Done.
- 2.1.3 Resource Utilization: top-down investigation as well as bottom-up investigation. Examples...
**Goal: > 90% of available CPU used by Data Prod (assuming demand limited at all times)**
**Goal: > 90% of available job slots used by Data Prod averaged over time (assuming demand limited at all times)**
**Goal: TBD... Uptime goal, downtimes limited.****	****	**Activities to maintain/achieve this overlap with the following...

## 2.2 Availability & Continuity Mgmt

- Some tasks should be done, others deferred to after April (post-Initiative). (+ FGS recommends)
- Initiative will keep a list of these and pass on to groups in place to pursue in normal course of business.
- 2.2.1 Decoupling
- All work deferred post-Initiative
- 2.2.2 Queue Optimization for Stability, High Utilization, Reduced Effort
- Goal: System runnable day-to-day by Shifter, supervised by expert Coordinator
- Deeper Queues (yes), Scripted workflow vs PBS behavior (may-be) – Revisit after PBS head node upgrade
- MD: Increase limits to allow either FWD node to handle the entire load if the other were to go down.
- 2.2.3 Resilience
- Eliminate SPOF, add/improve redundancy (see following slides)
- 2.2.4 Configuration Recovery
- Capture configuration and artefacts in CVS consistently. (continue to do)

## FGS Recommendations

-  Configure d0cabosg[1,2] to have the capability to manage jobs on both d0cabsrv[1,2]. (Yes, do in Phase 2, by FGS, at low priority)
- Requires a bit of “hackery” in the jobmanager-pbs script, but is doable. Both d0cabosg[1,2] would have jobmanager-d0cabsrv[1,2]. jobmanager-pbs would become a symlink to the appropriate jobmanager-d0cabsrv[1,2]. This would be in addition to work being performed by FermiGrid on high availability gatekeepers.
-  Open up additional slots for opportunistic use on both clusters. (cabsrv1 1st)
- Ideally make all pbs job slots available for opportunistic use by Grid.
- Research/develop automatic “eviction” policy for pbs when slot is needed by dzero (as with condor).
- (requires D0 CPB input, should go through  QL. This is in-kind for D0 opportunistic use)
-  Review Globus Errors: Adam’s text table of held reasons.
-  Conduct review(s) covering the following topics, plan for future work:
- Consider extra queues to segregate D0 MC production (J. Snow) from other VO opportunistic usage. Consider adding extra roles in dzero VO (such as /Role=mc or /Role=monte-carlo).
- Investigate if special-purpose d0farm nodes still needed. FEF, FGS, Grid Dev. If so, do the coding that should have been done long ago to report them accurately.
- Review the layout of D0 resources advertising to ReSS, in order to see if it can be done in a more uniform way as opposed to the special-case hackery for CAB that is done now. (related to above 2 bullets). Do we want to have VOMS:/dzero/users access rules?
- Can d0cabsrv1 worker nodes be increased to have 10GB of scratch (like d0cabsrv2 worker nodes already have) instead of 4GB?
- Eliminate specialized queues in favor of priorities to allow greater CPU utilization.
- Load-balancing/combining CAB1/CAB2 (avoid user’s manually load balancing)

- 1/22/2009

- Keith Chadwick

## 2.3 Operations-Driven Projects

- 2.3.1 Monitoring
- Workshop to assess what we all have now, where our gaps are, what would be most cost-effective to address
- Can we “see” above in real-time? Collect what we all have, defined requirements (w/i resources available), and execute.
- Can we trace jobs “up” as well as down? Enhance existing script to automate batch job to grid job “drill-up”?
- 2.3.2 Issues
- Address “stuck state” issue affecting both Data and MC Production – PM has some examples. Update?
- Large job issue from Mike? (RDK to research what this was. Was it 2+ GB memory use? If so, add memory to few machines to create a small “BIG MEMORY” pool?)
- 2.3.3 Features
- Add state at queuing node (from Phase 1 work). Waiting on Condor dev. PM following up on this. GG to try to push this.
- FWD Load Balancing: Distribute jobs “evenly” across FWD... however easiest to do or approximate.
- 2.3.4 Processes
- Enable REX/Ops to deploy new Condor. In Phase 2, but Lower Priority. Condor deployments for bug fixes coming up.
- Revisit capacity/config 2/year? Continuous Service Improvement Plan – RDK towards end of Phase 2.
- 2.3.5 Phase 1 Follow-up
- Enable auto-update of gridmap files on Queuing nodes. Enable monitoring on Queuing nodes. AL Partly done.
- Lessons Learned from recent ops experience: <to be discussed> (RDK revive list, reconsider towards end of Phase 2)

## Phase 2 Deferred Work List (1)Do not do until proven necessary and worthwhile

- Phase 1 Follow-up Work
- Uniform OS’s: Upgrade FWD1-3 and QUE1 to latest SLF 4.0, same as FWD4-5.
- Only a minor OS version difference. Wait until this is needed to avoid another disruption and risk
- 2.2.1 Decoupling
- @ SAM station... no proven value after Station bug fix?  DEFER
- @ Durable Storage... does decoupling this have as much value now?  DEFER
- Virtualization & FWD4 (FWD5 was used for decoupling, like to retire)  DEFER
- Estimate for FEF readiness to deploy virtualization? JA: Months away... (starting from 0, using Virtual Iron)
- 2.2.2 Queue Optimization for Stability, High Utilization, Reduced Effort
- Few Long Batch Jobs holding Grid Job Slots – not an issue since we split QUE/FWD  DEFER.
- 2.2.3 Resilience
- Means to implement or emulate Durable Storage fail-over? (hardwired config in FWD nodes)  DEFER
- 2.3.2 Issues
- Context Server issues? Believe these are cured by cron’d restart by RI.  DEFER

## Phase 2 Deferred Work List (2)Do not do until proven necessary and worthwhile

- FGS Recommendations
- Add slots for I/O bound jobs on worker nodes. (Defer. Not as motivated now by current job mix. May have been motivated by unusual jobs. Needs study.
- 1-2 additional slots per system to handle I/O.
- Balance dzeropro job slots across both clusters.(Defer, reconsider w/new WN)
- Move of corresponding worker nodes probably best, since it would also balance job slots.
- This mitigates what the next bullet would attempt to provide outright. Could be part of CAB config review too.
- Explore mechanisms for redundant pbs masters. (FEF. Long time scale?)
- Will allow access to the worker nodes to continue even if the “primary” pbs master is down.
- Monitor the samgrid forwarding nodes, look at the globus errors going into CAB, see if there are any patterns.
- Consider extra queues to segregate D0 MC production (J. Snow) from other VO opportunistic usage.
- Consider adding extra roles in dzero VO (such as /Role=mc or /Role=monte-carlo for instance).
- Investigate if special-purpose d0farm nodes still needed. FEF, FGS, Grid Dev
- If so, do the coding that should have been done long ago to report them accurately.
- Review the layout of D0 resources advertising to ReSS, in order to see if it can be done in a more uniform way as opposed to the special-case hackery for CAB that is done now. (related to above 2 bullets)
- Do we want to have VOMS:/dzero/users access rules?
- Can d0cabsrv1 worker nodes be increased to have 10GB of scratch (like d0cabsrv2 worker nodes already have) instead of 4GB?