# Presentation: 863634

## D0 Grid Data Production:Evaluation Mtg 2

- Version 1.1 (Post-Meeting Edition w/notes embedded)
- 11 September 2008
- Rob Kennedy and Adam Lyon
- Attending: AdamL, RobK, MikeD, KeithC, SteveT, EileenB, JoelS*, SteveS, MargaretV, JoeB*, ParagM*, QizhongL, RobertI

## Outline

- Roadmap
- Rough Work List with Priorities
- (discussion)

## Roadmap

- September 2008: Planning (verbatim from last week)
- Rob Kennedy, working with Adam Lyon, charged by Vicky White to lead effort to pursue this.
- First stage is to list, understand, and prioritize the problems and the work in progress.
- Next, develop a broad coarse-grained plan to address issues and improve the efficiency.
- Present plan to Vicky and D0 Spokespeople towards end of September 2008.
- October 2008 – December 2008: Phase 1 (In detail in later slides)
- 1. Server Expansion and Decoupling Data/MC Production at Services
- 2. Condor 7.0 Upgrade and Support
- 3. Metrics
- 4. Small Quick Wins
- Meeting: Balance the benefit and costs to get the most improvement soonest without overwhelming disruption
- EB: Note impact on FY09 plans with SAM-Grid dev sharp ramp-down. GG: 20% to 10% to 5% (just consult).
- January 2009: Re-Assess
- 1. Formally re-assess D0 Grid Data Production
- 2. Plan new work as needed...
- January/February 2009 – April-ish 2009: Phase 2 ... very rough picture
- 1. Additional work on systems as needed...
- 2. After basic Data Production goals achieved though, recommend moving on to MC Production issues

## Priorities (1 of 4)

- 1. Server Expansion and Decoupling of Data/MC Prod at Services
- Where are the nodes? Can we horse-trade to get first arrivals? Server nodes were ordered separately from WN
- Post-Meeting: Jason – expect arrival towards end of Sep, at latest Oct 6. (Qizhong and Rob both talked to Jason)
- Only one high priority customer, small count for them, so we could get servers ~ as soon as they arrive.
- Support: Transfer Queue node support from FGS to FEF
- Post-Meeting: Jason is open to discussing this. Another open topic: CAB head nodes administration to FGS?
- Make this a topic for next meeting – someone from FEF will attend.
- Logistics: Location in FCC1, FCC2, or GCC? Move of Que1 needed? or FwdN?
- Power, physical location, network subnet; Existing Fwd/Que nodes too.
- Do we care if they go to GCC? Not a 24x7 service, but critical servers for the service.
- Config: Optimal h/w and svc config for data prod and MC prod -- start planning this soon too.
- Meeting: First, get new nodes into production. Next: separate Data Prod and MC Prod. Hopefully, do both at same time.
- Then, optimize Data Prod config for Mike’s use pattern. Finally, optimize MC Prod config for Joel’s use pattern.\
- MD: Can we clone a FWD node quickly? In theory yes, but in practice it takes a few weeks. Not done often enough for anyone to become expert (but is procedure being refined each time? Are environment conditions making it hard, like OS or Python versions?)
- SAM Station(s): Possible in time frame to move station off Fwd1 to own node; 2 stations for decoupling?
- Meeting: AL: Make this happen. Coupling took Data Prod down for a service problem offsite related to MC Prod.
- Emphasize station(s) running on separate node not running other services.
- Could run 2 independent stations on same node (as long as that is all running there)... Say one for Data Prod and one for MC Prod.
- No station fail-over. Redundancy does not remove this SPOF. Less of an advantage then to put 2 stations on separate hardware.
- As with FermiGrid, consider using virtualization to enable fast cloning, simpler administration.
- Follow-up issue: sufficient spares to rapidly replace a misbehaving server node?
- (Some VDT/Condor deployment procedure and metrics work assumed done in parallel)

## Priorities (2 of 4)

- 2. Condor 7 Upgrade and Support
- Issues: Eliminates (we are told) the Periodic Expression hangs
- Communication: establish closer communications with REX/FGS/Condor Supp
- Good discussion (Steve, Keith, Rob, Adam) after the last FermiGrid Users Meeting (AL to post notes... URLs of some useful plots included).
- Upgrade is prerequisite for working closely with Condor developers on issues
- Packaging: In past, depended on VDT packaging by Grid/OSG group. Major dependency. Avoidable?
- Requires root to install? ST: May-be not, but very hard to do and not recommended.
- CDF Offline also facing this upgrade, opportunity for leveraging? (not as much as hoped)
- Standard install of VDT. Thin ups/upd packaging of Condor.
- Not the same Condor use-case as D0 efforts though. CDF uses less of its services.
- SAM-Grid repackaging may be involved if distribution mechanism is changed... Complicated.
- Cost/benefit of reducing this (VDT <–> SAM-Grid) packaging coupling, make upgrades less complicated, now in order to save on less SAM-Grid-specific VDT/Condor repackaging by developers later?
- Consensus: Do recommend separate VDT and Condor installations to allow Condor to upgrade independently and more quickly.

## Priorities (3 of 4)

- 3. Metrics: See list in last week’s slides
- (Basics have to be done. This topic refers to a much more thorough treatment of metrics.)
- Document inputs and definitions, relate to concepts
- Automate gathering of data and presentation, where not already done
- Where historical data repository, put into usable format and automate input of new data
- Set goals for each (arbitrary if they must be) to set expectations
- Tie to the overall sub-goals with view towards capacity planning
- Tie into monitoring and operations and get experts online asap when something happens for specific issues
- Emphasize sustainability and long-term view of system as a service
- Mike has/will have his #s on web, will show and discuss next week.
- Make this a topic for next meeting

## Priorities (4 of 4)

- 4. Small Quick Wins: Scale= 1-2 FTE-weeks from 1-2 people
- Preliminary planning to allow prioritization/selection to be done, then...
- Select and execute 1 or 2 from list with best benefit/cost:
- a. Reliable Status Info returned by SAM-Grid *** big problem, easy fix or not? Pursue planning ASAP.
- RDK setting up a meeting for Monday late afternoon
- b. JobID tracking - try out GG's procedure. Simplification? More automated approach later? Does not require root access. Alternative with Steve Timm’s approach (more work).
- Everyone should try the procedure first to see whether it satisfies all requirements, satisfies enough of them for now, or whether some work could be done to simplify its use.
- Root access on CAB is not required to use the procedure, but user access to CAB is required.
- c. Slow FWD node to CAB Job Transition Investigation - can see when slow, but upstream or downstream cause?
- d. Intelligent selection of Fwd nodes from a pool of candidates (taking into account which are busy) or some mechanism which avoids sending jobs to a “full” Fwd node while another Fwd is available.
- e. Auto-restart on reboot (or make as restartable as possible). Integrating D0Farm occupancy into ops reporting.
- Robert I is working on this now.
- (a’?). Why are nodes rebooting? MEV: have root causes been done? No, just not realizing the persistent high frequency of the crashes and their not being related to other events. ST: high rate overall associated with the model in question.
- Depth of spare pool, to allow pre-emptive replacement? Revisited topic of virtualization and cloning.
- Virtualization also may help separate system root access (which FEF alone should have in principle) and application root access (which application administrators should have, and some of their log messages are in “root”-only files, for example).
- (a’?). Is “rapid replacement” (using virtualization and spares) a candidate for a task chain to address some of the stability issues? Not pure HA, but still greatly reduces downtime due to a hardware failure, esp. w/ flaky end-of-life hardware.
- Work being done... Evaluating setup of an automated quantitative report of occupancy averaged over a day or week.

## Backup Slides

- ... not used in the meeting ...

## Metrics List

- Resource Utilization
- Issue: We want to maximize resource utilization, mitigate SPOF, understand system capacity
- _% Job slots occupied_ from point of view of job queuing system
- _% CPU used_ for assigned job slots: includes data handling and DB access wait time outside of our scope.
- _MEvts/day successfully processed_: another top-level metric. Compare to data-taking rate.
- Effort to Run Data Production
- Issue: Reduce effort to operate
- _Mean time between touches_: how often does coordinator have to interact with system just to keep job queues full, assuming no unusual errors in the system.
- _Hours spent per week_: Launching jobs, working on error recovery, debugging jobs, etc. ESTIMATE....
- Metrics to Quantify Data Production Service Quality (what can we measure?)
- _First-pass success rate_: Fraction of jobs succeeding on first try
- May or may not be able to disentangle user executable failure from this
- _N-pass success rate_: Fraction of jobs succeeding after N retries – do jobs eventually succeed?
- _Mean tries to success_: Average number of tries until job succeeds – quantifies “rework” effort

## Issues List (p.1/4)

- 1) _Unreliable state information returned by SAM-Grid_: SAM-Grid under some circumstances does not return correct state information for jobs. Fixing this may entail adding some logic to SAM-Grid.
- 2) _Cleanup of globus-job-managers on forwarding nodes, a.k.a. “stale jobs”_: The globus job managers on the forwarding nodes are sometimes left running long after the jobs have actually terminated. This eventually blocks new jobs from starting.
- 3) _Scriptrunner on samgrid needs to be controlled, a.k.a. the Periodic Expression problem_: This is now locking us out of all operation for ~1 hour each day. This is due to a feature in Condor 6 which we do not use, but which cannot be fully disabled either. Developers say this is fixed in Condor 7, but this has not been proven yet.
- 4) _CORBA communication problems with SAM station_: The actual source of all CORBA problems is hard to pin down, but at least some of them seem to be associated with heavy load on samgfwd01 where the SAM station runs. Since the forwarding nodes are prone to getting bogged down at times, the SAM station needs to be moved to a separate node.
- 5) _Intelligent job matching to forwarding nodes_: SAM-Grid appears to assign jobs to the forwarding nodes at random without regard to the current load on the forwarding nodes. It will assign jobs to a forwarding node that has reached CurMatch max even if another forwarding node has job slots available.

## Issues List (p.2/4)

- 6) _Capacity of durable location servers_: Merge jobs frequently fail due to delivery timeouts of the unmerged thumbnails. We need to examine carefully what functions the durable location servers are providing and limit activity here to production operations. Note that when we stop running Recocert as part of the merge this problem will worsen.
- 7) _CurMatch limit on forwarding nodes_: We need to increase this limit which probably implies adding more forwarding nodes. We would also like to have MC and data production separated on different forwarding nodes so response is more predictable.
- 8) _Job slot limit on forwarding nodes_: The current limit of 750 job slots handled by each forwarding node has to be increased.   Ideally this would be large enough that one forwarding node going down only results in slower throughput to CAB rather than a complete cutoff of half the processing slots. Could be addressed by optimizing fwd node config for data production.
- 9) _Input queues on CAB_: We have to be able to fill the input queues on CAB to their capacity of ~1000 jobs.   The configuration coupling between MC and data production that currently limits this to ~200 has to be removed. Could be addressed by optimizing fwd node config for data production.

## Issues List (p.3/4)

- 10) _32,001 Directory problem_: ***Band-aid is in place***, but we should follow up with Condor developers to communicate the scaling issue of storing job state in a file system given the need to retain job state for tens of thousands of jobs in a large production system.11) _Spiral of Death problem_: See for example reports from 19-21 July 2008. Rare, but stop all processing. We do not understand the underlying cause yet. The only known way to address this situation is to do a complete kill/cold-stop and restart of the system.
- 12) _Various Globus errors_: We have repeated episodes where a significant number of jobs lose all state information and fall into a "Held" state due to various Globus errors. These errors are usually something like "Job state file doesn't exist", "Couldn't open std out or std err", "Unspecified job manager error". Mike doesn't think we have ever clearly identified the source of these errors.   His guess is they have a common cause. The above errors tend to occur in clusters (about half a dozen showed up last night, that's what brought it to mind). They usually don't result in the job failing, but such jobs have to be tracked by hand until complete and in some cases all log information is lost.
- 13) _Automatic restart of services on reboot_: Every node in the system (samgrid, samgfwd, d0cabosg, etc) needs to be set up to automatically restart all necessary services on reboot. We have lost a lot of time when nodes reboot and services do not come back up. SAM people appear to not get any notification when some of these nodes reboot.

## Issues List (p.4/4)

- 14) _SRM needs to be cleanly isolated from the rest of the operation_: This might come about as a natural consequence of some of the other decoupling actions. A more global statement would be that we have to ensure that problems at remote sites cannot stop our local operations (especially if the problematic interaction has nothing to do with the data processing operation).
- 15) _Lack of Transparency_: No correlation between the distinct grid and PBS id’s and inadequate monitoring mean it is very difficult to track a single job through the entire grid system, especially important for debugging.
- 16) _Periods of Slow Fwd node to CAB Job transitions_: newly added
- MC-specific #*1) File Delivery bottlenecks: use of SRM at site helps*
- MC-specific 2) Redundant SAM caches needed in the field
- MC-specific 3) ReSS improvements needed, avoid problem sites,....
- MC-specific 4) Get LCG forwarding nodes up and running reliably
- MC Note) 80% eff. on CAB, but 90% on CMS – Why? Something about CAB/PBS to note here for data production expectations too?