# Presentation: 938566

## D0 Grid Data Production Initiative:Coordination Mtg 10

- Version 1.1 (post-meeting edition w/notes)
- 13 November 2008
- Rob Kennedy and Adam Lyon
- Attending: MEV, RDK, EB, MD, GG, RI, PM,
- QL, AL, JS*, JB*

## Outline

- Summary and News
- Deployment “Feature List”
- No change since last week... tweak in meeting
- Task Status (4 slides)
- Focus on individual task status, what is needed
- Deployment 1 Plan
- Focus on overall schedule, task order

## Summary and News

- Summary
- Initiative Deployment 1 Planning Mtg held Monday
- Jobs successfully run through FWD4, FWD5, QUE2
- Reco succeeded, MC being tested (1 job succeeded so far)
- Merge on QUE2 to FWD3
- Deployment scheduled for today through Monday. Running about 3 days behind this, but no roadblocks
- OpenFileMax change may require a reboot from sys admin point of view.
- News and Notes:
- D0 Collaboration Meeting this week
- ITSM all-day Workshops last Thu-Fri, this Tue-Fri, next Tue-Fri
- Rob K. co-leads series next week, will not be here. AL will host Coord Mtg.
- Joe B. vacation next Mon-Tue

## Current Deployment “Feature” Lists

- Deployment 1: Split Data/MC Production Services (NO CHANGE)
- Time frame: _November 13-17_, with 1 week+ observation before holidays
- 1. Config: Basic Splitting of Fwd,Que Services between Data and MC Production with 2 Fwd nodes assigned to each, plus 1 Fwd dedicated to all Merging
- 2. Fwd4 deployed (w/o virtualization)
- 3. Fwd5 deployed
- 4. Que2 deployed, with client software to enable parallel use of 2 QUE nodes
- 5. New SAM Station (moved off of FWD1)
- 6. Condor 7 via “new” 1.10.1m official release from UWisc
- 7. FileMax increase on all Fwd nodes to handle large nJob actions
- 8. D0Runjob Upgrade for Data Production: Prerequisite for deploying new SAM-Grid release
- Deployment 2: Optimize Data and MC Production Configurations
- Time frame: _December 8-10_, with 1 week+ observation before holidays
- 1. Config: Optimize Configurations separately for Data and MC Production, especially to increase Data Production “queue” length
- 2. New SAM-Grid Release with support for new Job status value at Queuing node
- 3. Clean-up of any items left off of Nov. deployment like perhaps OS sync’ing/upgrades.

## Task Status (1 of 4) (Red = critical tasks, Green = done, Blue = in progress, Yellow = added notes)

- 1.1.1	Forwarding Node 4 (Fwd4)
- <Snip some completed tasks>
- 1.1.1.15	Fwd4: Few Jobs OpenFileMax=As-Is Single Job Test	   AL	JB		Mon 11/3/08	Mon 11/10/08	6d
- 1.1.1.9	Fwd4: Few Jobs OpenFileMax=As-Is Large-Scale Tests AL	"JB,MD,JS"	Tue 11/11/08	Wed 11/12/08	2d
- 1.1.1.14	Fwd4: Increase OpenFileMax to 16k			    AL	FEF		Tue 11/11/08	Wed 11/12/08	2d
- 1.1.1.10	Fwd4: Pre-Deployment OpenFileMax=16k Large-Scale Test	AL"JS,MD,JB"Thu 11/13/08Fri 11/14/08		2d
- 1.1.1.11	Milestone: Fwd4 Ready to Deploy				    AL			Fri 11/14/08	Fri 11/14/08		0d
- 1.1.2	Forwarding Node 5 (Fwd5)
- <Snip some completed tasks>
- 1.1.2.11	Fwd5: Few Jobs OpenFileMax=As-Is Single Job Test	     AL	JB		Mon 11/3/08	Mon 11/10/08	6d
- 1.1.2.7	Fwd5: Few Jobs OpenFileMax=As-Is Large-Scale Tests  AL	"JB,MD,JS"	Tue 11/11/08	Wed 11/12/08	2d
- 1.1.2.12	Fwd5: Increase OpenFileMax to 16k			     AL	FEF		Tue 11/11/08	Wed 11/12/08	2d
- 1.1.2.8	Fwd5: Pre-Deployment OpenFileMax=16k Large-Scale Test	AL"JS,MD,JB"Thu 11/13/08Fri 11/14/08		2d
- 1.1.2.9	Milestone: Fwd5 Ready to Deploy				     AL			Fri 11/14/08	Fri 11/14/08		0d
- 1.1.8	FWD and QUE Packaging with Version-Based Umbrella Product
- <Snip some completed tasks>
- 1.1.8.6	Umbrella Product: Update FWD Installation Procedure	AL	JB	Wed 11/19/08	Thu 11/20/08	2d
- Notes: RDK – will enter the monitoring related JIRA tasks into schedule as well, not critical path IMHO since can be done in parallel with testing functionality.
- Ganglia plots, NGOP alarms/monitoring.

## Deployment 1 Configuration(adapted from Oct 6 proposal, tweaked in meeting)

- Reco
- FWD1: 1250 (now 750)
- FWD5: 1250
- MC, MC Merge
- FWD2: 1250 (now 750)
- FWD4: 1250
- Reco Merge
- FWD3: 750/300 grid each
- QUE1:	Reco, Reco Merge – keep here to maintain history
- QUE2:	MC, MC Merge
- SAM Station:	All
- Jim Client: can submit to QUE1 or QUE2 depending on qualifier

## Task Status (2 of 4) (Red = critical tasks, Green = done, Blue = in progress, Yellow = added notes)

- 1.1.8	FWD and QUE Packaging with Version-Based Umbrella Product
- 1.1.8.10	Umbrella Product: Update QUE Installation Procedure		AL	JB	Wed 11/19/08Thu 11/20/08	2d
- 1.1.8.13	Umbrella Product: FWD and QUE Installation Proc. archived	AL	REX	Fri 11/21/08	Mon 11/24/08	2d
- 1.1.3	Queuing Node 2 (Que2)
- <Snip some completed tasks>
- 1.1.3.10	Que2: Jim_Client 2-QUE Support: Client Deployment		AL	REX	Wed 11/5/08		Wed 11/5/08	1d
- 1.1.3.8	Que2: Regression Test w/1-QUE Client				AL	JB	Thu 11/6/08		Fri 11/7/08	2d
- 1.1.3.9	Que2: Integration Test w/2-QUE Client				AL	JB	Mon 11/10/08	Fri 11/14/08	5d
- 1.1.3.11	Milestone: Que2 Ready to Deploy					AL		Fri 11/14/08		Fri 11/14/08	0d
- 1.1.5	New Distinct Sam Station
- <Snip some completed tasks>
- 1.1.5.4	SAM Station: Install and Setup Station				AL	RI	Thu 11/6/08	Thu 11/6/08		1d
- 1.1.5.10	SAM Station: Request and Install SRM-related certs		AL	RI	Fri 11/7/08	Wed 11/12/08	4d
- 1.1.5.5	SAM Station: Pre-Deployment Test (MOOT)			AL	RI	Thu 11/13/08	Thu 11/13/08		1d
- 1.1.5.6	SAM Station: Deactivate old station, Activate new station	AL	AL	Fri 11/14/08	Fri 11/14/08		1d
- 1.1.5.7	Milestone: SAM Station Ready to Deploy			AL		Fri 11/14/08	Fri 11/14/08		0d
- Notes: RDK – will enter the monitoring related JIRA tasks into schedule as well, not critical path IMHO since can be done in parallel with testing functionality.
- Ganglia plots, NGOP alarms/monitoring.
- JIRA “Figure out what to do with SRMs” contains “Request and Install SRM-related certs”

## Deployment 1 Tasks (3 of 4) (Red = critical tasks, Green = done, Blue = in progress, Yellow = added notes)

- 1.1.6	Deployment Stage 1
- 1.1.6.1	Deployment 1: Plan: Split Data/MC Production Svcs	AL	ALL	Mon 11/10/08	Wed 11/12/08	3d
- <Pre-deployment Testing in progress>
- 1.1.6.2	Deployment 1: Execute					AL	REX	Wed 11/19/08	Thu 11/20/08		2d
- 1.1.6.3	Deployment 1: Monitor					AL	REX	Fri 11/21/08		Mon 11/24/08	2d
- 1.1.6.4	Deployment 1: Sign-off					AL	REX	Tue 11/25/08		Tue 11/25/08		1d
- 1.1.6.5	MILE 1: Deployment 1 Completed				AL		Tue 11/25/08		Tue 11/25/08		0d
- Meeting on Monday 10 November produced a schedule _**(next slide),**_ need more fall-back planning
_**17 November 2008**_ end of day _***was***_ the drop-dead date to be deployed, allows 7 days to observe.
- The soonest we can deployment IMHO is _Wed-Thu 19-20 November_, allows 4 days to observe.
- We cannot deploy later than 20 November (Thursday)... no deploy on Friday or holiday week.
- Past Notes:
- SAM Station – prefer downtime for switch. Prefer to drain-off for install (and un-install if it comes to that)
- New Condor is in this deployment too. THIS is a major risk. (Will be on FWD1-3,QUE1 as well)
- Full wipe/re-install versus marginal application-only install. Treat existing nodes differently than new nodes?
- How does umbrella product installation interact with existing products on old nodes (assuming not a full wipe/re-install)?
- Take working nodes and possibly making them non-working and without fall-back versions.
- Alternative, upgrade old nodes in Dec... perhaps via rolling upgrades.
- (new) Marginal application-only install on existing nodes in Nov. and a full wipe/re-install in Dec.? Appliance view.

## Deployment 1 Schedule(include more fall-backs and specific personnel assignments)

- Mon 10 Nov 2008
- Finish FWD4,5 tests, Finish QUE2 tests
- Tues 11 Nov 2008
- Finish FWD4,5 tests, Finish QUE2 tests
- Jim Client tweak (per JS,MD) - unnecessary
- SAM/SRM certs received (12 Nov)
- Forewarn of SAM/SRM changes coming
- No responses...
- Wed 12 Nov 2008
- Jim Client redeploy - unnecessary
- OpenFileMax change on FWD4,5
- Large-scale FWD4,5 tests
- 2-queue node tests: submit to both w/qual
- Thu 13 Nov 2008
- SAM/SRM Installed. No test until disks defined
- Rewarn of SAM/SRM changes coming
- Follow-up: OpenFileMax time, OS update time.
- Fri 14 Nov 2008
- SAM Station move off of FWD1
- Deactivate SAM station on FWD1
- Activate new SAM station
- Downtime-less change-over
- Fall-back: use old config, same way
- Sat/Sun 15-16 Nov 2008
- Test FWD4,5, QUE2 w/ new OpenFileMax
- Drain off SRMs connected to old SAM station
- Observe new SAM station in production

- Mon 17 – _Thu 20 Nov 2008_
- NB: GCC Power Out sometime this week?
- _Alt Resources to start this on Mon?_
- _Mon 17 Nov: : Need a Depl Plan Mtg 2, 9am_
- Tue 18 Nov: FWD2 certs expire.
- FWD1-3 _wipe/install_ via umbrella package; Increase OpenFileMax
- FWD2 first, then FWD3, then...
- _Do FWD1 __**last**__ in case SRMs still connected_
- QUE1 “~wipe”/install via umbrella package (QUE1 has brokering, web page)
- _AL: Be careful NOT to wipe state of old jobs..._
- Brokering, Web page should not be touched? We have not fully tested the new deployment of these.
- Prepare for sign off Tues 25 Nov.
- Thu 20 Nov 2008: Deadline for SRM switch
- Validate the configuration, _finish deployment_.
- Fri 21 – Mon 24 Nov 2008
- Observe system in production
- Post-Deployment Work: move context server?
- Tues 25 Nov 2008
- Sign-off on D0 Grid Production System

## Task Status (4 of 4) (Red = critical tasks, Green = done, Blue = in progress, Yellow = added notes)

- 1.3.1	SAM-Grid Job Status Info
- 1.3.1.1	"Use ""Same"" Proxy for Gridftps"		GG	PM	done by Fri 11/21/08	developed, needs testing
- 1.3.1.2	New Job Status Value at QUE Node	GG	PM	done by Fri 11/21/08	started
- Work needs to restart in order to be ready for December deployment... re-schedule Monday
- 1.3.1.6	Pre-deployment test of new SAM-Grid Release			AL	REX		Mon 12/1/08	Fri 12/5/08	5d
- 1.3.1.4	Upgrade D0Runjob version used by Data Production		AL	"MD,AL"	Thu 10/30/08	Fri 10/31/08	2d
- 1.3.1.5	Milestone: SAM-Grid Release Deployable for Data ProductionAL	REX		Fri 12/5/08	Fri 12/5/08	0d
- 1.3.2	Slow Fwd-CAB Job Transition
- Note: FileMax change requires a schedd restart (ST). Work into deployment plans.
- 1.3.3	Improved H/w Uptime
- 1.4	Metrics
- nSubmissions plot for Sep ’08 Mike?
- Post-Deployment Follow-up
- Archiving Installation Instructions with all note-worthy comments in JIRA integrated
- Lists of new-machine certs and new-operator authorization: location, process, what uses it, manual or auto updated
- Cost-benefit: push FWD, QUE nodes to be appliances
- spec’d from OS (including OpenFileMax) to applications to grid system configuration
- rapid wipe and re-install
- Notes: At mercy of off-site gridmap updates... need to use the existing automated system to keep all in sync
- Also: no remote site has new VDT (which has new VOMS)
- No installation instructions for durable locations server. Considering for Phase 2 of Initiative.