# Presentation: 125358

## FY08 Tactical Plan Status Report forSCF/DMS/WAN Group

- Phil DeMar
- 6/3/2008

## FY08 Tactical Plan for Wide Area Networking

- FY08 Tactical Plan for Wide Area Networking

- Relevant Strategic Plan(s):
- Strategic Plan for Wide Area Networking
- _[https://cd-docdb.fnal.gov:440/cgi-bin/ShowDocument?docid=2236](https://cd-docdb.fnal.gov:440/cgi-bin/ShowDocument?docid=2236)_
- Tactical Plan Leader: 		Phil DeMar
- Organizational Unit home: 		SCF/DMS/WAN
- Tactical Plan Goals:
- Support a highly redundant wide-area network infrastructure that provides reliable, capacious offsite access
- Implement & enhance high bandwidth end-to-end data paths
- Plan & implement activities that establish the Laboratory in a leadership position in the R&E network community
- Provide the CMS Tier-1 Center with network infrastructure optimized for high bandwidth offsite data movement

## Activities Summary: FTEs

| Level 0 Activity: | Wide Area Networking |  |  | % of FY Complete: |  | 68% |
| --- | --- | --- | --- | --- | --- | --- |
| Personnel Usage (FTEs) |  |  |  |  |  |  |
|   | Allocation |  | Actual YTD |  |   | Current |
| Tactical Plan       Level 1 Activity | FTE-yrs | FTE-mos | FTE-yrs (Ave/mo.) | FTE-mos | % Consumed YTD | FY08 Forecast |
| Advanced WAN Infrastructure |                1.3                      |                 15.7 |                  1.0 |                 11.7 |                      74% |  ~100% |
| Global Network Interfaces | 0.3 | 3.7 | 0.2 | 2.16 | 66% |  ~100%  |
| Leadership | 0.2 | 1.8 | 0.1 | 1.28 | 71% |  ~100%  |
| Management | 0.4 | 5.2 | 0.1 | 1.62 | 31% |  <100%  |
| Network Performance Improvement | 1.4 | 16.8 | 0.8 | 9.43 | 56% |  ~100%  |
| Network Investigations | 0.4 | 5.4 |  0.1 | 0.81 | 15% |  <100% |
| Network WAN Ops/Support/Maint* | 1.2 | 14.9 | 0.5   | 6.53 | 44% |  <100% |
| Network Research | 0.6 | 7.2 | 0.6 | 7.12 | 99% |  >100%  |
| Network Upgrades | 0.4 | 4.8 | 0.1 | 0.41 | 9% |  <100% |
| Network CMS WAN | 0.4 | 4.8 | 0.4 | 4.74 | 99% |  >100% |
| SciDAC Lambda Station  | 0.2 | 1.8 | 0.8 | 9.7 | 530% |  >>100% |
| Total | 6.8 | 82.1 | 4.7 | 55.5 | 68% |  100% |

- *  Listed under Site Networking Level-0 activity

## FTE Expenditures Notes:

- Under-spending on Network Upgrades & Network Investigations
- Product of furloughs & minimal upgrade funding for FY
- Both activities expected to pick up in remaining four months
- Higher-than-budgeted SciDAC Lambda Station effort due to:
- WAN technology shift created unanticipated demand
- Desire to spend down allocated funding by end-of-year
- Activity for WAN Operations, Support, & Maintenance erroneously listed under Site Networking
- Budgeted at 1.0 FTE of WAN effort & 0.2 FTE of Site Networking
- Causes Crystal Reports to understate budgeted WAN effort
- Will be fixed starting with June effort reporting
- Some adjustment to Crystal Report effort numbers needed to accurately reflect effort
- ~3/4 FTE of Advanced WAN Infrastructure mis-entered as Research

## Activities Summary: M&S (Internal Funding)

| Level 0 Activity: |  Wide Area Networking |  |  | 68% of FY Complete |
| --- | --- | --- | --- | --- |
|  |  |  |  |  |
| Operating & Equipment M&S |  |  CD Internal Funding |  |  |
|   | Operations and Equipment M&S |  |  |  |
| Tactical Plan       Level 1 Activity | FY Obligation Budget | YTD Obligations + RIPS | % Spent | Current FY08 Forecast |
| 50.02.10.05.01:    Advanced WAN Infrastructure | 47.7 | 4.6 | 10% |                50% |
| Network Upgrades | 72.5 | 45.3 | 62% |               100% |
| Management | 3.6 | 3.0 | 83%                    |               100% |
| 50.02.10.05.04.01:      Global Network Interfaces | 8.8 | 9.2 | 105% |             >100% |
| Leadership (training) | 6.0 | 0 | 0% |               100% |
| Network Performance Improvements | 3.0 | 0 | 0%                    |               100% |
| Network Investigations | 5.3 | 0 | 0%                    |               100% |
| Network Operations & Support     | 24.1 | 8.4 | 35% |               100% |
| 50.02.10.05.06.01:                   Network Research | 6.0 | 0.8 | 13% |               100% |
| Totals: | 177.0 | 71.3 | 40%                    |                 86% |
|  |   |  |  |   |

## M&S Expenditures Notes:

- Will not spend two WAN budget line items totaling $24k
- Ciena maintenance ($9k) covered under ANL contract
- DuPage Technology Park ($15k) switch - no justification right now
- Travel spending is muddled:
- Already overspent on LHC & DOE-related travel:
- Three unanticipated DOE Trusted Internet Connection (TIC) task force meetings
- Crawford & DeMar actively involved in Internet2 Joint Techs workshops
- Other travel budget line items under-spent in aggregate
- But not enough to cover anticipated travel to end of FY
- Additional Lambda Station-related travel on LS task code.
- WAN ProCard erroneously assigned to Site Networking task code (thanks, cousins...)
- Fixed, but some WAN & Site task code balances will be skewed

## Project Activity: 	Advanced WAN Infrastructure - FY08 Goals

- Project Activity: 	Advanced WAN Infrastructure - FY08 Goals

- _Complete deployment of Chicago MAN:_
- Milestones:  	MAN fully deployed in Nov., 2007
- 				CMS Tier-2 circuits migrated over to MAN channels
- Issues:		ESnet upgrading their MAN equipment this summer
- 				We may begin to build 2nd MAN hub in WH
- Risks:		Funding for 2nd Ciena hub not clear
- _Facilitate deployment & reliability of offsite end-to-end circuits:_
- Milestones:	Dynamic circuits deployed to UNL & Caltech
- 				Static circuits deployed for Duke, Tennessee, DE-KIT
- 				Upgraded circuits for ASGC & IN2P3
- Issues:		Circuit technology ahead of policy & support considerations
- Risks:		Increasing support effort needed as deployment scales & 				complexity increases.

## Project Activity: 	Advanced WAN Infrastructure  (cont.)

- Project Activity: 	Advanced WAN Infrastructure  (cont.)

-  _Develop & deploy perfSONAR monitoring for circuit infrastructure:_
- Milestones:	PingER adapted for deployment in perfSonar-PS
- 			Major contribution to perfSonar-PS distribution kit
- Issues:		Scaling perfSonar-PS deployment to T2s & T3s
- 			Deployment of European MDM perfSonar appliance
- Risks:		Support concerns as perfSonar deployment scales
-  _Develop passive network monitoring capabilities based on flow data:_
- Milestone:	Prototype FTWatch network traffic sensor deployed
- Issues:		Developing prototype into a general use service
- Risks:		Support concerns if demand for the service emerges

## Project Activity: 	Global Network Interfaces

- Project Activity: 	Global Network Interfaces

* *_*Promote & integrate the Laboratory’s WAN activities into the global R&E network community:*_
-  Milestones:	Successful demos at Internet2 Members Mtgs & SC07			Major contributions to emerging dynamic services (LS) & 				advanced network monitoring (perfSonar)
- 			Crawford & DeMar Internet2/ESCC JointTechs session 				organizers/co-chairs
- 			DeMar ESCC Chair-apparent (starts officially 10/1/08)
-  Issues:		Demos consume manpower
-  Risks:		Increasing effort required as participation grows

## Project Activity: 	Network Performance Analysis

- Project Activity: 	Network Performance Analysis

-  _Develop & enhance structured WAN performance analysis methodology_
- Milestones:	Methodology developed and documented:
- 				https://plone3.fnal.gov/P0/WAN/netperf/methodology/
- Issues:		Next step: integration into content management system
- 			Long term objective: perfSonar monitoring integration
- Risks:		-----
-  _Assist collaboration sites in improving data transfer rates_
- Milestones:	Worked with 10 CMS T2 sites to improve performance
- Issues:		Varying degrees of success; up to 10x improvement
- 			Sustaining effort on remote end often difficult
- Risks:		Increasing effort required as participation & interest 					grows

## Project Activity: 	Network Integration

- Project Activity: 	Network Integration

-  _Deploy & develop IPv6 test bed_
- Milestones:	Virtual testbed partially deployed, based around FAPL
- Issues:		Modest progress; casualty of lost effort from furloughs
- 			Expect to make progress in remaining 4 months of FY
- 			Close coordination with CST required
- Risks:		IPv6 emerging in unmanaged manner
-  _Connect Laboratory to Global Lambda Integrated Facility (GLIF)_
- Milestones:	Progress is mainly PR; hurt by lost effort from furloughs
- Issues:		Expect technical progress in remaining 4 months of FY
-  _Develop WAN-emulation capability in FAPL R&D network test facility _
- Milestones:	No progress; budget casualty in this FY
- Risks:		Limits our WAN systems R&D efforts

## Service Activity: 	WAN Operations & Support

- Service Activity: 	WAN Operations & Support

- _Operate WAN infrastructure at high level of reliability_
- Milestones:	No unplanned outages or disruptions
- 				Scheduled maintenance outages caused no 						unanticipated loss or degradation of service
- Issues:		Lower than anticipated effort
- 				Still working to develop suitable uptime metrics
- Risks:		------

## Project Activity: 	Network Research & Development

- Project Activity: 	Network Research & Development

- _Research system performance limitations affecting WAN data transfers_
- Milestones:	Two papers drafted:
- “End-to-End Network/Application Performance Analysis”
- "Reducing Packet Reordering thru Interrupt Coalescing"
- 				Funded for network flow data application-awareness
- Issues:		Preparing for next round of SciDAC proposals
- Risks:		------

## Project Activity: 	WAN Infrastructure Upgrades

- Project Activity: 	WAN Infrastructure Upgrades

- _Facilitate 10GE connections as needed for offsite connections_
- Milestones:	Border & circuit routers upgraded with addtl 10GE ports
- 				10GE connections provided for new MAN channels (2)
- Issues:		Planning for WH hub for offsite connectivity
- Risks:		------
- _Upgrade backup border router to 10GE & deploy redundant circuit path_
- Milestones:	Upgrade of backup border router to 10GE in progress
- Issues:		Working on LHCOPN (CERN) secondary circuit
- 				Failover of most E2E circuits left to routed IP path
- Risks:		------

## Project Activity: 	WAN Infrastructure Upgrades (cont.)

- Project Activity: 	WAN Infrastructure Upgrades (cont.)

-  _Deploy 10GE optical tap capability on facility DMZ_
- Milestones:	Optical tap strategy replaced by spanned ports to CST
- 			DMZ 10GE switch deployed
- Issues:		Adding 10GE port capacity just starting
- 			New strategy will require CST 10GE switch
- Risks:		Path for scaling beyond 10GE capacity not clear
-  _Enhance R&D cluster (FAPL) systems capabilities_
- Milestones:	Minimal effort so far; lower priority effort hurt by furlough
- Issues:		Expect to complete planned upgrades in next 4 months
- Risks:		------

## Project Activity: 	CMS (Tier-1) WAN Support

- Project Activity: 	CMS (Tier-1) WAN Support

- Upgrade Tier-1 facility, including redundancy at work group core
- Milestones:	FY08 Tier-1 network infrastructure upgrades completed
- 				Redundant LAN architecture in place
- 				New CMS LAN weather map deployed
- Issues:		FY09 Tier-1 network planning under way
- 				Evaluating benefits of redundancy vs addtl complexity
- Risks:		Path for scaling beyond 40GE capacity not clear
- Implement direct 10GE offsite path to facility border router
- Milestones:	10GE direct connection to border router in place
- Issues:		Dynamic traffic rerouting mechanisms under discussion
- Risks:		------

## Project Activity: 	Lambda Station Project

- Project Activity: 	Lambda Station Project

- _Harden Lambda Station (LS) & integrate with dynamic circuit services_
- Milestones:	LS integrated with Internet2/ESnet DCN/Oscars services
- 				LS-driven dynamic circuits in place at UNL & Caltech
- 				Flow analysis-based FT-Watch integrated with LS
- 				Successful LS demos at Internet2 Member mtgs & SC07
- Issues:		Sudden emergence of DCN service has LS in demand!
- 				Hardening delayed by lack of available scripting effort
- 				Working on convergence path w/ BNL TeraPaths project
- Risks:		Spend-down of project in FY08 well behind schedule
- 				Ongoing support concerns if LS is widely deployed
- 				Continual cycle of demos taking toll on effort

## Service Activity: 	ESnet MAN Support & Operations

- Service Activity: 	ESnet MAN Support & Operations

_*Operate MAN infrastructure at high level of reliability*_
- Milestones:	Fully-redundant MAN in operation since 12/2007
- 				Nagios monitoring & Ciena event logging deployed
- Issues:		Separate security plan under development
- 				Completing Work-for-others reimbursement with ESnet
- 				Developing in-house optical network infrastructure 					expertise
- 				Developing dual hub architecture at FNAL & ANL
- 				Documentation & public interface need effort
- Risks:		Direction & funding for MAN beyond 10GE unclear
- 				Primitive governance limits regional use possibilities

## Tactical Plan Status Summary

- Status Summary
- M&S spending and FTE expenditures seem reasonably in line with budget allocations
- Lost furlough time hit network upgrades & investigations hard
- Some catch-up anticipated over remaining 4 months in FY
- Concerns about impact on effort to sustain global outreach gains
- Configuration & operational support for E2E circuits
- Product support for Lambda Station & perfSONAR efforts
- Leadership efforts of Crawford & DeMar in Internet2 & ESnet realms
- Lambda Station spend-down hurt by furloughs & lost personnel

- CD FY08 Tactical Plan Status