# Presentation: 810274

## Run - II  Networks

- Run-II Computing Review
- 9/13/04
- Phil DeMar
- Networks Section Head

## LAN Overview

- Centralized management of the facility network
- Includes design, installation, & operation of Run-II networks
- LAN architecture based on work groups
- Run-II computing resources reside within its own LAN
- General philosophies on network design
- “Over-provisioning” capacity reduces problems
- Homogeneity & simplicity help management & operations
- Technical directions:
- Capacious switch fabric w/ high port density
- 10 GE backbone links; 1000B-T for host systems

## FNAL Networks: A Work Group Model

**FNAL Networks: ****A Work Group Model**

**Core**
**Network**

**Off Site**

**On Site**

**Fast routing **
**& switching**
**of workgroup**
**LANs, each with a 1000Mb/s link into**
** ****the core network**

**OC12 link  {622 Mb/s}**

**Workgroup**
**internal LANs have switched 10/100 Mb/s desktop links & **
**100/1000 Mb/s server links.**

_**Run-II work groups:**_
**  ****10GE & (n x 1GE) backbone links **
**1000 Mb/s server & farms support**
** ****switched 100 Mb/s desktop support  **

_**StarLight dark fiber:**_
**10GE & (2 x 1GE) channels **
**10 GE links to CDF, CMS, and (soon) D0 **
**Intended use: R & D**

**CMS**

**MINOS**

**Mini-**
**Boone**

**SDSS**

## CDF Run-II Network

## D0 Run-II Network

## FY05 Planned LAN Upgrades

- Core Network Upgrades:
- 10 Gb/s backbone links, including border router
-  Wireless LAN upgrade to 802.11b (54Mb/s)
- CDF Run-II Network:
- CAF expansion – 160 workers; 20 servers; 10GE uplink
- Off-line LAN upgrade – 10 GE uplink; GE for desktops
- D0 Run-II Network:
- CAS switch fabric upgrade, incl. addtl 1000B-T ports
- 10 GE backbone, incl. HDCF & StarLight
- Farms expansions – 280 CAB & 80 Recon nodes
- On-line switch fabric upgrade & 1000B-T ports

## Current WAN Status

- ESnet-funded off-site link carries production traffic
- Currently, OC12 (622 Mb/s)
- CMS robust data challenge: 10,000 Mb/s
- FNAL-funded StarLight dark fiber
- Initial configuration: one 10 GE & two 1 GE channels
- Theoretical capacity = > Thirty-three 10GE channels
- Intended uses:
- WAN network R & D projects
- Overflow link for production WAN traffic
- Redundancy for ESnet link

## ESnet Off-Site link utilization

- Current link is “stressed”:
- Carries all production traffic
- Only an OC12 (622 Mb/s)
- Traffic doubles every year
- Outbound:  blue = average; 	        magenta = peak
- Inbound:    green = average; 	       dark green = peak
- Traffic mostly Run-II data
- CMS traffic beginning to appear
- ESnet upgrade of existing link is problematic

## ESnet Networking Directions

- Three-tiered network architecture
- Production (what we have today...)
- High-impact (very large scale data movement)
- Research (next generation networking & apps)
- Cost-sharing model for site connectivity
- Labs expected to contribute to site link upgrades
- Metropolitan Area Network (MAN) initiative
- Metro fiber rings supporting multiple Labs
- ESnet proposing MANs for Bay Area & Chicago
- Bay Area MAN partially funded in FY04
- Chicago MAN not funded in FY04...

## ESnet Chicago MAN proposal

## FNAL Starlight dark fiber project

- Dark fiber to Starlight in place
- StarLight = international optical network exchange pt.
- Link optics w/ DWDM gear soon
- Capable of multiple λs
- Each λ can carry 2-10 Gb/s
- But expensive, (O) $100k/ 10GE λ
- Initially, one 10GE & one 2 GE λs
- Currently: 1 Gb/s ‘til DWDM turn-on
- Separate network infrastructure from production LAN path

## Off-Site Link Upgrade Planning

- Primary emphasis is to get ESnet MAN deployed
- Would provide both production & high impact paths
- Pursuing fiber availability options for missing segment
- In the end, the MAN needs DOE approval & funding
- Alternatives involve using StarLight link for production traffic
- Carrying overflow traffic would be one possible mode
- Keeping ESnet as our service provider remains long term goal
- Want to avoid being in the production network WAN business

## WAN Network R & D Projects

- Primary use of StarLight link intended to be R&D
- Practical benefits:
- Environment for collaborators to develop & refine their applications for high rate data movement
- Non-production (< five 9s...) facility for testing & introducing advanced network technologies
- Can be utilized to support production network facilities
- Redundancy; overflow traffic...
- Research benefits:
- Develop services & tools for HEP research environment
- LambdaStation:  2-3 year SciDac Project to facilitate selective path forwarding across advanced WAN research networks

## StarLight Link R&D Projects

- Build it and they will come.....

| 
Project | Local
Affiliation | Collaborating
Site(s) | 
Status | 
BW Needs | Usage
Pattern |
| --- | --- | --- | --- | --- | --- |
| UKlight | CDF | UCL | Active | 4 Gb/s | periodically |
| CMS/CERN | CMS (OSG?) | CERN | Active | Up to 10 Gb/s | periodically |
| CMS/
LambdaStation | CMS | Cal Tech/UCSD | Start  10/2004 | Up to 10 Gb/s | sustained |
| Toronto HEP | CDF | U. Toronto | Inactive | {undefined} | {undefined} |
| SC2004 BW
Challenge | CD | SC2004 | 11/06/2004 | Up to 10 Gb/s | One-time |
| Vanderbilt | OSG(?) | Vanderbilt | Pending | {undefined} | {undefined} |
| WestGrid (Ca) | D0 | Simon Fraser U | Pending | 4 Gb/s | sustained |
| LHC Tier 1 | CMS | BNL | Pending | < 622 Mb/s | {undefined} |
| UltraLight | CMS | Cal Tech, UF, others | Pending | Up to 10 Gb/s | {undefined} |

## Network support at FNAL -  Effort

- Network FTE level has been static for past decade
- Greater complexity & added services = increased demands on personnel resources
- Continually working toward more efficient, automated procedures
- ~1.5 FTE to upgrade/service each Run-II network/year
- Also a relatively static number for the past decade
- Core network, services, WAN, security, etc. efforts not included...
- Each experiment has liaison to coordinate network support...
- Emerging or significantly growing areas of effort
- Advanced WAN (StarLight...) infra. & ops support
- Network R&D (LambdaStation)
- Computer security demands on network support