# Presentation: 660459

##  Networking

- Shawn McKee
- University of Michigan
- DOE/NSF Review November 29, 2001

## Why Networking?

- Since the early 1980’s physicists have depended upon leading-edge networks to enable ever larger international collaborations.
- Major HEP collaborations, such as ATLAS, require rapid access to event samples from massive data stores, not all of which can be locally stored at each computational site.
- Evolving integrated applications, i.e. Data Grids, rely on seamless, transparent operation of the underlying LANs and WANs.
- Networks are among the most basic Grid building blocks.

- Shawn Mckee, UMich

- November 29, 2001

##  Hierarchical Computing Model

***Tier 1***

**Online System**

**Offline Farm,****CERN Computer Ctr ~25 TIPS**

**BNL Center**

**France**

-  **Italy**

**UK**

- Institute

- Institute

- Institute

**Institute ~0.25TIPS**

**Workstations**

**~100 MBytes/sec**

**100 - 1000 Mbits/sec**

- HPSS

**Physicists work on analysis “channels”**
**Each institute has ~10 physicists working on one or more channels**

**Physics data cache**

**~PByte/sec**

**~**_[2.5 ](slide21.xml)__[Gbits](slide21.xml)__[/sec](slide21.xml)_

**~2.5 Gbps**

***Tier****** ******0 +1***

***Tier 3***

***Tier 4***

***Tier 2***

**CERN/Outside Resource Ratio ~1:2****Tier0/(******** Tier1)/(******** Tier2)      ~1:1:1**

-  Hierarchical Computing Model

- Shawn Mckee, UMich

- November 29, 2001

## MONARC Simulations

- MONARC (**M****odels ****o****f ****N****etworked ****A****nalysis at ****R****egional ****C****entres) **has simulated Tier 0/ Tier 1/Tier 2 data processing for ATLAS.
- Networking implications: Tier 1 centers require ~ 140 Mbytes/sec to Tier 0 and ~200 Mbytes/sec to (each?) other Tier 1s, based upon 1/3 of ESD stored at each Tier 1.

- Shawn Mckee, UMich

- November 29, 2001

## TCP WAN Performance

- Mathis, *et. al*.,  Computer Communications Review v27,** 3**, July 1997, demonstrated the dependence of bandwidth on network parameters:

- BW - Bandwidth
- MSS – Max. Segment Size
- RTT – Round Trip Time
- PkLoss – Packet loss rate

**If you want to get ****90 Mbps**** via ****TCP/IP**** on a WAN link from LBL to IU you need a ****packet loss < 1.8e-6 ****!! (~70 ms RTT).**

- Shawn Mckee, UMich

- November 29, 2001

## Network Monitoring: Iperf

- We have setup testbed network monitoring using Iperf (V1.2) (S. McKee(Umich), D. Yu (BNL))
- We test both UDP (90 Mbps sending) and TCP between all combinations of our 8 testbed sites.
- Globus is used to initiate both the client and server Iperf processes.

- (http://atgrid.physics.lsa.umich.edu/~cricket/cricket/grapher.cgi)

- Shawn Mckee, UMich

- November 29, 2001

## USATLAS Grid Testbed

**UC Berkeley**
**LBNL-NERSC**

**Brookhaven**
**National**
**Laboratory**

**Indiana **
**University**

**Boston **
**University**

**Argonne**
**National**
**Laboratory**

- HPSS sites

**U Michigan**

**University of**
**Texas at**
**Arlington**

**University of**
**  ****Oklahoma**

*Prototype Tier 2s*

- Shawn Mckee, UMich

- November 29, 2001

## Testbed Network Measurements

| Site | UDP(Mbps) | TCP(Mpbs) | PkLoss (%)* | Jitter(ms) | TCP Wind, Bottleneck |
| --- | --- | --- | --- | --- | --- |
| ANL | 65.4/81.3 | 17.7/20.9 | 0.24/0.03 | 1.1/0.1 | 2 M, 100 |
| BNL | 66.4/83.5 | 10.5/13.6 | 0.51/0.19 | 1.7/0.5 | 4 M, 100 |
| BU | 63.4/78.6 | 10.8/13.4 | 0.70/0.25 | 2.4/1.27 | 128 K, 100 |
| IU | 35.8/40.3 | 26.7/35.0 | 0.31/0.048 | 0.9/0.55 | 2 M, 45 |
| LBL | 70.4/88.4 | 15.7/20.8 | 0.16/0.014 | 1.6/0.7 | 2 M, 100 |
| OU | 72.1/90.8 | 21.5/27.8 | 0.89/0.020 | 1.7/0.4 | 2 M, 100 |
| UM | 69.7/87.3 | 27.5/36.0 | 0.26/0.018 | 1.8/0.6 | 2 M, 100 |
| UTA | 9.5 | 3.8 | 0.57 | 1.3 | 128 K, 10 |

- Shawn Mckee, UMich

- November 29, 2001

## Networking Requirements

*There is more than a simple requirement of adequate network bandwidth for USATLAS. We ****need****:*
- A set of local, regional, national and international networks able to interoperate transparently, without bottlenecks.
- Application software that works together with the network to provide high throughput and bandwidth management.
- A suite of high-level collaborative tools that will enable effective data analysis between internationally distributed collaborators.
_**The ability of USATLAS to effectively participate at the LHC is closely tied to our underlying networking infrastructure!**_

- Shawn Mckee, UMich

- November 29, 2001

## Networking as a Common Project

- A new Internet2 working group has formed from the LHC Common Projects initiative: HENP (*High Energy/Nuclear Physics*), co-chaired by Harvey Newman (CMS) and Shawn McKee (ATLAS).
- Initial meeting hosted by IU in June, kick-off meeting in Ann Arbor October 26th
- The issues this group is focusing on are the *same *that USATLAS networking needs to address.
- USATLAS gains the advantage of a greater resource pool dedicated to solving network problems, a “louder” voice in standard settings and a better chance to realize necessary networking changes.

- Shawn Mckee, UMich

- November 29, 2001

## Network Coupling to Software

- Our software and computing model will evolve as our network evolves...*both are coupled.*
- Very different computing models result from different assumptions about the capabilities of the underlying network (Distributed vs Local).
- We must be careful to keep our software “*network aware*” while we work to insure our networks will meet the needs of the computing model.

- Shawn Mckee, UMich

- November 29, 2001

## Achieving High Performance Networking

- Server and Client CPU, I/O and NIC throughput sufficient
- Must consider firmware, hard disk interfaces, bus type/capacity
- Knowledge base of hardware: *performance, tuning issues, examples*
- TCP/IP stack configuration and tuning is **Absolutely Required**
- Large windows, multiple streams
- _No_ Local infrastructure bottlenecks
- Gigabit Ethernet “clear path” between selected host pairs
- To 10 Gbps Ethernet by ~2003
- Careful Router/Switch configuration and monitoring
- Enough router “Horsepower” (CPUs, Buffer Size, Backplane BW)
*Packet Loss must be ~Zero (well below 0.1%)*
*i.e*. No “Commodity” networks (need ESNet, I2 type networks)
*End-to-end* monitoring and tracking of performance

- Shawn Mckee, UMich

- November 29, 2001

## Local Networking Infrastructure

- LANs used to lead WANs in performance, capabilities and stability, but this is no longer true.
- WANs are deploying 10 Gigabit technology compared with 1 Gigabit on leading edge LANs.
- New protocols and services are appearing on backbones (***Diffserv, IPV6, multicast***) (ESNet, I2).
- Insuring our ATLAS institutions have the required *LOCAL* level of networking infrastructure to effectively participate in ATLAS is a major challenge.

- Shawn Mckee, UMich

- November 29, 2001

## Estimating Site Costs

| Site Costs | OC3 155Mbps | OC12 622Mbps | OC48 2.4Gbps |
| --- | --- | --- | --- |
| Fiber/campus Backbone | I2 req.
(Sup. Gig) | I2 req. 
(Sup Gig) | I2 req. 
(Sup Gig) |
| Network Interface | $100/conn. (Fast Eth.) | $1K/conn. (Gigabit) | $1K/conn.
(Gigabit) |
| Routers | $15-30K | $40-80K | $60-120K |
| Telecom service Provider | Variable (~$12K/y) | Variable (~$20K/y) | Variable (~$50K/y) |
| Network connection Fee | $110K | $270K | $430K |

**Network Planning for US ATLAS Tier 2 Facilities, ***R. Gardner, G. Bernbom*** **(IU)

- Shawn Mckee, UMich

- November 29, 2001

## Networking Plan of Attack

- Refine our requirements for the network
- Survey existing work and standards
- Estimate likely developments in networking and their timescales
- Focus on gaps between expectations and needs
- Adapt existing work for US ATLAS
- Provide clear, compelling cases to funding agencies about the critical importance of the network

- Shawn Mckee, UMich

- November 29, 2001

## Network Efforts

- Survey of current/future network related efforts
- Determine and document US ATLAS network requirements
- Problem Isolation (Finger pointing tools)
- Protocols (Achieving high bandwidth and reliable connections)
- Network testbed (implementation, Grid testbed upgrades)

- Services (QoS, Multicast, Encryption, Security)
- Network configuration examples and recommendations
- End-to-end knowledgebase
- Monitoring for both prediction and fault detection
- Liaison to network related efforts and funding agencies

- Shawn Mckee, UMich

- November 29, 2001

## Network Related FTEs/Costs

- Network related efforts to leverage and adapt existing efforts for ATLAS

- Shawn Mckee, UMich

- November 29, 2001

## Support for Networking?

- Traditionally, DOE and NSF have provided University networking support indirectly through the overhead charged to grant recipients.
- National labs have network infrastructure provided by DOE, but not at the level we are finding we require.
- Unlike networking, computing for HEP has never been considered as simply infrastructure.
- The Grid is blurring the boundaries of computing and the network is taking on a much more *significant, fundamental* role in HEP computing.
- It will be necessary for funding agencies to recognize the fundamental role the network plays in our computing model and to support it **directly.**

- Shawn Mckee, UMich

- November 29, 2001

## What can we Conclude?

- Networks will be vital to the success of our USATLAS efforts.
- Network technologies and services are evolving requiring us to test and develop with current networks while planning for the future.
- We must raise and maintain awareness of networking issues for our collaborators, network providers and funding agencies.
- We must clearly present network issues to the funding agencies to get the required support.
- We need to determine what deficiencies exist in network infrastructure, services and support and work to insure those gaps are closed *before* they adversely impact our program.

- Shawn Mckee, UMich

- November 29, 2001

## References

- US ATLAS Facilities Plan
- _[http://www.usatlas.bnl.gov/computing/mgmt/dit/](http://www.usatlas.bnl.gov/computing/mgmt/dit/)_
- MONARC
- _[http://monarc.web.cern.ch/MONARC/](http://monarc.web.cern.ch/MONARC/)_
- HENP Working Group
- _[http://www.usatlas.bnl.gov/computing/mgmt/lhccp/henpnet/](http://www.usatlas.bnl.gov/computing/mgmt/lhccp/henpnet/)_
- Iperf monitoring page
- _[http://atgrid.physics.lsa.umich.edu/~cricket/cricket/grapher.cgi](http://atgrid.physics.lsa.umich.edu/~cricket/cricket/grapher.cgi)_

- Shawn Mckee, UMich

- November 29, 2001

## Recommended BW for the US-CERN Link: TAN-WG 

- From the Transatlantic Networking Committee  (TAN) report

- Shawn Mckee, UMich

- November 29, 2001

## Network FTE Breakdown

|  | 2002 | 2003 | 2004 | 2005 | 2006 |
| --- | --- | --- | --- | --- | --- |
| Survey | 0.25 | 0.25 | 0.25 | 0.25 | 0.25 |
| Requirements | 0.5/0.25 | 0.5/0.25 | 0.25 | 0.25 | 0.25 |
| Protocols | 0.25 | 0.25  | 0.25 | 0.25 |  |
| Services | 0.25 | 0.5/0.25 | 0.75/0.25 | 0.5/0.25 | 0.5/0.25 |
| Configuration | 0.25 | 0.5/0.25 | 0.5/0.25 | 0.5/0.25 | 0.5/0.25 |
| Testbed | 0.25/0.25 | 0.5 | 0.5 | 0.5 |  |
| Monitoring | 0.25/0.25 | 0.25/0.25 | 0.25/0.25 | 0.5/0.25 | 0.5/0.25 |
| End-to-End KB | 0.25 | 0.5/0.25 | 0.5/0.5 | 0.5/0.5 | 0.5/0.5 |
| Problem Isolation | 0.25 | 0.5 | 0.5 | 0.5 | 0.5 |
| Liaison | 0.25/0.25 | 0.25/0.25 | 0.25/0.25 | 0.25/0.25 | 0.25/0.25 |

- Shawn Mckee, UMich

- November 29, 2001

## Network K$ Breakdown

|  | 2002 | 2003 | 2004 | 2005 | 2006 |
| --- | --- | --- | --- | --- | --- |
| Survey | 4 | 4 | 4 | 2 | 2 |
| Requirements | 4 | 4 | 4 | 4 | 4 |
| Protocols | 5 | 5 | 10 | 5 |  |
| Services | 5 | 15 | 20 | 20 | 40 |
| Configuration | 5 | 10 | 10 | 15 | 20 |
| Testbed | 60 | 75 | 120 | 30 |  |
| Monitoring | 12 | 12 | 12 | 25 | 25 |
| End-to-End KB | 10 | 20 | 20 | 20 | 20 |
| Problem Isolation | 4 | 5 | 6 | 8 | 6 |
| Liaison | 7 | 7 | 8 | 8 | 8 |

- Shawn Mckee, UMich

- November 29, 2001