# Presentation: 576273

##                                  U.S. Department of Energy’s                                Office of Science

**Mary Anne Scott**
**Program Manager**
**Advanced Scientific Computing Research**
**Mathematical, Information, and Computational Research Division**
- _[scott@er.doe.gov](mailto:scott@er.doe.gov)_

**Connecting Scientists**
**...****to collaborators, to facilities, to datastores, to **
**computational resources, to....**
**Energy Sciences Network (ESnet)**
**November 3, 2004**
**BERAC Meeting**

## Science Drivers for Networks

- Modern, large-scale science is dependent on networks
- Data sharing
- Collaboration
- Distributed data processing
- Distributed simulation
- Data management

## Science Mission Critical Infrastructure

- ESnet is a visible and critical piece of ***DOE science infrastructure***
- If ESnet fails, tens of thousands of DOE and University users know it within minutes if not seconds
- Requires high reliability and high operational security in the network operations

## What is ESnet?

- Mission:
- Provide, interoperable, effective and reliable communications infrastructure and leading-edge network services that support missions of the Department of Energy, especially the Office of Science
- Vision:
- Provide seamless and ubiquitous access, via shared collaborative information and computational environments, to the facilities, data, and colleagues needed to accomplish their goals.
- Role:
- A component of the Office of Science infrastructure critical to the success of its research programs (funded through ASCR/MICS and managed and operated by ESnet staff at LBNL).

- Essentially all of the national data traffic supporting US science is  carried by two networks—ESnet and Internet-2/Abilene (which plays a similar role for the university community)

## What is ESnet’s user base?

- Between 10,000 and 100,000 researchers in the US (guesstimate)
- Mainly Office of Science programs—ASCR, BER, BES, FES, HEP, NP
- Also traffic for NNSA and others
- All the US national labs
- Hundreds of universities
- Hundreds of foreign institutions
**Characteristics of the user base**
- Many casual users
- Users from science disciplines that span SC interests
- Users concerned with any data intensive and computationally intensive tasks
- Collaborators distributed geographically, small to large groups

## What Does ESnet Provide?

- ESnet builds a comprehensive IP network infrastructure
- Telecommunications services ranging from T1 (1 Mb/s) to OC192 SONET (10 Gb/s) and uses these to connect core routers and sites to form the ESnet IP network
- Peering access to commercial networks
- High bandwidth access to DOE’s primary science collaborators: Research and Education institutions in the US, Europe, Asia Pacific, and elsewhere
- Comprehensive user support, including “owning” all trouble tickets involving ESnet users (including problems at the far end of an ESnet connection) until they are resolved – 24x7 coverage
- Grid middleware and collaboration services supporting collaborative science
- trust, persistence, and science oriented policy

## Why is ESnet important?

- Enables thousands of DOE, university and industry scientists and collaborators worldwide to make effective use of unique DOE research facilities and computing resources independent of time and geographic location
- Direct connections to all major DOE sites
- Access to the global Internet (managing 150,000 routes at 10 commercial peering points)
- User demand has grown by a factor of more than 10,000 since its inception in the mid 1990’s—a 100 percent increase every year since 1990
- Capabilities not available through commercial networks
- Architected to move huge amounts of data between a small number of sites
- High bandwidth peering to provide access to US, European, Asia-Pacific, and other research and education networks.

- Objective: *Support scientific research* by providing seamless
- and ubiquitous access to the facilities, data, and colleagues

**Notes:**
The way science is done has changed dramatically in the past 10 years.  Some people describe it as “Science is a team sport.”

These changes lead to an increase in scientific productivity –shorter turn around time for disseminating, assimilating and testing new ideas

Basic services – email, file transfer, remote login, distibuted file systems
Teleconferencin
Remote access to unique facilities (experiments, supercommputers, databases, installed codes)

## ESnet Connects DOE Facilities and Collaborators

- Japan

**International (high speed)**
**OC192 (10G/s optical)**
**OC48 (2.5 Gb/s optical)**
**Gigabit Ethernet (1 Gb/s)**
**OC12 ATM (622 Mb/s)**
**OC12 **
**OC3  (155 Mb/s)**
**T3  (45 Mb/s)**
**T1-T3**
**T1 (1 Mb/s)**

**Office Of Science Sponsored (22)**

**NNSA Sponsored (12)**

**Joint Sponsored (3)**

**Other Sponsored (NSF LIGO, NOAA)**

**Laboratory Sponsored (6)**

**QWEST**
**ATM**

_**42 end user sites**_

**ESnet IP**

**CA*net4**
**CERN**
**MREN**
**Netherlands**
**Russia**
**StarTap**
**Taiwan**
**  ****(ASCC)**
**	**

**ESnet core: Optical Ring and Hubs**

** ****SNV HUB**

** **

**peering points**

**MAE-E**

- Chi NAP

**Fix-W**

**PAIX-W**

**MAE-W**

**NY-NAP**

**PAIX-E**

**Euqinix**

**PNWG**

** ****SEA HUB**

**ESnet Connects DOE Facilities and Collaborators**

**hubs**

** ****SNV HUB**

**Abilene**

**Abilene**

**Abilene**

**Abilene**

## ESnet’s Peering InfrastructureConnects the DOE Community With its Collaborators

**LBNL**

**ESnet’s Peering Infrastructure****Connects the DOE Community With its Collaborators**

_**ESnet Peering**_** (connections to other networks)**

**NYC  HUBS**

**SEA HUB**

- Japan

**SNV HUB**

**MAE-W**

**FIX-W**

**PAIX-W**

**26 PEERS**

**CA*net4**
**CERN**
**MREN**
**Netherlands**
**Russia**
**StarTap**
**Taiwan**
**  ****(ASCC)**

**Abilene +**
**7 Universities**

**22 PEERS**

**MAX GPOP**

**GEANT**
** ****- Germany**
** ****- France**
** ****- Italy**
** ****- UK**
** ****- etc **
**SInet (Japan)**
**KEK**
**Japan – Russia (BINP)**

**Australia**
**CA*net4**
**Taiwan**
**  ****(TANet2)**
**Singaren**

**20 PEERS**

**3 PEERS**

**LANL**

**TECHnet**

**2 PEERS**

**39 PEERS**

**CENIC**
**SDSC**

**PNW-GPOP**

**CalREN2**

**Distributed 6TAP**
**19 Peers**

**2 PEERS**

**KDDI (Japan)**
**France**

**1 PEER**

**1 PEER**

**5 PEERS**

- ESnet provides access to all of the Internet by managing the full complement of Global Internet routes (about 150,000) at 10 general/commercial peering points + high-speed peerings w/ Abilene and the international R&E networks. *This is a lot of work, and is very visible, but provides full access for DOE.*

**ATL HUB**

**University**

**International**

**Commercial**

**Abilene**

**Abilene**

**6 PEERS**

**Abilene**

## How is ESnet Managed?

- A community endeavor
- Strategic guidance from the OSC programs
- Energy Science Network Steering Committee (ESSC)
- BER representatives are T.P. Straatsma, PNNL; Raymond McCord, ORNL
- Network operation is a shared activity with the community
- ESnet Site Coordinators Committee
- Ensures the right operational “sociology” for success
- Complex and specialized – both in the network engineering and the network management – in order to provide its services to the laboratories in an integrated support environment
- Extremely reliable in several dimensions
- Taken together these points make ESnet a unique facility supporting DOE science that is quite different from a commercial ISP or University network

## What’s a possible future?

**VISION** –
- A seamless, high-performance network infrastructure in which science applications and advanced facilities are "n-way" interconnected to terascale computing, petascale storage, and high-end visualization capabilities.
- This advanced network facilitates collaborations among researchers and interactions between researchers and experimental and computational resources,
- Science, especially large-scale science, moves to a new regime that eliminates isolation, discourages redundancy, and promotes rapid scientific progress through the interplay of theory, simulation, and experiment.

## Network and Middleware Needs of DOE Science

**Focused on science requirements driving advanced network infrastructure**
***Middleware research***
***Network research***
***Network governance model***
**Requirements for DOE science developed for a representative cross-section of the OSC scientific community**
***Climate simulation***
***Spallation Neutron Source facility***
***Macromolecular Crystallography***
***High Energy Physics experiments***
***Magnetic Fusion Energy Sciences***
***Chemical Sciences***
***Bioinformatics***

## Evolving Quantitative Science Requirements for Networks

| Science Areas	 | Today End2End Throughput | 5 years End2End Throughput | 5-10 Years End2End Throughput | Remarks |
| --- | --- | --- | --- | --- |
| High Energy Physics | 0.5 Gb/s | 100 Gb/s | 1000 Gb/s | high bulk throughput |
| Climate (Data & Computation) | 0.5 Gb/s | 160-200 Gb/s | N x 1000 Gb/s | high bulk throughput |
| SNS NanoScience | Not yet started | 1 Gb/s | 1000 Gb/s + QoS for control channel | remote control and time critical throughput |
| Fusion Energy | 0.066 Gb/s(500 MB/s burst) | 0.198 Gb/s(500MB/20 sec. burst) | N x 1000 Gb/s | time critical throughput |
| Astrophysics | 0.013 Gb/s(1 TBy/week) | N*N multicast | 1000 Gb/s | computational steering and collaborations |
| Genomics Data & Computation | 0.091 Gb/s(1 TBy/day) | 100s of users | 1000 Gb/s + QoS for control channel | high throughput and steering |

## New Strategic Directions to Address Needs of DOE Science

- 	Focused on what is needed to achieve the science driven network requirements of the previous workshop
**THE #1 DRIVER for continuing advancements in networking and middleware** – Petabyte scale experimental and simulation data systems will be increasing to exabyte scale data systems.
- Bioinformatics, Climate, LHC, etc.
- Computational systems that process or produce data continue to advance with Moore’s Law
- .....
- 	Organized by the ESSC
- Workshop Chair, Roy Whitney, JLAB
- Workshop Editors, Roy Whitney, JLAB; Larry Price, ANL

## Observed Drivers for ESnet Evolution

- Are we seeing the predictions of two years ago come true?
- Yes!

## ...what now???

**VISION** - A scalable, secure, integrated *network environment* for ultra-scale distributed science is being developed to make it possible to combine resources and expertise to address complex questions that no single institution could manage alone.
- Network Strategy
- Production network
- Base TCP/IP services; +99.9% reliable
- High-impact network
- Increments of 10 Gbps; switched lambdas (other solutions); 99% reliable
- Research network
- Interfaces with production, high-impact and other research networks; start electronic and advance towards optical switching; very flexible
- Revisit governance model
- SC-wide coordination
- Advisory Committee involvement

## Evolution of ESnet

- Upgrading ESnet to accommodate the anticipated increase from the current 100%/yr traffic growth to 300%/yr over the next 5-10 years is priority number 7 out of 20 in DOE’s “Facilities for the Future of Science – A Twenty Year Outlook”
- Based on the requirements, ESnet must address
- I.  Capable, scalable, and reliable production IP networking
- University and international collaborator connectivity
- Scalable, reliable, and high bandwidth site connectivity
- II. Network support of high-impact science
- provisioned circuits with guaranteed quality of service(e.g. dedicated bandwidth)
- III. Evolution to optical switched networks
- Partnership with UltraScienceNet
- Close collaboration with the network R&D community
- IV. Science Services to support Grids, collaboratories, etc

## New ESnet Architecture Needed to Accommodate OSC

- The essential DOE Office of Science requirements cannot be met with the current, telecom provided, hub and spoke architecture of ESnet

- The core ring has good capacity and resiliency against single point failures, but the point-to-point tail circuits are neither reliable nor scalable to the required bandwidth

**ESnet****Core**

**New York (AOA)**

**Chicago (CHI)**

**Sunnyvale (SNV)**

**Atlanta (ATL)**

**Washington, DC (DC)**

**El Paso (ELP)**

**DOE sites**

## A New ESnet Architecture

- Goals
- Fully redundant connectivity for every site
- High-speed access for every site (at least 20 Gb/s)
- Three part strategy
***Metropolitan Area Network*** (MAN) rings provide dual site connectivity and much higher site-to-core bandwidth
- A ***Science Data Network*** core for
- multiply connected MAN rings for protection against hub failure
- expanded capacity for science data
- a platform for provisioned, guaranteed bandwidth circuits
- alternate path for production IP traffic
- carrier circuit and fiber access neutral hubs
- A ***High-reliability IP core*** (e.g. the current ESnet core)

## ESnet MAN Architecture

- Site gateway router

- site equip.

- Site gateway router

**Qwest hub**

**ESnet IP core**

**ANL**

**FNAL**

**ESnet MAN Architecture**

**CERN (DOE funded link)**

- ESnet management and monitoring

- ESnet managedλ / circuit services
- tunneled through the IP backbone

- site equip.

- ESnet production IP service

- ESnet managedλ / circuit services

**StarLight**

**International peerings**

- Site LAN

- Site LAN

**ESnet SDN core**

## A New ESnet Architecture:Science Data Network + IP Core

**GEANT (Europe)**

**Asia-Pacific**

**ESnet****IP Core**

**New York****(AOA)**

**Chicago (CHI)**

**Sunnyvale****(SNV)**

**Washington,****    DC (DC)**

**El Paso (ELP)**

- DOE/OSC Labs

- New hubs

- Existing hubs

**ESnet**
**Science Data Network**
**(2****nd**** Core)**

**A New ESnet Architecture:****Science Data Network + IP Core**

- Possible new hubs

**Atlanta (ATL)**

**Metropolitan****Area****Rings**

**CERN**

**Notes:**
ESnet new architecture goals: full redundant connectivity for every site and high-speed access for every site (at least 10 Gb/s)
Two part strategy
1) MAN rings provide dual site connectivity and much higher site bandwidth
2) A second backbone will provide
multiply connected MAN rings for protection against hub failure
extra backbone capacity
a platform for provisioned, guaranteed bandwidth circuits
alternate path for production IP traffic
carrier neutral hubs

## Goals for ESnet – FY07

**DEN**

**ELP**

**ALB**

**ATL**

- MANs

- High-speed cross connects with Internet2/Abilene

- Major DOE Office of Science Sites

**Goals for ESnet – FY07**

**Japan**

**CERN**

**Europe**

**SDG**

**AsiaPac**

**SEA**

- SDN – ESnet hubs

- Qwest – ESnet hubs

**SNV**

**Europe**

- 10Gb/s
- 30Bg/s 40Gb/s

**Japan**

**CHI**

- High-impact science core

- Lab supplied

- Major international

- 2.5 Gbs10 Gbs

- Future phases

- Production IP ESnet core

**DC**

**Japan**

**NYC**

## Conclusions

- ESnet is an infrastructure that is critical to DOE’s science mission
- Focused on the Office of Science Labs, but serves many other parts of DOE
- ESnet is working hard to meet the current and future networking need of DOE mission science in several ways:
- Evolving a new high speed, high reliability, leveraged architecture
- Championing several new initiatives which will keep ESnet’s contributions relevant to the needs of our community
- Grid middleware services for large numbers of users are hard – but they can be provided if careful attention is paid to scaling

## Where do you come in?

- Early identification of requirements
- Evolving programs
- New facilities
- Interaction with BER representatives on ESSC
- Participation in management activities

## Additional Information

## Planning Workshops

**High Performance Network Planning Workshop, August 2002**
- _[http://](http://www.doecollaboratory.org/meetings/hpnpw)__[www.doecollaboratory.org/meetings/hpnpw](http://www.doecollaboratory.org/meetings/hpnpw)_
**DOE Workshop on Ultra High-Speed Transport Protocols and Network Provisioning for Large-Scale Science Applications, April 2003**
- _[http://www.csm.ornl.gov/ghpn/wk2003](http://www.csm.ornl.gov/ghpn/wk2003)_
**Science Case for Large Scale Simulation, June 2003**
- _[http://www.pnl.gov/scales/](http://www.pnl.gov/scales/)_** **
**DOE Science Networking Roadmap Meeting, June 2003**
- _[http://www.es.net/hypertext/welcome/pr/Roadmap/index.html](http://www.es.net/hypertext/welcome/pr/Roadmap/index.html)_
**Workshop on the Road Map for the Revitalization of High End Computing, June 2003**
- _[http://www.cra.org/Activities/workshops/nitrd](http://www.cra.org/Activities/workshops/nitrd)_** **
- _[http://www.sc.doe.gov/ascr/20040510_hecrtf.pdf](http://www.sc.doe.gov/ascr/20040510_hecrtf.pdf)_** (public report)**
**ASCR Strategic Planning Workshop, July 2003**
- _[http://www.fp-mcs.anl.gov/ascr-july03spw](http://www.fp-mcs.anl.gov/ascr-july03spw)_** **
**Planning Workshops-Office of Science Data-Management Strategy, March & May 2004**
- _[http://www-conf.slac.stanford.edu/dmw2004](http://www-conf.slac.stanford.edu/dmw2004)_**  (report coming soon)**

## ESnet is Architected and Engineered to Move a Lot of Data

**ESnet is Architected and Engineered to Move a Lot of Data**

**Annual growth in the past five years has increased from 1.7x annually to just over 2.0x annually.**

- TBytes/Month

*ESnet is currently transporting about 250 terabytes/mo.**(250,000,000 Mbytes/mo.)*

*ESnet Monthly Accepted Traffic*

## Who Generates Traffic, and Where Does it Go?

**Traffic leaving ESnet = Blue**
**Traffic between sites**
- % = of total ingress or egress traffic

_**Note that more that 90% of the ESnet traffic is OSC traffic**_
_**ESnet Appropriate Use Policy (AUP)**_
- All ESnet traffic must originate and/or terminate on an ESnet an site (no transit traffic is allowed)

**Who Generates Traffic, and Where Does it Go?**

**ESnet Inter-Sector Traffic Summary,****Jan 2003 / Feb 2004 (1.7X overall traffic increase, 1.9X OSC increase)**** ****(the international traffic is increasing due to BABAR at SLAC and the LHC tier 1 centers at FNAL and BNL)**

- Commercial

- R&E (mostlyuniversities)

- International

**21/14%**

**17/10%**

**9/26%**

**14/12%**

**10/13%**

**4/6%**

- ESnet

**~25/18%**

- Peering Points

- DOE collaborator traffic, inc.data

**72/68%**

**53/49%**

- DOE is a net supplier of data because DOE facilities are used by universities and commercial entities, as well as by DOE researchers

- DOE sites

**Notes:**
This slide shows a break-down of traffic flows between ESnet sites and other sectors of the Internet for Jan 2003.  It shows that 72% of incoming (or accepted) traffic came from ESnet sites, while the remaining 28% of incoming traffic came from the various Sectors as shown above.  Similarly 53% of the outgoing (or delivered) traffic went to Sites, while the remaining 47% went to the 3 external Sectors

This would indicate that DOE is a net exporter of data – i.e. more data sent than received.  The data flowing between sites and to the R&E and International sectors could clearly be considered scientific activities.  Data flowing to the Commercial sector would be a mix of direct scientific activities and activities in support of science such as making research data available to the general public.

It is important to note that external sector traffic can only flow to or from ESnet sites; traffic between external sectors cannot flow over ESnet.  This is one major distinction between ESnet and a commercial ISP.  The fact that ESnet does not need to provide bandwidth for transit traffic between external sectors is one factor of its cost effectiveness.  A second factor is that we do not pay for traffic to/from the external sectors except for the costs to connect to the peering points.

## ESnet Top 20 Data Flows, 24 hr. avg., 2004-04-20

- Fermilab (US)  CERN

- SLAC (US)  IN2P3 (FR)

- 1 **Terabyte/day**

- SLAC (US)  INFN Padva (IT)

- Fermilab (US)  U. Chicago (US)

- CEBAF (US)  IN2P3 (FR)

- INFN Padva (IT)  SLAC (US)

- U. Toronto (CA)  Fermilab (US)

- Helmholtz-Karlsruhe (DE) SLAC (US)

- DOE Lab  DOE Lab

- DOE Lab  DOE Lab

- SLAC (US)  JANET (UK)

- Fermilab (US)  JANET (UK)

- Argonne (US)  Level3 (US)

- Argonne  SURFnet (NL)

- IN2P3 (FR)  SLAC (US)

- Fermilab (US)  INFN Padva (IT)

- A small number of science users account for a significant fraction of all ESnet traffic

- Since BaBar data analysis started, the top 20 ESnet flows have consistently accounted for > 50% of ESnet’s monthly total traffic (~130 of 250 TBy/mo)

- ESnet is Engineered to Move a Lot of Data

## ESnet Top 10 Data Flows, 1 week avg., 2004-07-01

- FNAL (US)  IN2P3 (FR)2.2 Terabytes

- SLAC (US)  INFN Padua (IT)5.9 Terabytes

- U. Toronto (CA)  Fermilab (US)0.9 Terabytes

- SLAC (US) Helmholtz-Karlsruhe (DE)
- 0.9 Terabytes

- SLAC (US)  IN2P3 (FR) 5.3 Terabytes

- CERN  FNAL (US)1.3 Terabytes

- FNAL (US)  U. Nijmegen (NL)1.0 Terabytes

- FNAL (US) Helmholtz-Karlsruhe (DE)
- 0.6 Terabytes

- FNAL (US) SDSC (US)
- 0.6 Terabytes

- U. Wisc. (US) FNAL (US)
- 0.6 Terabytes

- The traffic is not transient: Daily and weekly averages are about the same.
- _SLAC is a prototype for what will happen when Climate, Fusion, SNS, Astrophysics, etc., start to ramp up the next generation science_