# Presentation: 153651

## Integrated Control Plane Developments: CHEETAH, HOPI, OSCARS, USN

- Nagi Rao
- Oak Ridge National Laboratory, _[raons@ornl.gov](mailto:raons@ornl.gov)_
- Malathi Veeraraghavan
- University of Virginia, _[mv@ece.virgina.edu](mailto:mv@ece.virgina.edu)_
- Tom Lehman
- Information Sciences Institute East, _[tlehman@isi.edu](mailto:tlehman@isi.edu)_
- Nasir Ghani
- Tennessee Technological University, _[nghani@tntech.edu](mailto:nghani@tntech.edu)_
- Chin Guok
- ESnet, chin@es.net

- ESCC/Internet2 Joint Techs WorkshopJul 16-19, 2006
- Madison, Wisconsin

## Outline

- Introduction to Control Planes
- USN Update
- CHEETAH, HOPI-DRAGON, OSCARS
- Multi Domain Resource Provisioning
- A Framework
- Integrated Path Computation
- VLAN-Based Channel Alignment

## Disparate Control Plane Technologies

- We area at a critical point in control-plane technologies for advanced networks:
**MPLS/GMPLS being enabled and/or deployed by vendors**
**Several control planes are being developed under different paradigms:**
**ESnet-OSCARS, CHEETAH, HOPI-DRAGON, UCLP, USN, GEANT, DANTE, and others**
**Underlying challenges and solutions are slowly being understood: Standards are being discussed**
**security of control plane access**
**bandwidth optimization**
**data and control plane alignment**
**synchronization of schedulers and signaling**
**multi-domain authorization**
**others**
- Solutions appear to be diverging, at least on the surface but they address different application domains

## Control Plane Paradigms

- Two main paradigms
- On-Demand Provisioning:
- TL1,CLI: earlier host-level mechanisms primarily for manual configuration
- MPLS for layer 3, GMPLS for lower layers meant for automated configuration at network-level
- In-Advance Provisioning: Using a front-end for
- Path computation and/or
- Bandwidth optimization
- followed by mostly in-time signaling
- These paradigms address different application needs:
- in-advance provisioned bandwidth to coordinate with reservations on other facilities such as supercomputers and experimental facilities.
- on-demand provisioning to provide dedicated bandwidth

## Some Control Plane Technologies

- HOPI-DRAGON – on-demand
- GMPLS front end with CLI/TL1 for lower layers
- Ethernet switches and routers
- OSCARS-ESnet – in-advance
- Path computation + MPLS at layer 3
- Juniper and Cisco routers
- USN – in-advance
- Bandwidth optimization + CLI/TL1
- Ciena CDCI + Force10 E300
- CHEETAH – on-demand
- On-demand provisioning using GMPLS
- Sycamore SN16000 switches

## USN Data-Plane: Node Configuration

- In the Core:
- Two OC192 switched by Ciena CDCIs
- At the Edge
- 10/1 GigE provisioning using Force10 E300s

- Data Plane User Connections:
- Direct connections to:
- core switches –SONET &1GigE
- MSPP – Ethernet channels
- Utilize UltraScience Net hosts

- Node Configuration

**CDCI**

**e300**

**Linux host**

- 10GigE

- GigE

- OC192
- to Seattle

- 10GigE
- WAN PHY

- Connections to
- CalTech and ESnet

## USN Secure Control-Plane

- VPN-based authentication, encryption and firewall
- NetScreen ns-50 at ORNL
-  	NetScreen ns-5 at each node
- Centralized server at ORNL
**	****Bandwidth scheduling**
**	****Signaling**

**ns-5**

**CDCI**

**e300**

**linuxhost**

- VPN  tunnel

**CDCI**

**e300**

**linuxhost**

**Control**
**server**

**ns-50**

## USN Control Plane

- Phase I
- Centralized path computation for bandwidth optimization
- TL1/CLI-based communication with CoreDirectors and E300s
- User access via centralized web-based scheduler
- Phase II (currently being tested)
- GMPLS wrappers for TL1/CLI
- Inter-domain “secured” GMPLS-based interface
- Webservices interface
- X509 authentication for web server and service

## CHEETAH Data and Control Planes

**Data Plane OC192**

**CHEETAH**

- CHEETAH: Sycamore SN16000 data plane – OC192
-       GMPLS control-plane over IP tunnels
-       Additional 1GigE L2VLANS between ORNL and Atlanta

**SN16K**

**SN16K**

**SN16K**

- GMPLS supported on
- Secure control plane
- Ns-5 terminated IP tunnels

## OSCARS: Guaranteed Bandwidth VC Service For SC Science

- ESnet On-demand Secured Circuits and Advanced Reservation System (OSCARS)
- In its current phase this effort is being funded as a research project by the Office of Science, Mathematical, Information, and Computational Sciences (MICS) Network R&D Program
- A prototype service has been deployed as a proof of concept
- To date more then 20 accounts have been created for beta users, collaborators, and developers
- More then 100 reservation requests have been processed

## Functional View of OSCARS

- Support user/application VC reservation requests
- Users enter reservations via a web-page
- Applications uses API to send signed SOAP messages
- Manage allocations of scarce, shared resources
- Centralized resource management
- Authentication is done using X509 certificates
- Authorization TBD
- Provide circuit setup and teardown mechanisms
- OSPE-TE for routing
- BGP for reachability
- RSVP-TE for signaling
- MPLS for switching
- Enable the claiming of reservations
- Policy based filtering for traffic destined for VC
- Enforce usage limits
- Per VC admission control
- Separate router queue for VCs

## ESnet Virtual Circuit Service Roadmap

- 2005            2006            2007            2008

- Dedicated virtual circuits

- Dynamic virtual circuit allocation

- Generalized MPLS (GMPLS)

- Dynamic provisioning of Multi-Protocol Label Switching (MPLS) circuits (Layer 3)

- Interoperability between VLANs and MPLS circuits(Layer 2 & 3)

- Interoperability between GMPLS circuits, VLANs, and MPLS circuits (Layer 1-3)

- Initial production service

- Full production service

## DRAGON: Project Features and Objectives

**Utilize GMPLS protocols for dynamic provisioning of Light Paths**
- Addition of CSPF Path Computation algorithms for wavelength routing
**Inter-domain service routing techniques**
- Network Aware Resource Broker (NARB) for service advertising, inter-domain ERO generation, AAA
**Application Specific Topology Description Language**
- Formalized means to describe the application topology and network service requirements
**Integration with real applications**:
- E-VLBI
- HD-CVAN

## DRAGON Technologies

- Open Source GMPLS Control Plane
- Virtual Label Swapping Router – **VLSR**
- Open source OSPF-TE & RSVP-TE to control Ethernet switches and fiber switches
- Network Aware Resource Broker – **NARB**
- GMPLS-OSPF-TE listener
- Performs the inter-domain routing, AAA, scheduling, PC
- Advanced Constrained Shortest Path First  Path Computation Element – CSPF PCE
- Domain selectable abstraction levels and end-to-end LSP
- Application Specific Topologies - **ASTB**
- Formalization of application’s resource requirements – particularly the network resources.
- Photonic metro-scale wavelength services
- Reduce/eliminate unnecessary OEO,
- allow user generated ITU waves to transit the metro network (alien waves)
- Framing and encoding agnostic

## DRAGON Control Plane 

- DRAGON Control Plane

- Virtual Label Switching Router (VLSR)
-  Open source protocols running on PC act as GMPLS signaling entity
-  Control PCs participate in protocol exchanges and reprovision covered switch according to protocol events (PATH setup, PATH tear down, state query, etc)

**Control PC**

- VLSR

**Ethernet Switch**

**Data Plane**

**Control Plane**

**Data Plane**

**NARB**

**OSPF & RSVP**

**RSVP**

**SNMP / TL1 / CLI**

- CHEETAH control-plane is quite similar – uses Sycamore SN16000

## A General Control-Plane Architecture:Single Domain

- Network Device Node #1

- state and
- scheduling

- signaling

- Resource
- database

- Network Device Node #N

- state

- signal

- state

- signal

- Centralized
- or
- Distributed

- Some modules might be trivial in some control planes

- User
- Interface

**AAA**

**Secure**
**channels**

## Integrated Multi-Domain Resource Provisioning – A Perspective

**Front-End website/webservice**
- Integrates all data bases; SOAP interface
**Meta-Scheduler**
- Path Composition and Alignment
- Computes multi-domain path; Queries domain schedulers
- May include host scheduling
**Coordinated Signaling**
- Sends path requests to domain signaling modules
**Multi-Domain Authentication, Authorization and Accounting**
**A proposal is under consideration with DOE High-Performance Networking program on this topic**

## Multi-Domain Resource Provisioning Architecture

**Federated**
**state**

**signaling**

** ****meta**
**scheduler**

**Resource**
**database**

**Reserve**
**database**

- Network Domain #M

**state**

**signal**

**schedule**

**state**

**signal**

**schedule**

- Centralized
- Or Distributed

- AAA and secure channels are domain-specific
- User authorization and credentials are domain-specific

- Network Domain #1

## USN Path Computation – Bandwidth Optimization

**A specified bandwidth in a specified time slot,**
**Earliest available time with a specified bandwidth and duration,**
**Highest available bandwidth in a specified time slot,**
**All available time slots with a specified bandwidth and duration. **
**All are computed by extending the shortest path algorithms using a closed semi-ring structure defined on sequences of real intervals**
**(i)-(iii): Variation of Dijkstra’s shortest path algorithm**
**(iv): Variation of Bellman-Ford algorithm; **
**	****- previously solved using transitive-closure algorithm**

- Sequence of disjoint real intervals

- Point-wise union

- Point-wise intersection

## All-Slots Algorithm

**Given network with band width allocations on all links**
**ALL-SLOTS returns all possible starting times for a connection with bandwidth *****b***** duration *****t *****between source node *****s *****and destination node *****d***

**Modified  Bell-Ford algorithm: Time-complexity: **
**More efficient than transitive-closure algorithm: **

## VLAN – Unifying Data-Plane Technology

- IP networks
- VLANs Implemented in MPLS tunnels (L2VPN)
- Circuit switched networks
- VLANs Implemented on top of Ethernet channels
- Align IP and circuit connections at VLAN level

**Circuit Switched**

**IP network**** **

**MPLS tunnel**

**Layer-2 connection**

**Alignment **
**of VLANs**

**VLAN over**
**MPLS**

**VLAN**
**Over Ehthernet**

## USN–CHEETAH VLAN Alignment

**UltraScienceNet**

- CHEETAH

**UltraScience Net: Layer-2 **
**      ****VLAN: E300 – CDCI - ... - CDCI – E300**
**CHEETAH: layer-3 + layer 2**
**      ****VLAN: T640-T640 – SN1600**** ****– Cisco 3750**

**USN L2 VLAN**

**CHEETAH  L2MPLS+L2 VLAN**

## Measurements: VLANs over L1-L3 connections

- Two types of measurements:
- Ping (standard version), and TCP roundtrip time of 1K packets
- _Three channels:_
- 1. USN: host-SONET-host – 1000 miles; OC-21
- 2. CHEETAH/ORNL: host-MPLS-host – 300 miles; 1Gig VLAN
- 3. USN-CHEETAH: host-SONET-MPLS-host; 1300 miles 1Gig VLAN
- Comparison Method for jitter and delay dynamics:
- 	normalization of wide-area measurements based on mean delay

- E300
- switch

- host

- host

- E300
- switch

- CDCI
- switch

- CDCI
- switch

**1000 miles**

**T640**
**router**

**T640**
**router**

- host

- host

**ORNL**

**Atlanta-SOX**

**1Gig MPLS **
**tunnel**

**300 miles**

## Normalized Comparison of VLANs:SONET - SONET-MPLS composed – L2MPLS

- Normalized Comparison of VLANs:SONET - SONET-MPLS composed – L2MPLS

**Ping measurements normalized for comparison:**

**L2MPLS**

**SONET-MPLS**
**composite**

- mean time=9.384557msstd_dev (%)=  3.281692

-  mean time=35.981812ms std_dev (%)=  0.151493

**SONET  channels**
**have smaller jitter levels**

**SONET**

- mean time=26.845877ms
-  std_dev (%)=  0.187035

## Components of Multi-Domain Control-Plane

- Security: AAA + Encryption System
- User/Client authentication over multiple domains
- Path Optimization: Meta-Scheduler
- Composite graph
- Nodes: peering points
- Edge: in-domain connections or resources
- Path computation and composition
- Issue individual domain path requests
- Agglomerate domain paths
- Integrated Signaling System
- Issue reservation commands
- Exception handling

## Domain Technical Tasks

- USN-CHEETAH
- Webservice interface
- Integrated VLAN specification
- OSCARS
- VLANs over MPLS tunnels
- Provisioning to Chicago and Sunnyvale routers
- HOPI-DRAGON
- Webservice interface
- Advanced Scheduling
- Peering in Chicago and Seattle

## An Integrated Scheduler for Hybrid Networks

- MPLS tunnel over IP network
- Layers 1-2 over switched networks
- Approach
- Higher level scheduler composes end-to-end VLAN consisting of portions of MPLS tunnels and layer 1-2 channels
- Extensions of Dijkstra and Bellman-Ford Algorithms
- Coordinated daemons will be launched to setup the needed portions of path

**Layer 3**
**connection**

**Layer 3**
**connection**

**Layer 2/1**
**circuit**

- On-demand

**GMPLS**

**MPLS**

**MPLS**

**OSCARS**

**USN+(TL1,CLI)**

**OSCARS**

- In-advance

## Conclusions

- Coming Together of  Control-Plane Technologies
- Different application domains are optimally supported by different control-planes
- Multiple-domain paths are needed for several applications
- Several new technologies exist but more are needed, and they all must be integrated
- Hybrid Integrated Architecture
- Networks that combine best of shared IP and dedicated channels are feasible but need further development
- Data-planes must be aligned and peered in addition to control planes
- Peering Multiple Control-Planes is Complex
- Synchronization of schedulers
- Multi-domain path composition
- Coordinated signaling
- Multi-Domain AAA