# Presentation: 937912

## Lambda Station: On-demand flow based routing for data intensive GRID applications over multitopology networks

*GridNets 2006, San Jose,CA, October 1 – 2, 2006*

*Fermi National Accelerator Laboratory*
*              **Don Petravick (PI), Phil Demar, Matt Crawford*
*California Institute of Technology.*
*                        **Harvey Newman (co-PI)*

*by Andrey Bobyshev, Fermilab.*

## Outline of the talk:

-  goals and major directions of the project
-  software architecture, API,  SOAP/XML, some details
-  results of using Lambda Station service,  SC05 Demo
-  problems and challenges, plans

## Basic terms

-  Lambda Station – a host with special software to control traffic path across  LAN and WAN on-demand of applications
-  PBR – policy based routing
-  PBR Client –  a system or cluster and applications running on it  sourcing traffic flows that can be subject for policy based routing
-  Flow -  a stream of packets with some attributes in common such as endpoint IP addresses (or range of addresses), protocols, protocol's ports if applicable and differentiated services code point (DSCP).

## The goal of the project... 

-     The main goal of Lambda Station project is to design, develop and deploy  a  network path selection services to interface production storage and computing facilities with advanced research networks.
- selective forwarding on a ***per flow*** basis
- alternate network paths for high impact data movement
- access control in site edge routers for those selected flows
- on-demand from applications (authentication & authorization)
- current implementation based on policy-based routing & including the support of DSCP marking

**The goal of the project... **
-  	deals with the last-mile problem in local area networks

## Lambda Station TestBed

- two HEP sites, Fermilab and Caltech
-  10G & 1G End-To-End paths via DOE UltraScienceNet and NFS UltraLight
-  components of test and production computing/storage/network  infrastructures
-  10Gb  and 1Gb servers

## Lambda Station Building Blocks

**LSI**nterface

- mySQL:requests, history,security

**LSC**ontroller

- SOAP/Clarens

- Vendor specific
- modules

- NETWORK CONFIGURATOR

- CISCO

- Force10

- WAN

- Storage & application space

- Management

- SOAP/Clarens

- LSInterface

- Remote lambdastation

- Data Exchange

- Control & Management

**Service-based Architecture:**
-  CLARENS is framework for service based architecture, mutual authentication of requests
- LSController – synchronizes work of all services, has control functions
- LSI - unified interface for intercommunication between LS and applications, and LS-to-LS
-  LSDIScovery service – detects new lambdastations, and PBR clients at remote and local sites
-  LSRESource Scheduler – estimates bandwidth allocation, monitor real-time usage of resources
-  NETWORK CONFIGURATOR – dynamic reconfiguring of LAN and WAN

- SOAP

## Lambda Station Software

-  Multiple API-calls or primitives (~ 20)  divided into several groups:
-  informative (whoiam,sayHello,getClientInfo, getKnownLambdastations....)
-  service or operational  (openSvcTicket,cancelTicket,getTicket,getFlowSpec...)
-  internal (getNextDSCP, assignDSCP,updateMyLambdaDefs)
-  single interface for client-to-LS and LS-to-LS communication

**Scenarios to configure LANs:**
-   dynamically configurable  local networks  with dynamically or statically assigned DSCP
-  without DSCP tagging
-  statically configured networks with predefined matching criterion for flows

**LS Software v1.0  – a fully functional prototype but no interoperability across different platforms**

## Java implementation of LS Software

-  Service Oriented Architecture
-  build by using JClarens  framework as a grid-aware tool
-  messages are defined by XML schemas
-  core authentication is based on gLite library to support standard Grid proxies and KCA-issued  certificates
-  secure document/literal wrapped SOAP messages, Web Services Interoperability Profile (WS-I Basic Profile Version 1.1)
- Multi threaded LS implementation (LS Controller, NetConfig and LS/PBR Config  as JClarens background processors)

## Flows and DSCP tagging

-  Any combination of flow's attributes  can be used by Lambda Station software to identify flows on per ticket's basis.

- Typical steps of alternate paths:
-  generate request for service (application, application's proxy, admin)
-  LS negotiates service and parameters with remote site
-  configure local and *wide area network (not yet available)*
-  marking traffic (if specified).

- Current LS software is capable to complete all these steps within 3 – 5 mins. That is why it is desirable to know flow selection parameters before transferring  is started:
-  endpoint IP addresses
-  DSCP

## DSCP Tagging

- Complexity of using DSCP tagging:
-  preservation of DSCP is not guaranteed in WAN
-  DSCP tagging needs to be synchronize between sites for dynamically configurable networks ( asymmetry is bad for high-performance transferring )

- LS software does support two different modes of DSCP tagging :
-  fixed DSCP values to identify site's traffic.
-  DSCP  value is assigned dynamically on per ticket base.

## Interaction between applications and LS

- Multiple scenarios how applications and Lambda Station may communicate:
-   modified applications with LS awareness capability to place and manipulate tickets
-  w/ and w/o DSCP traffic marking
-  unmodified applications, a proxy controls paths for their traffic

**Operational modes of openSvcTicket  request ****(subject to authentication, authorization and quotas):**

-  new ticket -  creating a new ticket
-  join -  provides ID and required parameters(DSCP) of  already opened ticket
-  extend ticket – allows to extend already created ticket.

## Lambda Station Aware Applications

-  **lsiperf, lsTraceRoute:**
-  Full support of Lambda Station awareness
-  Places a ticket for an alternative path, negotiates and synchronizes  with remote server's application the parameters (DSCP tagging with IPTables ) of request
-  watches ticket's status
**SRM/dCache:**
-   limited Lambda Station awareness
-   places ticket, join and extend
-   recognizes absence of Lambda Station
-   tries to use it only for specific destinations

## LSiperf End-to-End Test

- 10GE host; 5 tcp streams
- Network path is via ESnet
- OC12 bottleneck...
- Path MTU is 1500B
- LambdaStation service ticket is opened
- LambdaStation changes network path to USN
- Host path MTUD check detects a larger path MTU
- LambdaStation service ticket expires:
- Network path changed back to ESnet

## Flows based switching demo at SC05

## Project accomplishments

-   Software version 1 (a fully functional prototype supporting whole cycle of LS functionality)
-  positive results of testing between Fermilab and Caltech
-  positive results during SC05 via general Internet and SCinet (10G path)
-  lsiperf, lsTraceroute – wrappers around well known applications to add Lambda Station awareness
-  SRM/dCache integration – testing, added in production 1.7.0 release

## Problems and challenges

-  Traffic Asymmetry is bad for high performance applications
-   Network (Lambda Station) awareness is too complex
-  Definition of PBR Clients is a complex issue, auto definition is not yet available

-  References:  http://www.lambdastation.org/

**Plans**

-  release jLS, interoperability across platforms
-  SRM/dCache with production quality LS support
-  add real-time monitoring of resources
-  add WAN control plane module

## END

## Miscellaneous  Slides

-  Simulation of multiple  Lss
-  screen shoots of ticket's queue
-  SC05 Demo

## LS Java API

- Multi threaded LS implementation:
- LS Controller, NetConfig and LS/PBR Config  as JClarens background processors
- Apache XmlBeans API for XML data binding
- JClarens  utilized to provide User Session management, authorization, authentication and transparent DB connections pooling
- Interoperability tests with perl, python clients

## Netconfig  Module

-  dynamically modifies the  configurations of local network devices.
-  a vendor dependent components.
-  Cisco routers with IOS version supporting sequencing type of named ACLs.
- Tasks to configure  PBR in Cisco devices:
-   interface on which PBR is applied needs to be configured with ***“ip policy route-map”*** statement
-  route map needs to be configured as ordered  list of match/action statements
-  match criteria need to be associated with ACLs

**Notes:**
Currently  NetConfig module implements the  least disruptive approach (from our point of view) to modify   PBR dynamically by updating ACLs associated with match criteria. It means that basic PBR configuration needs to be prepared by network administrators based on site's specific and needs.  Devices with same policy rules are grouped together to simplify management.  

## openSvcTicket request

-  new ticket -  creating a new ticket
-  join -  provides ID and required parameters(DSCP) of  already opened ticket
-  extend ticket – allows to extend already created ticket.

-  Accepts many input parameters, most of them could also be determined automatically:
-  Dst site's identifiers
-  Src/Dst PBRClient identifier or/and list of  Src/Dst IP in (CIDR)
-  IP protocols, protocol's ports
-  localPath, remotePath identifiers
-  BWout,BWin
-  boardTime, startTime, endTime

- Returns   ID  of locally assigned ticket

**Operational modes ****(subject authentication, authorization and quoting):**

## Directions of the project.

-  building a wide-are testbed infrastructure
-  designing, developing Lambda Station software, Lambda Station (network) aware applications
-  researching  effects  of flow based switching on applications

## Application's behavior in flow based switching environment

-  Tuning of end systems for maximum rates is not a subject of the Lambda Station project, however, we need to see an increase of data movement performance when selectively switch flows
-  DSCP tagging with IPtables
-  switching between  two paths with different MTUs

## Effect of DSCP tagging with IPTables

## LS JAVA API Components

## Motivation of the project

-  unprecedented demands for data movement in physics experiments such as CDF, D0, BarBar and coming LHC experiments
-  massive, globally distributed datasets growing to the 100 petabytes by 2010
-  collaborative data analysis by global communities of thousands of scientists
-  available data communication technology will not be able to satisfy these demands simply by plain increasing bandwidth in LANs and WANs due to technology limitations and high deployment and operation costs.
-   Advance Research optical networks – greater capacity, no production quality of service, are    not universally available for all pairs of communicating endpoints

## LS multitopology network model 

- Cisco IOS, dynamic configuring of PBR, extended sequencing ACLs + access policy ACLs

## Slide 28

## Slide 29

## Note A

**Note A**: We believe it is a HW/ASIC problem with SNMP monitoring, a time to time SNMP -get returns the same counters as in previous cycle.

- PMTUD

## Slide 31

## Slide 32