# Presentation: 300611

## NET100Development of network-aware operating systems

- Tom Dunigan   *thd@ornl.gov*

## Net100 project

- New DOE-funded (Office of Science) project ($1M/yr, 3 yrs)
- Principal investigators
- Wendy Huntoon and the PSC/Web100 team (Janet Brown, Matt Mathis)
- Brian Tierney, LBNL
- Tom Dunigan, ORNL
- Rich Wolski, UCSB
- collaborators: Basil Irwin, Bill Wing, Nageswara Rao
- Objective: develop network aware operating systems
-  optimize and understand end-to-end network and application performance
-  eliminate the “wizard gap”
-  Web100 to Web1000?
- Motivation
- DOE has a large investment in high speed networks (ESnet) and distributed applications

## Net100 approach

- Deploy/enhance Web100 into DOE network applications
- auto-tune network applications to optimize performance
- collect performance statistics to understand/tune networks and applications
- evaluate network applications over DOE’s ESnet (OC12, OC48?)
- bulk transfers over high bandwidth/delay network
- distributed applications (grid)
- Develop  Network Tools Analysis Framework (NTAF)
- configure/launch network tools (*NWS, pathrate, pipechar*, ...)
- aggregate and transform output from tools and Web100
- Develop Network Analysis Information Base (NAIB)
- repository for NTAF data
- API to collect and query

## Bulk transfers

- ORNL/NERSC Probe project
- wide-area distributed storage testbed (HPSS)
- investigate protocols, software, devices
- climate model data transfers were slow
-  OC3 with 60 ms RTT
- classic TCP tuning problem
-  also broken TCP stacks
-  developed (almost) TCP-over-UDP test harness
- instrumented and tunable
- Recent upgrade (?) to OC12, 100 ms RTT

## TCP losses

- Packet losses during startup, linear recovery 0.5 Mbs

- instantaneous

- average

- Packet  loss

- Early packet
- drops

## Net100 to the rescue?

- Avoid losses
- retain/probe for “optimal” buffer sizes
- autotuning (Web100/Net100)
- ECN capable routers/hosts
- reduce bursts
- Faster recovery
- shorter RTT (“fix” routes)
- bigger MSS (jumbo frames)
- speculative recovery
- modified congestion avoidance?
- SCTP, out-of-order delivery

## Bulk transfer speedups

- Parallel streams (*psockets*)
- how to choose number of streams, buffer sizes?
- Web100 autotune ?
- Application routing daemons
- indirect TCP
- alternate path (Wolski, UCSB)
- multipath (Rao, ORNL)
- Are these fair?

## Network Tool Analysis Framework (NTAF)

- Configure and launch network tools
- measure bandwidth/latency (*iperf, pchar, pipechar*)
- collect passive data (SNMP from routers, OS counters)
- forecast bandwidth/latency (NWS) for grid resource scheduling
- augment tools to report Web100 data
- Collect and transform tool results into a common format
- Save results for short-term auto-tuning  and archive (NAIB) for later analysis
- compare predicted to actual performance
- measure effectiveness of tools and auto-tuning
- Use NetLogger to format and send data to NAIB

## NetLogger

- End-to-end performance monitoring tool
- Modify application to log interesting events
- Support for distributed applications (NTP timestamps)
- Identify application/network bottlenecks
- Components
- IETF draft standard message format (ULM)
- API for event logging
- tools for collecting/sorting log files
- visualization tool for monitoring/playback

## NetLogger

## Network Analysis Information Base (NAIB)

- Extensible infrastructure for performance data
- Collect data from active and passive Net100 probes via NetLogger
- Gather and serve data via programmatic and graphical interfaces
- Based on Network Weather Service
- distributed set of performance sensors
- latency/bandwidth/memory/CPU
- robust periodic data collection
- forecasting module
- database of sensor configurations and sensor data
- Needed for selecting/scheduling grid resources

## NTAF Use Case

- The NTAF is configured to run the following network tests every few hours over a period of several days:
- ping -- measure network delay
- pipechar -- actively measure speed of the bottleneck link
- iperf -- actively measure TCP throughput. Multiple *iperf*  tests could be run with different parameters for the number of parallel streams {e.g.: 1,2,4} and the method of tuning the TCP buffers {Web100 auto-tuned, hand-tuned}
- NWS -- measure and predict network delay and bandwidth using NWS’ own sensors
- All tools will use the Web100 TCP-KIS interface to collect TCP information from the Web100 kernel, and then use NetLogger to format and send this data to the NWS NAIB database.

## Use Case (cont.)

- Analysis based on this test configuration includes:
- The ability to compare Web100 tuned throughput to hand-tuned throughput.
- The ability to compare NWS predicted bandwidth with application and *iperf* bandwidth.
- The ability to determine the advantage, if any, of parallel data streams, using both hand-tuned and autotuned (Web100-tuned) TCP.
- The ability to see the variability of the results over time.
- The ability to compare *pipechar* and *pathrate* to see which is most accurate.
- The ability to measure the impact of tuned TCP streams on non-tuned streams.

## Usage

## Net100 outreach

- Web pages describing current results
- Downloadable Net100 software
- NAIB data available
- Tutorials, talks, and papers
- Interact with DOE grid projects and Data Grid projects

*http://www.csm.ornl.gov/~dunigan/net100*