# Presentation: 733382

## Achieving high performance throughput in production networks 

**Achieving high performance throughput in production networks **

***Les Cottrell ****– SLAC*
- Presented at the Internet 2 HENP Networking Working Group kickoff meeting at Internet 2 Ann Arbor, Michigan, Oct 26 ‘01

- Partially funded by DOE/MICS Field Work Proposal on Internet End-to-end Performance Monitoring (IEPM), also supported by IUPAP

- www.slac.stanford.edu/grp/scs/net/talk/thru-i2henp-oct01.html

## High Speed Bulk Throughput

**Driven by:**
- Data intensive science, e.g. data grids
- HENP data rates, e.g. BaBar 300TB/year, collection doubling yearly, i.e. PBytes in couple of years
- Data rate from experiment today ~ 20MBytes/s ~ 200GBytes/d
- Multiple regional computer centers (e.g. Lyon-FR, RAL-UK, INFN-IT, LBNL-CA, LLNL-CA, Caltech-CA) need copies of data
- Tier A gets 1/3 data in 1/3 year (full rate), SLAC does not keep copy
- Boeing 747  high throughput, BUT poor latency (~ 2 weeks) & very people intensive
- So need high-speed networks and ability to utilize
- High speed today = few hundred GBytes/day (100GB/d ~ 10Mbits/s)

- Data vol

-  Moore’s law

## How to measure network throughput

- Selected about 2 dozen major collaborator sites in US, CA, JP,  FR, CH, IT, UK over last year
- Of interest to SLAC
- Can get logon accounts
- Use **iperf **
- Choose window size and # parallel streams
- Run for 10 seconds together with ping (loaded)
- Stop iperf, run ping (unloaded) for 10 seconds
- Change window or number of streams & repeat
- Record # streams, window, throughput  (Mbits/s), loaded & unloaded ping responses, cpu utilization, real time
- Verify window sizes are set  properly by using **tcpdump******** ****can’t believe what application tells you** 
- Note cpu speeds, interface speeds, operating system, path characteristics

## Typical results

- 8kB

- 16kB

- 32kB

- 100kB

- 64kB

## Windows vs Streams

- Often for fixed ***streams*window*** product, streams are more effective than window size, e.g. SLAC>CERN, Jul ‘01:
- There is an optimum number of streams above which performance flattens out
- Common for throughputs to be asymmetric
- more congestion one way, different routes, host dependencies

| Window | Streams | Mbps | W*S |
| --- | --- | --- | --- |
| 64KB | 25 | 68 | 1600KB |
| 128KB | 12 | 56 | 1536KB |
| 256KB | 6 | 66 | 1536KB |
| 512KB | 3 | 62 | 1536KB |
| 1024KB | 2 | 26 | 2048KB |
| 2048KB | 1 | 17 | 2048KB |
| 4096KB | 1 | 25 | 4096KB |

## Windows vs Streams

- Multi-streams often more effective than windows
- more agile in face of congestion
- Often easier to set up
- Need root to configure kernel to set max window
- Network components may not support big windows
- Some OS’ treat max windows strangely 
- May be able to take advantage of multiple paths
**But:**
-  may be considered over-aggressive (RFC 2914) 
- can take more cpu cycles
- how to know how many streams?

## Iperf client CPU utilization

- As expected increases with throughput (mainly kernel)
-   0.7*MHz/Mbits/s

- For fixed throughput
- Fewer streams take less cpu 
- E.g. 1-4 streams take 20% less cpu than 8-16 streams for same throughput (if can get it)

**Notes:**
With 1MHz ~ 1 Mbps, cpus are still slightly ahead of being able to keep up with bandwidth
Need to keep testing, evaluating faster cpus.
What happens next year when 10Gbps Ethernet becomes standard
Multiple cpus are not as effective as a single cpu for how they can handle the network interface.
Multiple cpus with multiple interfaces may be able to make better use of multiple streams

## Throughput quality improvements

***TCP******BW****** < MSS/(RTT*sqrt(loss))***

*Macroscopic Behavior of the TCP Congestion Avoidance Algorithm*, Matthis, Semke, Mahdavi, Ott, Computer Communication Review 27(3), July 1997

- Note E. Europe keeping up

**80% annual improvement ~ factor 10/4yr**

- China

**Notes:**
Note this is measured from ESnet Labs in US so biased towards US sites
Lines are medians for many measured pairs, so half pairs have better than shown

## Bandwidth changes with time 1/2

- Short term competing cross-traffic, other users, factors of 3-5 observed in 1 minute
- Long term: link, route upgrades, factors 3-16 in 12 months

- All hosts had 100Mbps NICs.
- Recently have measured 105Mbps SLAC > IN2P3 and 340Mbps Caltech > SLAC with GE

## Network Simulator (ns-2)

- From UCB, simulates network
- Choice of stack (Reno, Tahoe, Vegas, SACK...)
- RTT, bandwidth, flows, windows, queue lengths ...
- Compare with measured results
- Agrees well
- Confirms observations (e.g. linear growth in throughput for small window sizes as increase number of flows)

**Notes:**
The peaks measured for a single flow are not understood, and only occur on some links. As can be seen they are not predicted from the simulation. 

## Agreement of ns2 with observed

## Ns-2 thruput & loss predict

- Indicates on unloaded link can get 70% of available bandwidth without causing noticeable packet loss
- Can get over 80-90% of available bandwidth
- Can overdrive: no extra throughput BUT extra loss

**90%**

## Simulator benefits

- No traffic on network (nb throughput can use 90%)
- Can do what if experiments
- No need to install iperf servers or have accounts
- No need to configure host to allow large windows
- BUT
- Need to estimate simulator parameters, e.g.
- RTT use ping or synack
- Bandwidth, use pchar, pipechar etc., moderately accurate
- AND ***its not the real thing***
- Need to validate vs. observed data
- Need to simulate cross-traffic etc

## Impact on Others

- Make ping measurements with & without iperf loading
- Loss loaded(unloaded)
- RTT
- Looking at how to avoid impact: e.g. QBSS/LBE, application pacing, control loop on stdev(RTT) reducing streams, want to avoid scheduling

## File Transfer

- Used **bbcp** (written by *Andy Hanushevsky*)
- similar methodology to iperf, except ran for file length rather than time, provides incremental throughput reports, supports /dev/zero, adding duration
- looked at /afs/, /tmp/, /dev/null
- checked different file sizes
- Behavior with windows & streams similar to iperf
- Thrubbcp ~0.8*Thruiperf

- For modest throughputs (< 50Mbits/s) rates are independent of whether destination is /afs/, /tmp/ or /dev/null.
- Cpu utilization ~ 1MHz/Mbit/s is  ~ 20% > than for iperf

## Application rate-limiting

- Bbcp has transfer rate limiting
- Could use network information (e.g. from Web100 or independent pinging) to bbcp to reduce/increase its transfer rate, or change number of parallel streams

- No rate limiting, 64KB window, 32 streams

- 15MB/s rate limiting, 64KB window, 32 streams

## Using bbcp to make QBSS measurements

- Run bbcp src data /dev/zero, dst=/dev/null, report throughput at 1 second intervals
- with TOS=32 (QBSS)
- After 20 s. run bbcp with no TOS bits specified (BE)
- After 20 s. run bbcp with TOS=40 (priority)
- After 20 more secs turn off Priority
- After 20 more secs turn off BE

## QBSS test bed with Cisco 7200s

- Set up QBSS testbed
- Configure router interfaces
- 3 traffic types:
- QBSS, BE, Priority
- Define policy, e.g.
- QBSS > 1%, priority < 30%
- Apply policy to router interface queues

- 10Mbps

- 100Mbps

- 100Mbps

- 100Mbps

- 1Gbps

- Cisco 7200s

**Notes:**
class-map match-all qbss 
match ip dscp 8 
class-map match-all hi-priority 
match ip dscp 10 
policy-map qos-policy 
class qbss 
bandwidth percent 1 
random-detect 
class hi-priority 
bandwidth percent 30 
random-detect 
class class-default 
bandwidth percent 69 
random-detect 
interface ... 
service-policy output qos-policy 

## Example of effects

- Also tried: 1 stream for all, and priority at 30%

## QBSS with Cisco 6500

- 6500s + Policy Feature Card (PFC)
- Routing by PFC2, policing on switch interfaces
- 2 queues, 2 thresholds each
- QBSS assigned to own queue with 5% bandwidth – guarantees QBSS gets something
- BE & Priority traffic in 2nd queue with 95% bandwidth
- Apply ACL to switch port to police Priority traffic to < 30%

- 100Mbps

- 1Gbps

- 1Gbps

- 1Gbps

- 1Gbps

- Cisco 6500s +
- MSFC/Sup2

- Time

- 100%

- BE

- Priority
- (30%)

- QBSS
- (~5%)

## Impact on response time (RTT)

- Run ping with Iperf loading with various QoS settings, iperf ~ 93Mbps
- No iperf ping avg RTT ~ 300usec (regardless of QoS)
- Iperf = QBSS, ping=BE or Priority: RTT~550usec
- 70% greater than unloaded
- Iperf=Ping QoS (exc. Priority) then RTT~5msec
- > factor of 10 larger RTT than unloaded
- If both ping & iperf have QoS=Priority then ping RTT very variable since iperf limited to 30%
- RTT quick when iperf limited, long when iperf transmits

## Possible HEP usage

- Apply priority to lower volume interactive voice/video-conferencing and real time control
- Apply QBSS to high volume data replication
- Leave the rest as Best Effort
- Since 40-65% of bytes to/from SLAC come from a single application, we have modified to enable setting of TOS bits
- Need to identify bottlenecks and implement QBSS there
- Bottlenecks tend to be at edges so hope to try with a few HEP sites

## Acknowledgements for SC2001

- Many people assisted in getting accounts, setting up servers, providing advice, software etc.
- Suresh Man Singh, Harvey Newman, Julian Bunn (Caltech), Andy Hanushevsky, Paola Grosso, Gary Buhrmaster, Connie Logg (SLAC), Olivier Martin (CERN), Loric Totay, Jerome Bernier (IN2P3), Dantong Yu (BNL), Robin Tasker, Paul Kummer (DL), John Gordon (RL), Brian Tierney, Bob Jacobsen, (LBL), Stanislav Shalunov (Internet 2), Joe Izen (UT Dallas), Linda Winkler, Bill Allcock (ANL), Ruth Pordes, Frank Nagy (FNAL), Emanuele Leonardi (INFN), Chip Watson (JLab), Yukio Karita (KEK), Tom Dunigan (ORNL), John Gordon (RL), Andrew Daviel (TRIUMF), Paul Avery, Greg Goddard (UFL), Paul Barford, Miron Livny (UWisc), Shane Canon (NERSC), Andy Germain (NASA), Andrew Daviel (TRIUMF), Richard baraniuk, Rold Reidi (Rice).

## SC2001 demo

- Send data from SLAC/FNAL booth computers (emulate a tier  0 or 1 HENP site) to over 20 other sites with good connections in about 6 countries
- Throughputs from SLAC range from 3Mbps to > 300Mbps
- Part of bandwidth challenge proposal
- Saturate 2Gbps connection to floor network
- Apply QBSS to some sites, priority to a few and rest Best Effort
- See how QBSS works at high speeds
- Competing bulk throughput streams
- Interactive low throughput streams, look at RTT with ping

## WAN thruput conclusions

- High FTP performance across WAN links is possible
- Even with 20-30Mbps bottleneck can do > 100Gbytes/day
- Can easily saturate a fast Ethernet interface over WAN
- Need GE NICs, > OC3 WANs & to improve performance
**Performance is improving**
- OS must support **big windows** selectable by application
- Need **multiple parallel streams **in some cases
**Loss is important** in particular interval between losses
- Can get close to max thruput with small  (<=32Mbyte) with sufficient (5-10) streams
**Improvements of 5 to 60 in thruput** by using multiple streams & larger windows
- Impacts others users, **QBSS looks hopeful**

## More Information

- IEPM/PingER home site:
- _[www-](http://www-iepm.slac.stanford.edu/)__[iepm](http://www-iepm.slac.stanford.edu/)__[.](http://www-iepm.slac.stanford.edu/)__[slac](http://www-iepm.slac.stanford.edu/)__[.](http://www-iepm.slac.stanford.edu/)__[stanford](http://www-iepm.slac.stanford.edu/)__[.edu/](http://www-iepm.slac.stanford.edu/)_
- Bulk throughput site:
- www-iepm.slac.stanford.edu/monitoring/bulk/
- Transfer tools:
- _[http://dast.nlanr.net/Projects/Iperf/release.html](http://dast.nlanr.net/Projects/Iperf/release.html)_
- _[http://doc.in2p3.fr/bbftp/](http://doc.in2p3.fr/bbftp/)_
- www.slac.stanford.edu/~abh/bbcp/
- _[http://](http://hepwww.rl.ac.uk/Adye/talks/010402-ftp/html/sld015.htm)__[hepwww](http://hepwww.rl.ac.uk/Adye/talks/010402-ftp/html/sld015.htm)__[.](http://hepwww.rl.ac.uk/Adye/talks/010402-ftp/html/sld015.htm)__[rl](http://hepwww.rl.ac.uk/Adye/talks/010402-ftp/html/sld015.htm)__[.ac.](http://hepwww.rl.ac.uk/Adye/talks/010402-ftp/html/sld015.htm)__[uk](http://hepwww.rl.ac.uk/Adye/talks/010402-ftp/html/sld015.htm)__[/](http://hepwww.rl.ac.uk/Adye/talks/010402-ftp/html/sld015.htm)__[Adye](http://hepwww.rl.ac.uk/Adye/talks/010402-ftp/html/sld015.htm)__[/talks/010402-ftp/html/sld015.](http://hepwww.rl.ac.uk/Adye/talks/010402-ftp/html/sld015.htm)__[htm](http://hepwww.rl.ac.uk/Adye/talks/010402-ftp/html/sld015.htm)_
- TCP Tuning:
- _[www.ncne.nlanr.net/training/presentations/tcp-tutorial.](http://www.ncne.nlanr.net/training/presentations/tcp-tutorial.ppt)__[ppt](http://www.ncne.nlanr.net/training/presentations/tcp-tutorial.ppt)_
- _[www-didc.lbl.gov/tcp-wan.html](http://www-didc.lbl.gov/tcp-wan.html)_
- QBSS measurements
- www-iepm.slac.stanford.edu/monitoring/qbss/measure.html

## Extra supporting slides

## Streams usually share bandwidth

- TCP fair share with best effort

## Progress towards goal:100 Mbytes/s Site-to-Site

- Focus on SLAC – Caltech over NTON;
- Using NTON wavelength division fibers up &  down W. Coast US;
- Replaced Exemplar with 8*OC3 & Suns with Pentium IIIs & OC12 (622Mbps)
- SLAC Cisco 12000 with OC48 (2.4Gbps) and 2 ×* *OC12;
- Caltech Juniper M160 & OC48
- ~500 Mbits/s single stream achieved recently over single OC12.
- 2 OC12s, 1 machine at Caltech and 2 machines at SLAC  gave >~ 600Mbits/

## Optimizing streams

- Choose # streams to optimize throughput/impact
- Measure RTT from Web100
- App controls # streams

## SC2000 WAN Challenge

- SC2000, Dallas to SLAC RTT ~ 48msec
- SLAC/FNAL booth: Dell PowerEdge PIII 2 * 550MHz with 64bit PCI + Dell 850MHz both running Linux, each with GigE, connected to Cat 6009 with 2GigE bonded to Extreme SC2000 floor switch
- NTON: OC48 to GSR to Cat 5500 Gig E to Sun E4500 4*460MHz and Sun E4500 6*336MHz
- Internet 2: 300 Mbits/s
- NTON 960Mbits/s
- Details:
- _[www-](http://www-iepm.slac.stanford.edu/monitoring/bulk/sc2k.html)__[iepm](http://www-iepm.slac.stanford.edu/monitoring/bulk/sc2k.html)__[.](http://www-iepm.slac.stanford.edu/monitoring/bulk/sc2k.html)__[slac](http://www-iepm.slac.stanford.edu/monitoring/bulk/sc2k.html)__[.](http://www-iepm.slac.stanford.edu/monitoring/bulk/sc2k.html)__[stanford](http://www-iepm.slac.stanford.edu/monitoring/bulk/sc2k.html)__[.](http://www-iepm.slac.stanford.edu/monitoring/bulk/sc2k.html)__[edu](http://www-iepm.slac.stanford.edu/monitoring/bulk/sc2k.html)__[/monitoring/bulk/sc2k.html](http://www-iepm.slac.stanford.edu/monitoring/bulk/sc2k.html)_

## Optimum window & streams 1/2

- Require RTT * BandWidth  *window*streams
- E.g. for SLAC CERN:
- Window*streams=159msec*85Mbits/s~1.7Mbytes
- Over 1000 pkts in pipe at a time

| W KB | S | W * S MB |
| --- | --- | --- |
| 64 | 30 | 1.92 |
| 128 | 11 | 1.41 |
| 256 | 7 | 0.93 |
| 512 | 3 | 1.5 |

- Use ping for RTT
- Pipechar for BW

## LAN MTU measure @ Caltech

- 2 Dells dual 730MHz & dual 860MHz PCs
- Connect with  Syskonnect dual port GE interfaces,
- Linux 2.4.2-2smp
- Run iperf & ttcp, window
- Got 475Mbps/port for 1.5KB MTU
- Got 960MBytes/port for 9KB MTU
- Windows 2000
- Run Microsoft “speedy” application
- Got 820Mbits/s with 9KB MTU, max IRQ 3000/s, 128KB buffers, 20MB window size for TCP, single stream

**Notes:**
Used Microsoft “speedy” application, varied buffer lengths, the max IRQ rate on the cards, frame size used and number of threads (1 or 2). Set windows registry on both machines to 20MB and Tcp1323Opts to 3 (settings used by Microsoft to achieve Land speed record. Got ~ 820Mbits/s with 128KB buffers  and 9K MTU and set a max of 3000 IRQs per second. Much poorer results if use defaults.

## LAN windows & streams

- With today’s GE interfaces need big windows:
- e.g. 2 hosts on same Ethernet switch, RTT~300usec, needs window size of 300KBytes
- 90-100MB/s for window >= 64KB & streams 5-8
- windows*streams does not work
- windows alone no good
- streams alone no good

- Between Sun E420R with 4 cpus at 450MHz  and Netra 1405 with 4 cpus running at 440MHz, running Solaris 5.7

## Bandwidth changes with time 2/2

- CERN to Caltech for last year and a half
- Improvements come in steps, when bottleneck upgraded

## Other Improvements

- Improved TCP stacks_[ ](http://www.web100.org/index.php)__[ ](http://www.web100.org/index.php)_
- Faster recovery after losses, selective acknowledgement
- Web100 experimental TCP stack_[ ](http://www.web100.org/index.php)_
- Allows read/write access to TCP internal variables
- Can provide improved monitoring & diagnostics
- So application can tune stack on the fly

## Applications

- Main network application focus today is on replication at multiple sites worldwide (mainly N. America, Europe and Japan)
- Need fast, secure, easy to use, extendable way to copy data between sites
- Need to interactive and real time  at same time, e.g. experiment control, video & voice conferencing
- HEP community has developed 2 major (freely available) applications to meet replication need: **bbftp** and **bbcp**

## Bbcp

- Peer-to-peer copy program with multiple (<=64) streams, large window support, secure password exchange (ssh control path, single use passwords (data path)), similar syntax to scp

- Peer-to-peer
- No server, if have program, have service (usually no need for admins), any node can act as source or sink, 3rd party copies
- Provides sequential I/O (e.g. from /dev/zero, to pipe or tape or /dev/null) and progress reporting

- C++ component design allows testing new algorithms (relatively easy to extend)

## Bbcp: algorithms

- Data pipelining
- Multiple streams “simultaneously” pushed
- Automatically adapts to router traffic shaping
- Can control maximum rate
- Can write to tape, read from /dev/zero, write to /dev/null, pipe
- Check-pointing (resume failed transmission)
- Coordinated buffers
- All buffers same-sized end-to-end
- Page aligned buffers
- Allows direct I/O on many file-systems (e.g. Veritas)