How NOC staff canhelp the users activities?
Yasuichi Kitamura (kita@jp.apan.net)
Jin Tanaka (tanaka@ote.kddi.com)
Yuichi Kurokawa (kurokawa@ote.kddi.com)
Takatoshi Ikeda(ikeda@ote.kddi.com)

2010年8月16日月曜日

end
user

Internet
R&E networks

You learned the technique here.

2010年8月16日月曜日

end
user

end user’s interest

end
user

2010年8月16日月曜日

communication performance

end
user

• Detect the user’s traffic.
• Performance between domains• Other side issue

2010年8月16日月曜日

Detect the user’s traffic.
• The user’s traffic has reached you?
• The user’s traffic can be transited at you?
• The user’s traffic is transferred to the
expected direction?

2010年8月16日月曜日

The user’s traffic hasreached you?
• Access network issue• End user side
• Not quite sure who can be responsible• NOC side
• you job
• Hosts issue
2010年8月16日月曜日

The user’s traffic can betransited at you?
• your job• router performance• hardware issue• routing information• flow
2010年8月16日月曜日

The user’s traffic is transferredto the expected direction?

• your job• routing information• your side?
• other sides?
2010年8月16日月曜日

Performance between
domains• your job• Available bandwidth• Paths

2010年8月16日月曜日

Other side issue• NOC• How the other side is receiving yourtraffic?

• How the other side is sending the return
traffic?

• Access network• How the traffic reaches the end user?
2010年8月16日月曜日

Dynamic Circuit
Network
end
userInternet
R&E networks

end
user

• Internet part disappears.
• Direct connection is established virtually.
2010年8月16日月曜日

Compath
A
end
user

Internet
R&E networks

C

end
user

B

• The tool can detect if the packets are going
through the expected routes.

2010年8月16日月曜日

How about others?

2010年8月16日月曜日

• ping• Most of the OSes have this command or
the same function.

• The target host is active?
• Round Trip Time (RTT)• Packet loss rate
• Timt To Live (TTL)
2010年8月16日月曜日

- ICMP/Ping Polling 2 Optional ParameterIn case of daily operation
Packet size (byte)
Sending interval (sec)Sending count (n)Timeout (sec)TTL (n)Pattern (0x????)etc.

At Monitoring system
Sending interval
Sending countTime out

Set up the value which is adapted for critical level or service level!

2010年8月16日月曜日

Network Diagnostic Tool
(NDT)
• Bottleneck Link Detection• Duplex Mismatch Detection• 2 clients are available.
• command line
• Web based client (Java applet)2010年8月16日月曜日

NDT (Cont’d)• How to use NDT?
• Follow the installation guide of http://
www.internet2.edu/pubs/ndtcookbook.pdf• Linux is required.
• Linux Kernel version sensitive
2010年8月16日月曜日

Netalyzr• http://netalyzr.icsi.berkeley.edu/• Java applet and no OS specification• Tool for checking the end user side

2010年8月16日月曜日

Netalyzr (Cont’d)• Network address translation• Network link properties• Port filtering• HTTP tests• DNS tests• Misc items• IPv6• clock drift
2010年8月16日月曜日

Iperf Tutorial
Jon Dugan <jdugan@es.net>

Summer JointTechs 2010, Columbus, OH

2010年8月16日月曜日

Outline
What are we measuring?
TCP Measurements
UDP Measurements
Useful tricks
Iperf Development

Lawrence Berkeley National Laboratory
2010年8月16日月曜日

U.S. Department of Energy | Office of ScienceWhat are we measuring?

Lawrence Berkeley National Laboratory
2010年8月16日月曜日

U.S. Department of Energy | Office of ScienceThroughput? Bandwidth? What?
The term “throughput” is vague• Capacity: link speed- Narrow Link: link with the lowest capacity along a path
- Capacity of the end-to-end path = capacity of the narrow link

• Utilized bandwidth: current traffic load
• Available bandwidth: capacity – utilized bandwidth- Tight Link: link with the least available bandwidth in a path

• Achievable bandwidth: includes protocol and host issues

45 Mbps

10 Mbps

100 Mbps

45 Mbpssourcesink

Narrow Link

Tight Link

(Shaded portion shows background traffic)
Lawrence Berkeley National Laboratory
2010年8月16日月曜日

U.S. Department of Energy | Office of ScienceIperf data flow
Client is the sender(data source)

ClientServerServer is the receiver(data sink)

Iperf discards
the data
Lawrence Berkeley National Laboratory
2010年8月16日月曜日

U.S. Department of Energy | Office of ScienceTCP Measurements

Lawrence Berkeley National Laboratory
2010年8月16日月曜日

U.S. Department of Energy | Office of ScienceTCP Measurements
Measures TCP Achievable Bandwidth• Measurement includes the end system• Sometimes called “memory-to-memory” tests• Set expectations for well coded application
Limits of what we can measure• TCP hides details• In hiding the details it can obscure what is causing errors
Many things can limit TCP throughput• Loss• Congestion• Buffer Starvation• Out of order delivery
Lawrence Berkeley National Laboratory
2010年8月16日月曜日

U.S. Department of Energy | Office of ScienceExample Iperf TCP Invocation
Server (receiver):$ iperf -s-----------------------------------------------------------Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)-----------------------------------------------------------[ 4] local 10.0.1.5 port 5001 connected with 10.0.1.10 port 60830[ 4] 0.0-10.0 sec 1.09 GBytes
933 Mbits/sec[ 4] local 10.0.1.5 port 5001 connected with 10.0.1.10 port 60831[ 4] 0.0-10.0 sec 1.08 GBytes
931 Mbits/sec

Client (sender):$ iperf -c 10.0.1.5-----------------------------------------------------------Client connecting to 10.0.1.5, TCP port 5001
TCP window size: 129 KByte (default)-----------------------------------------------------------[ 3] local 10.0.1.10 port 60830 connected with 10.0.1.5 port 5001
[ ID] Interval
Transfer
Bandwidth[ 3] 0.0-10.2 sec 1.09 GBytes
913 Mbits/sec

Lawrence Berkeley National Laboratory
2010年8月16日月曜日

U.S. Department of Energy | Office of ScienceTCP performance: window size
Use TCP auto tuning if possible• Linux 2.6, Mac OS X 10.5, FreeBSD 7.x, and Windows Vista
The –w option for Iperf can be used to request a particular buffer size.
• Use this if your OS doesn’t have TCP auto tuning• This sets both send and receive buffer size.
• The OS may need to be tweaked to allow buffers of sufficient size.
• See http://fasterdata.es.net/tuning.html for more details
Parallel transfers may help as well, the –P option can be used for thisTo get full TCP performance the TCP window needs to be large enough toaccommodate the Bandwidth Delay Product

Lawrence Berkeley National Laboratory
2010年8月16日月曜日

U.S. Department of Energy | Office of ScienceTCP performance:
Bandwidth Delay Product
The amount of “in flight” data allowed for a TCP connection
BDP = bandwidth * round trip time
Example: 1Gb/s cross country, ~100ms
1,000,000,000 b/s * .1 s = 100,000,000 bits
100,000,000 / 8 = 12,500,000 bytes
12,500,000 bytes / (1024*1024) ~ 12MB

Lawrence Berkeley National Laboratory
2010年8月16日月曜日

U.S. Department of Energy | Office of ScienceTCP performance: read/write buffer size
TCP breaks the stream into pieces transparently
Longer writes often improve performance• Let TCP “do it’s thing”• Fewer system callsHow?
• -l <size> (lower case ell)
• Example –l 128K
UDP doesn’t break up writes, don’t exceed Path MTU

Lawrence Berkeley National Laboratory
2010年8月16日月曜日

U.S. Department of Energy | Office of ScienceTCP performance: parallel streams
Parallel streams can help in some situations
TCP attempts to be “fair” and conservative• Sensitive to loss, but more streams hedge bet• Circumventing fairness mechanism- 1 Iperf stream vs. n background: Iperf gets 1/(n+1)- x Iperf streams vs. n background: Iperf gets x/(n+x)- Example: 2 background, 1 Iperf stream: 1/3 = 33%- Example: 2 background, 8 Iperf streams: 8/10 = 80%
How?
• The –P option sets the number of streams to use• There is a point of diminishing returns

Lawrence Berkeley National Laboratory
2010年8月16日月曜日

U.S. Department of Energy | Office of ScienceTCP performance:
congestion control algorithm selection
Classic TCP (aka TCP Reno) is very conservative
Linux supports several different algorithms• http://en.wikipedia.org/wiki/TCP_congestion_avoidance_algorithm• CUBIC seems to work well for RE&E traffic flows
How?
• -Z allows the selection of a congestion control algorithm

Lawrence Berkeley National Laboratory
2010年8月16日月曜日

U.S. Department of Energy | Office of Science

UDP Measurements

Lawrence Berkeley National Laboratory
2010年8月16日月曜日

U.S. Department of Energy | Office of Science

UDP Measurements
UDP provides greater transparency
We can directly measure some things TCP hides• Loss• Jitter• Out of order delivery
Use -b to specify target bandwidth• Default is 1M• Two sets of multipliers- K. m, g multipliers are 1000, 10002,10003- K, M, G multipliers are 1024, 10242,10243• Eg, -b 1m is 1,000,000 bits per second

Lawrence Berkeley National Laboratory
2010年8月16日月曜日

U.S. Department of Energy | Office of ScienceExample Iperf UDP Invocation
Server (receiver):$ iperf -u -s-----------------------------------------------------------Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size:
107 KByte (default)-----------------------------------------------------------[ 3] local 10.0.1.5 port 5001 connected with 10.0.1.10 port 65299[ 3] 0.0-10.0 sec 1.25 MBytes 1.05 Mbits/sec 0.008 ms
0/ 893 (0%)

Client (sender):$ iperf -u -c 10.0.1.5 -b 1M-----------------------------------------------------------Client connecting to 10.0.1.5, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 9.00 KByte (default)-----------------------------------------------------------[ 3] local 10.0.1.10 port 65300 connected with 10.0.1.5 port 5001
[ ID] Interval
Transfer
Bandwidth[ 3] 0.0-10.0 sec 1.25 MBytes 1.05 Mbits/sec[ 3] Server Report:[ 3] 0.0-10.0 sec 1.25 MBytes 1.05 Mbits/sec 0.003 ms
0/ 893 (0%)[

3] Sent 893 datagrams

Lawrence Berkeley National Laboratory
2010年8月16日月曜日

U.S. Department of Energy | Office of ScienceUseful tricks

Lawrence Berkeley National Laboratory
2010年8月16日月曜日

U.S. Department of Energy | Office of Science

Using Iperf to generate high rate streams
UDP doesn’t require a receiver
If you have good counters on your switches & routers those can be used
to measure
Turns out UDP reception can be very resource intensive resulting in drops
at the NIC at high rates (8-9 Gb/s)

Lawrence Berkeley National Laboratory
2010年8月16日月曜日

U.S. Department of Energy | Office of ScienceRouting loops for fun and profit
Generate 10 Gb/s of traffic using a 1 Gb/s source host

1 Gb/s Host
Iperf UDP 1 Gb/s stream
destined for 10.1.1.1

10.0.1.1

10.0.1.2

Static route:
10.1.1.1/32 10.0.1.2

Static route:
10.1.1.1/32 10.0.1.1

Use the –T option to Iperf to control the number of times the traffic loops
Can also use firewall filters to discard a certain TTL range.
Other filters may be prudent as well.
Use firewall filters to count traffic on your router.
(Do not try this at home, the author is a highly insane network engineer.)
Lawrence Berkeley National Laboratory
2010年8月16日月曜日

U.S. Department of Energy | Office of ScienceIperf Development

Lawrence Berkeley National Laboratory
2010年8月16日月曜日

U.S. Department of Energy | Office of ScienceIperf Development(Iperf is dead, long live Iperf)
Iperf 2
• Iperf 2 is widely used
• Current version is 2.0.5 (released July 8, 2010)• No further development, maintenance only
- critical patches- sporadic releases, only when necessary
Iperf 3• Currently in development• Current version is 3.0b1 (released July 8, 2010)• Weekly beta releases (starting this past Thursday)• Eventually replace Iperf 2

Lawrence Berkeley National Laboratory
2010年8月16日月曜日

U.S. Department of Energy | Office of ScienceIperf 3: current status
Working• TCP• Control channel- Stream setup- Test parameter negotiation- Results Exchange• Clean code!
Coming Soon
• UDP tests• API with sane error reporting, library• Timeline at: http://code.google.com/p/iperf/wiki/Iperf3Roadmap

Lawrence Berkeley National Laboratory
2010年8月16日月曜日

U.S. Department of Energy | Office of ScienceMore Information
Iperf 2:
http://sourceforge.net/projects/iperf/
Iperf 3:
http://code.google.com/p/iperf/
User Discussion:
iperf-users@lists.sourceforge.net
Developer Discussion:
iperf-dev@googlegroups.com
Network performance:
http://fasterdata.es.net/
Jon Dugan <jdugan@es.net>

Lawrence Berkeley National Laboratory
2010年8月16日月曜日

U.S. Department of Energy | Office of ScienceUDP/TCP pollingEffective in monitoring service ports of server
Using client for service
 DNS - nslookup

Using telnet
 WWW,SMTP,POPUsing tool
 Radius - radping
bash-2.05$ telnet ns.jp.apan.net 80
Trying 203.181.248.3...
Connected to ns.jp.apan.net.
Escape character is '^]'.
get<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML
2.0//EN"><html><head><title>501 Method Not Implemented</title>:

2010年8月16日月曜日

Telnet with service port
reply

SNMP

- Framework - SNMP: Simple Network Management Protocol
 Polling: UDP 161, Trap : UDP 162
 Protocol for monitoring/managing equipment via
network Enables us to monitor the state and traffic of various
equipment without being dependent on vender Management is realized by UDP between…
 monitoring/managing server : Manager
 e.g. HP Open View, Sun NNM

 network equipment : Agent resides in device e.g. Unix	
 daemon、Cisco	
 IOS

 Most general technique for acquiring detailed informationfrom a router or a switch

2010年8月16日月曜日

SNMP - Version  SNMP v1 RFC1157
 When Manager requests, Agent returns response Agent sends trap when specific event has occurred SNMP v2 RFC1902
 Basic of features are almost the same as those of v1
 Additional regulation
 64bit counter : can deal with large numerical value
 Get-bulk request : used to efficiently retrieve large blocks of data Supports the use of encryption of messages

 SNMP v3 RFC2271~2275
 Additional regulation
 Various security function : MD5 user authentication,
DES encryption

2010年8月16日月曜日

SNMP - MIB & OID  SNMP Manager can acquire the management information
defined by MIB(Management Information Base) fromAgentCurrent version : MIBv2 RFC 1213
MIB is the aggregate of object (information) on theequipment which SNMP Agent holds
Identifier is defined for each object = OIDMIB performed by Agent is roughly divided into:MIBv2 : standard, public, specified by IETF
Enterprise MIB : private, specified by vendor company

2010年8月16日月曜日

SNMP - MIB Tree  Objects are managed by the tree Expressed in a row of values divided by the periodroot
iso(1)

ccitt(0)Joint-iso-ccitt(2)org(3)
dod(6)
Internet(1)directory(1)2010年8月16日月曜日

mgmt(2)

exprimental(3)private(4)mib(1)

enterprise(1)Standard MIBs

Vendor-specific MIBs

SNMP - OID  OID Expression
 iso(1). org(3). dod(6). internet(1). mgmt(2). mib2(1)-> .1.3.6.1.2.1
e.g.

sysDscr = .1.3.6.1.2.1.1.1 = mib-2.1.1 = system.1

Subtree
Name

OID

Description

system

1.3.6.1.2.1.1

Defines a list of objects that pertain to system operation, such as the system uptime, system contact, and system name.

interfaces

1.3.6.1.2.1.2

Keeps track of the status of each interface on a managed entity. The interfaces group monitors which interfaces are up or down
and tracks such things as octets sent and received, errors and discards, etc.

at1.3.6.1.2.1.3

The address translation (at) group is deprecated and is provided only for backward compatibility. It will probably be dropped
from MIB-III.

ip

1.3.6.1.2.1.4

Keeps track of many aspects of IP, including IP routing.

icmp

1.3.6.1.2.1.5

Tracks things such as ICMP errors, discards, etc.

tcp

1.3.6.1.2.1.6

Tracks, among other things, the state of the TCP connection (e.g., closed, listen, synSent, etc.).

udp

1.3.6.1.2.1.7

Tracks UDP statistics, datagrams in and out, etc.

egp

1.3.6.1.2.1.8

Tracks various statistics about EGP and keeps an EGP neighbor table.

transmission

1.3.6.1.2.1.10

There are currently no objects defined for this group, but other media-specific MIBs are defined using this subtree.

snmp

1.3.6.1.2.1.11

Measures the performance of the underlying SNMP implementation on the managed entity and tracks things such as thenumber of SNMP packets sent and received.

2010年8月16日月曜日

SNMP - SNMP Message SNMP version : Check the version of SNMP(0 is for version 1)
Community : Password between Manager and AgentPDU (Protocol Data Unit) : Actual command
Manager -> AgentGetRequest
 Used to request the values of one or more MIB variables

GetNextRequest
 Used to read the values of variables in the MIB sequentially. It is often
used to read through a table of values. After reading the Getrequest,

2010年8月16日月曜日

SNMP - SNMP Message Handling 1 GetRequest (What is the value of MIB?)

SNMP Manager

GetResponse (The value is XXXX!)
GetNextRequest(What is the next value of MIB Tree ?)GetResponse (The value is XXXX!)

SetRequest (Modify the value of OID)
GetResponse (The value is XXXX!)

Trap (Problem happened!)

2010年8月16日月曜日

SNMP AgentSNMP - SNMP Message Handling 2  Command examples
GetRequest
inetapan@tools:˜>	
 snmpget	
 -v2c	
 -c	
 xxxx	
 tpr2.jp.apan.net	
 	
 .1.3.6.1.2.1.2.2.1.4.136
IF-MIB::ifMtu.136	
 =	
 INTEGER:	
 9192

GetNextRequest
inetapan@tools:˜>	
 snmpget	
 -v2c	
 -c	
 xxxx	
 tpr2.jp.apan.net	
 system
SNMPv2-MIB::system	
 =	
 No	
 Such	
 Object	
 available	
 on	
 this	 agent	 at	 this	
 OID
inetapan@tools:˜>	
 snmpwalk	
 -v2c	
 -c	
 xxxx	
 tpr2.jp.apan.net	
 system
SNMPv2-MIB::sysDescr.0	
 =	
 STRING:	
 m20	
 internet	
 router,	
 kernel	 6.2R3.10
SNMPv2-MIB::sysObjectID.0	
 =	
 OID:	
 SNMPv2-SMI::enterprises.2636.1.1.1.2.2
DISMAN-EVENT-MIB::sysUpTimeInstance	
 =	 Timeticks:	 (423280751)	
 48	
 days,	
 23:46:47.51
SNMPv2-MIB::sysContact.0	
 =	
 STRING:
SNMPv2-MIB::sysName.0	
 =	
 STRING:	
 tpr2
SNMPv2-MIB::sysLocation.0	
 =	
 STRING:
SNMPv2-MIB::sysServices.0	
 =	
 INTEGER:	
 4

SetRequest
inetapan@tools:˜>	
 snmpset	 ‒v2c	
 ‒c	
 xxxx	
 tppr.jp.apan.net	
 system.sysLocation.0	
 system.sysLocation.0	
 =	
 ""	
 
inetapan@tools:˜>	
 snmpset	 ‒v2c	 ‒c	
 yyyy	
 tppr.jp.apan.net	
 system.sysLocation.0	
 s	
 “Tokyo,	
 JP“
system.sysLocation.0	
 =	
 “Tokyo,	 JP"	
 inetapan@tools:˜>	
 snmpset	 ‒v2c	
 ‒c	
 xxxx tppr.jp.apan.net	
 system.sysLocation.0
system.sysLocation.0	
 =	
 “Tokyo,	
 JP"	
 	
 	
 	
 

2010年8月16日月曜日

SNMP - Trap Message The way for Agent to inform Manager about event of something
undesirable
Trap originates from Agent and is sent to the trap destination, as
configured within Agent itself
When Manager receives a trap, it needs to know how to interpret it
PDUEnterprisevendor identification (OID) for the agent

AgentAddress
The IP address of the node where the trap was generated.

Trap TypeGeneric / Specific (not used)TimestampThe length of time between the last re-initialization of the agent that issued a trap and themoment at which the trap was issued

2010年8月16日月曜日

Monitoring Software - HP OpenView HP OpenView Network Node Manager ®
http://www.openview.hp.com/products/nnm/index.html

Overview
 Auto discovery and mapping Drill-down views (Hierarchy Map) Fault monitoring : ICMP / SNMP polling Event monitoring : Trap receiving/Event configuration
 SNMP tools : Status polling MIB Browser
 Web-based reports
 Extended software is enhanced
 Platform : Windows 2000/XP, Solaris 8/9, HP-UX
APAN-JP NOC monitors its network using OpenView mainly!
2010年8月16日月曜日

Monitoring Software- HP OpenView Sample 1 OpenView Contracture

Event log

Network map
ICMP polling forconnectivity check

Network sub-map

2010年8月16日月曜日

Router map

Monitoring Software- HP OpenView Sample 2 OpenView Tools

Event configuration

Snmp configuration for polling- parameters- community

Data collection & Thresholds for SNMP

2010年8月16日月曜日

Monitoring Software- Nagios OverviewNagios ®
Freely available from http://www.nagios.org

OverviewA host and service monitor designed to
inform you of network and end system problems
Provides simple ping availability of resources on the
network
Works with a set of “plugins” to provide local and remote
host service status
Custom “plugins” are relatively easy to develop
Web-based monitoring system
Platform : Linux, UNIX

APAN-JP NOC uses Nagios as secondary monitoring system!

2010年8月16日月曜日

Monitoring Software- Nagios Sample 1Nagios
Service Overview For All Host Groups

Service Status Details For All Hosts

2010年8月16日月曜日

Monitoring Software- Nagios Sample 2Nagios
Network Map For All Hosts

Event log

2010年8月16日月曜日

MRTG (Multi-Router Traffic Grapher) Overview
Monitors the load of network equipment using SNMP, mainly used for
creation of traffic graphExcellent graphing tool developed by Tobias Oetiker
Plots graph with any two variables against time, It is graph-ized with
PNG format on HTML pageAble to create scripts to feed data into MRTG
Implements data collection, image, web-page collection
Very widely deployed in large networks and still being actively
developedPlatform : UNIX system / Windows NT
Supports SNMPv2 : able to read 64bit countershttp://people.ee.ethz.ch/~oetiker/webtools/mrtg/

2010年8月16日月曜日

MRTG - Workflow  Display of graphGreen area typically represents
incoming maximum bits per second
Blue line typically represents outgoing
maximum bits per second

Workflow
1.Read configuration file
2.Collect graphing data from network equipment, based on configuration
3.Update database file and generate graph
4.If required, generate HTML file
MRTG performs above workflow then completesSince MRTG collects data of the past 5 minutes (default value of source code),
it is desirable to set “crontab” for every 5 minutes

2010年8月16日月曜日

MRTG - Data Storage Data Storage
Keeps 5 minute data only for 2.5
days. The data is thrown away
afterward.

Daily grafh/5min

There is no referring to historical data with highresolution
Keeps 1-day data for approx. 2 years
Weekly grafh/30min

Monthly grafh/2hours

Rougher
Resolution
Yearly grafh/1day

2010年8月16日月曜日

Interval

Num of record

Storage periodGraph

5 minutes

600

2.5 daysdaily

30 minutes

600

12.5 days

Weekly2 hours

600

50 daysMonthly

1 day

731

2 years

YearlyMRTG - Configuration 1 MRTG Configuration
cfgmaker
Helps to create configuration file form
Example
cfgmaker -global ‘WorkDir: /home/httpd/html/mrtg’ \-global "Options[_]: bits,growright’ \-output /home/httpd/html/mrtg/cfg/mrtg.cfg \ncommunity@router.domain.name

Graph & log data: /home/httpd/html/mrtg
Configuration file: /home/https/html/cfg/mrtg.cfg
Option : unit = bits(bps), Horizontal axis = grow right way

Detailed information
http://people.ee.ethz.ch/~oetiker/webtools/mrtg/cfgmaker.html

2010年8月16日月曜日

MRTG - Configuration 2 Target Configuration
Target Expression
Target[<target name>]:<target kind>:<community>@<address>
 <target name> : Identify equipment
 <target kind> : Measurement item
 <community> : SNMP community string <address> : Hostname or IP address of equipment

SNMP data collection specification method
Basic / Port (ifindex)
Target[myrouter]: 2:public@wellfleet-fddi.ethz.ch

Explicit OIDs / MIB Variables
Target[myrouter]: 1.3.6.1.2.1.2.2.1.14.1&1.3.6.1.2.1.2.2.1.20.1:public@myrouter

Target[myrouter]: ifInErrors.1&ifOutErrors.1:public@myrouter

You can use cfgmaker to generate references with the options-- ifref=?
iｆref=ip: Interface by IP
ifref=descrf: Interface by Descriptionifref=name: Interface by Name
ifref=eth: Interface by Ethernet Address

2010年8月16日月曜日

MRTG - Configuration 3 Example of Configuration
Target[la]: ifHCInOctets\so-2/0/0&ifHCOutOctets\so-2/0/0:xxxxxxx@tpr2.jp.apan.net:::::2
MaxBytes[la]: 300000000
Title[la]: Traffic Analysis of TransPAC LA Link
PageTop[la]: <H1>Traffic Analysis of TransPAC LA link</H1>
WithPeak[la]: ymw
Directory[la]: tpr2
Options[la]: bits, growrightTarget[la-err]: ifInErrors\so-2/0/0&ifOutErrors\so-2/0/0:xxxxxxx@tpr2.jp.apan.net
MaxBytes[la-err]: 300000000
Title[la-err]: Packet Error for TransPAC LA link
PageTop[la-err]: <H1>Packet Error for TransPAC LA link</H1>
Directory[la-err]: tpr2
Options[la-err]: growright, integer, nopercent
YLegend[la-err]: Number of Error Packets
ShortLegend[la-err]: nLegend1[la-err]: Number of Error Packets for Incoming Traffic
Legend2[la-err]: Number of Error Packets for Outgoing Traffic
Legend3[la-err]: Peak of Number of Error Packets for Incoming Traffic
Legend4[la-err]: Peak of Number of Error Packets for Outgoing Traffic
LegendI[la-err]: &nbsp;In:
LegendO[la-err]: &nbsp;Out:
WithPeak[la-err]: w

2010年8月16日月曜日

MRTG - Comments Comments / Disadvantages
 If you are to monitor a lot of devices (1000s), it is better to have afast disk
 If using external monitoring scripts, a fast processor and a lot ofmemory is necessary Not particularly fast when compared to other data retrieval andstorage schemes (Flat text files can slow down processing.)
 MRTG can’t customize graphing periods
 Flat text files are difficult to process when scripting against the data
 Use 64bit counters with SNMPv2 for OC3-OC192 speed interface,
GbE if it is 115Mbps traffic can wrap 32bit counters around in 5
minutes
 MRTG can’t modify collected data which is summarized Only two variables are available in processing a graph2010年8月16日月曜日

RRDtool (Round Robin Database Tool)
Overview Successor to MRTG
 Developed by the same developer of MRTG : Tobias Oetiker Tool group for RRD can flexibly define data item, time interval, dataamount, graph depiction, etc.
 Binary file format that can store data at any interval for any lengthof time File does not grow in size over time Ability to make custom graphs across user-defined intervals
 Ability to graph multiple variables on a single graph Additional scripts are necessary in creating graphs and web-page 25-30 percent faster than MRTG

 Does not have the function to collect data
 http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/

2010年8月16日月曜日

RRDtool - Architecture Comparison of architecture between MRTG and RRD
SNMPengineGraphrouterIndex
log
Frontend
ProgramFrontend
Programrouterserver

Index
RRD

text

2010年8月16日月曜日

Graph

RRDtool - Basic Usage Basic usage of RRD tools
Set up new Round Robin Database (RRD) ・・・①
 Define RRD used as vessel of data Command : rrdtool create filename

Store new set of values into RRD periodically ・・・②
 Write the data collected by frontend program in RRD
 Command : rrdtool update filename

Generate Graph ・・・③
 Create graph from data stored in one or several RRDs
 Command : rrdtool graph filename (specify the graph name to generate)
・・・・・ data

data

data ②③①

2010年8月16日月曜日

RRD

Graph

RRDtool - Practice Example
Object
 Gigabit Ethernet Switch

Definition
 Definition of RRD record
Interval

Num of RRD
file

Storage

Graph

Period1 minute360

6 hours

4 hours

5 minutes

576

2 daysDaily

2 hours

600

50 daysMonthly

1 day

731

2 years

Yearly4 days915

10 years

10 years Ability to describe peak graph from data of 1-day to 10-years

2010年8月16日月曜日

RRDtool - Create Set up a new Round Robin Database (RRD)
Command Example/usr/local/rrdtool-1.0.46/bin/rrdtool create \/home/httpd/html/traffic/traffic_vlan.rrf \–step 60 \DS:vlan2in:counter60:0:125000000 \DS:vlan2out:counter60:0:125000000 \DS:vlan7in:counter60:0:125000000 \DS:vlan7out:counter60:0:125000000 \:RRA:AVERAGE:0.5:1:360 \RRA:AVERAGE:0.5:5:576 \RRA:AVERAGE:0.5:120:600 \RRA:AVERAGE:0.5:1440:731 \RRA:AVERAGE:0.5:5760:915 \RRA:MAX:0.2:5:576 \RRA:MAX:0.1:120:600 \RRA:MAX:0.1:440:731 \RRA:MAX:0.1:5760:915 \

2010年8月16日月曜日

DS : Define the data item

COUNTER: continuous increasing counters60 : if no new data is supplied for more than 60
sec, it is considered as “unknown”
0 : minimum acceptable value (byte)
125000000 : maximum acceptable value (byte)

RRA (Round Robin Archive) : Define
the data consolidations

AVARAGE/MAX: average /maximum of consolidated ofdata
0.5 : consolidation interval is be made up from*UNKNOWN* data while the consolidated value is still
regarded as known.
- Average 50%. MAX 20% or 10%
1: consolidated data point where the data then goes
into the archive
360 : how many generations of data values are kept
in RRARRDtool - Update Stores a new set of values into RRD periodically Data collection
 Collect the data from targets using frontend program Original tool
 Cricket - http://cricket.sourceforge.net/
 Orca - http://www.orcaware.com/orca/
 SNAPP - http://sourceforge.net/projects/snapp/

 Updating an RRD
 Feed collected data into a RRD database using following commands
Command Example
rrdtool update /home/httpd/html/traffic/traffic_vlan.rrd\--template in:out N:11222:1

DS1: DS2
The data sources are defined in the RRD

2010年8月16日月曜日

The name of the RRD you want to update.

‘N’=Update time is set to be the current time

RRDtool - Graph 1 Generating Graph -1Command Example
rrdtool graph /home/httpd/html/traffic/traffic.png -s -4h –w 800 –h 800 –a PNG \–t “VLAN Traffic” –v “bit/s” \DEF:vlan2in_ave=/home/httpd/html/traffic/traffic_vlan.rrd:vlan2in:AVERAGE \DEF:vlan2out_ave=/home/httpd/html/traffic/traffic_vlan.rrd:vlan2out:AVERAGE \DEF:vlan7in_ave=/home/httpd/html/traffic/traffic_vlan.rrd:vlan7out:AVERAGE \DEF:vlan7in_ave=/home/httpd/html/traffic/traffic_vlan.rrd:vlan7out:AVERAGE \CDEF:vlan2in_ave_bit=vlan2in_ave,8 * \CDEF:vlan7in_ave_bit=vlan7in_ave,8 * \CDEF:vlan2out_ave_bit=vlan2out_ave,-8 * \CDEF:vlan7out_ave_bit=vlan7out_ave,-8 * \AREA:vlan2in_ave_bit#ff5e5e:VLAN2-in \STACK:vlan7in_ave_bit#5eff5e:VLAN7-in \AREA:vlan2out_ave_bit#aa0101:VLAN2-out \STACK:vlan7out_ave_bit#0101aa:VLAN7-out \Options-s: start time (default : seconds), -e: end seconds (default : seconds),-w,h : width and height pixels, -a : image format GIF|PNG, -t : Graph title,-v vertical-label text

2010年8月16日月曜日

RRDtool - Graph 2 Generating a Graph -2DEF
Define virtual name for data source DEF:<vname>=<RRDfilename>:<DS-name>:CF
CF: consolidation function
select AVARAGE, MAX, MIN, LAST ( Newest data)

CDEF
Create new virtual data source by evaluating mathematical expression
 CDEF:<vname>=rpn-expression (Reverse Polish Notation)

Graph depiction parameter<Style>:<vname>#<color>:<legend>
LINE : Plot for the request data, using the color specified
AREA : Area between 0 line and the graph line will be filled with the color specified
STACK : Graph gets stacked on top of the previous LINE, AREA, or STACK graphBy updating graph generation periodically using “crontab”, you cansee updated graphs on the Web

2010年8月16日月曜日

RRDtool - Sample Sample Graph

http://mrtg.jp.apan.net/cricket/router-interfaces/

2010年8月16日月曜日

BWCTL (Bandwidth Control)
Users attempting to run bandwidth tests used to be notcertain whether or not their test was scheduled in a timeframe where other tests were not to run
BWCTL is a resource allocation and scheduling daemon for arbitrationof iperf tests
BWCTL client application works by contacting a bwctld process on
both endpoints of test systems
Requires that NTP be running to synchronize the system clock
Open mode : everyone can use
Authentication mode : need to exchange AES key
Support IPv6, Platform : UNIX systems
Developed by Internet2 http://e2epi.internet2.edu/bwctl/

2010年8月16日月曜日

OWAMP (One-way Active Measurement Protocol)
Roundtrip-based measurement can not identify the delay in eachdirection, especially when asymmetric routes are usedOWAMP is a command line client application and a policy daemon used
to determine one way latencies between hosts
It is possible to collect active measurement data
- e.g., one-way delay, packet loss, jitter
NTP must be setup correctly on the system to calculate a reasonableestimate of time error and to stabilize clockSupport IPv6. Platform : UNIX systems
Current Draft : draft-ietf-ippm-owdp-10.txt
Developed by Internet2 http://e2epi.internet2.edu/owamp/
icmp ping : RTT

2010年8月16日月曜日

owping : one-way

OWAMP - Protocol Consists of two inter-related protocols
OWAMP-Control
Used to initiate, start/stop test sessions, and fetch test results

OWAMP-Test
Define the format of probe packet

Sample measurement data
http://pe2.koganei.wide.ad.jp/cgi-bin/owd-stat
http://qpe.jp.apan.net/cgi-bin/owd-stat
2010年8月16日月曜日

Netflow - Overview Overview
Enables IP traffic flow analysis without probes
Invented and patented by Cisco
Juniper (called cflowd), Foundry, ･･･ many venders are supporting

Flow cash data on routers is exported
to a flow tool, so that traffic flow is to be analyzed
Enable NetFlow

flow Definition:

Traffic

Source IP addressDestination IP addressSource port
Destination port
Layer 3 protocol type
TOS byte (DSCP)
Input logical interface(ifIndex)Core Network

UDP
NetFlow
ExportPackets

Collector(Solaris, HP-UX, or Linux)

2010年8月16日月曜日

Application GUI

Netflow - Flow Data Flow data export
Enable NetFlow on the routerThere is difference in architecture between Cisco and Juniper routers
Take care! the load of a router does not become high!
- Check CPU, memory, bandwidth, sampling rate

Flow data collection & Analysis
Prepare the software for receiving flow-export dataflow-tools http://www.splintered.net/sw/flow-tools/
cflowd http://www.caida.org/tools/measurement/cflowd/
Cisco : NetflowCollector

Analyze traffic from raw data with software
flow-scan http://net.doit.wisc.edu/~plonka/FlowScan/(If you want to graph-ize analysis data, I recommend you to use RRDtool)Cisco : CiscoWorks
 Source and destination IP address Source and destination TCP/UDP ports
 Packet and byte counts
 Routing information (next-hop address, source autonomous system (AS) number,destination AS number, source prefix mask, destination prefix mask)

2010年8月16日月曜日

Netflow - Example Netflow Example

2010年8月16日月曜日

Introduction of other advanced tools

Abilene Router Proxy - Overview  Similar to Looking Glass, but with some advanced functions
 Web-form allows users to submit various commands to backbone
routers
 Allows remote network operators to troubleshoot problems without
contacting NOC
 Unix-based Uses scripted telnet to login to the routers and grab the output
 Not designed for high-speed access to backbone information
 Very useful operation tool among inter-domain network
 Enable us to view operational situation of almost all Abilene routers

 APAN Tokyo XP Router Proxy Service
 http://monitor.jp.apan.net/routerproxy/

2010年8月16日月曜日

perfSONAR
2010年8月16日月曜日

July	
  7th	  2010,	  perfSONAR	  Workshop	
  –	
  perfSONAR	  Tutorial
Jason	
  Zurawski,	  Network	
  SoAware	
  Engineer,	
  Research	
  Liason

perfSONAR	
  Preliminaries

2010年8月16日月曜日

Outline• MoGvaGon– A	
  Typical	
  Scenario– Possible	  SoluGons• What	  is	
  perfSONAR?
– IncepGon– Architecture	
  Primer– Example	
  Use	
  Case• Who	  is	
  involved	
  in	
  perfSONAR?
– perfSONAR-­‐MDM– perfSONAR-­‐PS• Who	  is	
  adopGng	
  perfSONAR?
• PracGcal	
  InformaGon• Workshop	
  Overview3	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Why	
  Worry	
  About	
  Network	
  Performance?
• Most	  network	
  design	
  lends	
  itself	  to	
  the	
  introducGon	
  of	
  ﬂaws:– Heterogeneous	
  equipment– Cost	  factors	
  heavily	
  into	  design	
  –	
  e.g.	
  Get	
  what	  you	
  pay	
  for
– Design	
  heavily	
  favors	
  protec'on	
  and	  availability	
  over	  performance• CommunicaGon	
  protocols	
  are	  not	
  advancing	
  as	
  fast	
  as	
  networks– TCP/IP	
  is	
  the	
  king	  of	
  the	
  protocol	
  stack• Guarantees	
  reliable	
  transfers• Adjusts	
  to	
  failures	
  in	
  the	  network• Adjusts	
  speed	
  to	
  be	
  fair	  for	
  all

• User	  ExpectaGons– Big	
  Science	
  is	
  prevalent	
  globally– The	  “8	
  Second	  Rule”	  is	
  present	
  in	
  ScienGﬁc	
  CommuniGes	
  too	
  [1]

4	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

MoGvaGon	
  –	
  A	
  Typical	
  Scenario• User	  and	  resource	
  are	
  geographically	
  separated• Both	  have	  access	
  to	
  high	  speed	
  communicaGon	
  network– LAN	
  infrastructure	
  -­‐	
  1Gbps	  Ethernet– WAN	
  infrastructure	
  –	
  10Gbps	
  OpGcal	  Backbone5	  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

MoGvaGon	
  –	
  A	
  Typical	
  Scenario• User	  wants	
  to	
  access	
  a	
  ﬁle	  at	  the	  resource	
  (e.g.	
  ~600MB)• Plans	  to	
  use	
  COTS	
  tools	
  (e.g.	
  “scp”,	
  but	
  could	
  easily	
  be	
  something	  scienGﬁc	
  like	  “GridFTP”	
  or	
  simple	  like	  a	
  web	
  browser)• What	  are	  the	
  expectaGons?
– 1Gbps	
  network	
  (e.g.	
  bo>leneck	
  speed	  on	
  the	  LAN)– 600MB	
  *	
  8	
  =	
  4,800	
  Mb	
  ﬁle
– User	  expects	
  line	  rate,	  e.g.	
  4,800	
  Mb	
  /	
  1000	  Mbps	
  =	
  4.8	
  Seconds– Audience	  Poll:	  Is	
  this	
  expectaGon	
  too	
  high?

• What	  are	  the	
  realiGes?
– CongesGon	
  and	
  other	  network	
  performance	
  factors– Host	  performance– Protocol	
  Performance– ApplicaGon	
  performance6	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

MoGvaGon	
  –	
  A	
  Typical	
  Scenario• Real	  Example	  (New	
  York	  USA	  to	
  Los	
  Angeles	
  USA):• 1MB/s	
  (8Mb/s)	
  ???	  10	  Minutes	
  to	
  transfer???
• Seems	
  unreasonable	
  given	  the	
  investment	
  in	
  technology– Backbone	
  network– High	  speed	
  LAN– Capable	
  hosts• Performance	  realiGes	
  as	
  network	
  speed	
  decreases:– 100	
  Mbps	  Speed	  –	
  48	
  Seconds– 10	
  Mbps	  Speed	
  –	
  	
  8	
  Minutes– 1	
  Mbps	  Speed	  –	
  80	
  Minutes• How	
  could	
  this	  happen?	
  	
  More	  importantly,	
  why	  are	  there	  not	  more	  complaints?
• Audience	  Poll:	
  Would	
  you	
  complain?	
  	
  If	
  so,	  to	
  whom?
• Brainstorming	  the	
  above	
  –	
  where	
  should	
  we	
  look	  to	
  ﬁx	
  this?
7	  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

MoGvaGon	
  –	
  A	
  Typical	
  Scenario• ExpectaGon	
  does	  not	
  even	  come	  close	  to	
  experience,	
  Gme	  to	
  debug.	
  	
  Where	  to	
  start	  though?
– ApplicaGon• Have	  other	
  users	
  reported	
  problems?	
  	
  Is	
  this	  the	
  most	  up	
  to	
  date	  version?

– Protocol• Protocols	
  typically	
  can	  be	
  tuned	
  on	
  an	
  individual	
  basis,	  consult	
  your	  operaGng	
  system.	
  	
  – Host• Are	  the	
  hardware	
  components	
  (network	
  card,	  system	
  internals)	
  and	
  soAware	
  (drivers,	
  operaGng	
  system)	
  funcGoning	
  as	
  they	  should	
  be?

– LAN	  Networks	  • Consult	
  with	  the	
  local	
  administrators	
  on	
  status	
  and	
  potenGal	
  choke	
  points– Backbone	
  Network• Consult	
  the	
  administrators	
  at	
  remote	
  locaGons	
  on	
  status	
  and	
  potenGal	
  choke	
  points	  (Caveat	
  –	
  do	
  you	  [should	
  you]	
  know	
  who	  they	  are?)	
  8	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

MoGvaGon	
  –	
  A	
  Typical	
  Scenario• Following	
  through	  on	
  the	  previous,	
  what	  normally	
  happens	
  …
– Applica'on• This	  step	
  is	
  normally	
  skipped,	
  the	
  applicaGon	
  designer	
  will	  blame	  the	  network– Protocol• These	  sepngs	
  may	  not	  be	
  explored.	
  	
  Shouldn’t	
  this	  be	
  automaGc	
  (e.g.	
  autotuning)?

– Host• Checking	
  and	  diagnosGc	
  steps	
  normally	
  stop	
  aAer	  establishing	
  connecGvity.	
  	
  
E.g.	
  “can	  I	
  ping	  the	
  other	  side”– LAN	  Networks	  • Will	  assure	
  “internal”	
  performance,	
  but	  LAN	
  administrators	
  will	  ignore	
  most	  user	
  complaints	
  and	
  shiA	  blame	  to	
  upstream	
  sources.	
  	
  E.g.	
  “our	  network	
  is	
  
ﬁne,	
  there	  are	  no	
  complaints”– Backbone	
  Network• Will	  assure	
  “internal”	
  performance,	
  but	  Backbone	
  responsibiliGes	
  normally	
  
stop	
  at	
  the	
  demarcaGon	
  point,	
  blame	  is	
  shiAed	
  to	
  other	  networks	
  up	
  and	  down	
  stream9	  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

MoGvaGon	
  –	
  A	
  Typical	
  Scenario• Stumbling	
  Blocks	
  to	
  solving	  performance	
  problems– Lack	  of	  a	
  clear	
  process• Knowledge	
  of	
  the	
  proper	
  order	
  to	
  approach	
  problems	
  is	
  paramount• This	  knowledge	
  is	
  not	
  just	  for	
  end	
  users	
  –	
  also	
  for	
  applicaGon	
  developers	
  and	
  network	
  operators	
  too– Impa'ence• Everyone	
  is	
  impaGent,	
  from	  the	
  user	  who	
  wants	
  things	
  to	
  work	  to	
  the	  network	
  staﬀ	
  and	
  applicaGon	
  developers	
  who	  do	
  not	
  want	  to	
  hear	  complaints– Informa'on	
  Void• Lack	  of	  a	
  clear	
  locaGon	
  that	  describes	
  symptoms	
  and	
  steps	  that	  can	  be	
  taken	  to	
  miGgate	
  risks	
  and	
  solve	
  problems• Lack	  of	
  available	
  performance	
  informaGon,	
  e.g	  the	
  current	
  status	
  of	
  a	
  given	
  network	  in	  a	
  public	
  and	
  easily	
  accessible	
  forum	  – Communica'on• Finding	
  whom	  to	
  contact	
  to	
  report	
  problems	
  or	
  get	
  help	
  in	
  debugging	
  is	
  frustraGng
10	  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

MoGvaGon	
  –	
  Possible	  SoluGons• Finding	  a	
  soluGon	
  to	
  network	
  performance	
  problems	
  can	  be	
  broken	
  into	
  two	  disGnct	
  steps:– Use	  of	
  DiagnosDc	
  Tools	
  to	
  locate	
  problems• Tools	
  that	  acGvely	
  measure	
  performance	
  (e.g.	
  Latency,	
  Available	  Bandwidth)• Tools	
  that	  passively	
  observe	
  performance	
  (e.g.	
  error	
  counters)– Regular	
  Monitoring	
  to	
  establish	
  performance	
  baselines	
  and	
  alert	  when	
  expectaGon	
  drops.
• Using	
  diagnosGc	
  tools	
  in	
  a	
  structured	  manner• VisualizaGons	
  and	
  alarms	
  to	
  analyze	
  the	  collected	
  data• IncorporaGon	
  of	
  either	
  of	
  these	  techniques	
  must	  be:– ubiquitous,	
  e.g.	
  the	
  soluGon	
  works	
  best	  when	  it	
  is	
  available	
  everywhere– seamless	
  (e.g.	
  federated)	
  in	
  presenGng	
  informaGon	
  from	  diﬀerent	
  resources	
  and	
  domains11	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

MoGvaGon	
  –	
  Possible	  SoluGons• Desirable	
  design	
  features	
  for	
  any	
  soluGon– Component	  Based• FuncGonality	
  should	
  be	
  split	
  into	  logical	
  units• Each	  funcGon	
  (e.g.	
  visualizaGon)	
  should	
  funcGon	
  through	
  well	  deﬁned	
  communicaGon	
  with	  other	  components	
  (e.g.	
  data	  storage)– Modular• Monolithic	
  designs	
  rarely	
  work• Components	
  allow	
  choice	  of	
  how	  to	
  operate	  a	
  customized	
  end	
  soluGon.	
  	
  – Accessible• Well	  deﬁned	
  interfaces	
  (e.g.	
  APIs)	
  • IniGal	
  design	
  should	
  facilitate	
  future	
  expansion

12	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

MoGvaGon	
  –	
  Possible	  SoluGons

13	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

What	  is	
  perfSONAR?
• Most	  organizaGons	
  perform	
  monitoring	
  and	  diagnosGcs	
  of	
  their	
  own	  network– SNMP	  Monitoring	
  via	
  common	
  tools	
  (e.g.	
  MRTG,	
  CacG)– Enterprise	
  monitoring	
  (e.g.	
  Nagios)• Networking	
  is	
  increasingly	
  a	
  cross-­‐domain	  eﬀort– InternaGonal	
  collaboraGons	
  in	
  many	  spaces	
  (e.g.	
  science,	
  the	
  arts	  and	  humaniGes)	
  are	
  common– Interest	
  in	
  development	
  and	
  use	  of	
  R&E	  networks	
  at	
  an	
  all	
  Gme	
  high• Monitoring	
  and	  diagnosGcs	
  must	  become	
  a	
  cross-­‐domain	
  eﬀort– Complete	
  view	
  of	
  all	
  paths– Eliminate	
  “who	  to	
  contact”	
  and	
  “what	
  to	
  ask	
  for”	  -­‐	
  24/7	
  availability	
  of	
  diagnosGc	
  observaGons14	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

What	  is	
  perfSONAR?
• A	
  collaboraGon– Produc'on	
  network	
  operators	
  focused	
  on	
  designing	
  and	
  building	
  tools	
  that	  they	
  will	  deploy	
  and	
  use	  on	  their	
  networks	
  to	
  provide	  monitoring	
  and	  diagnos'c	
  capabili'es	
  to	
  themselves	
  and	  their	
  user	  communi'es.

• An	
  architecture	
  &	
  set	  of	
  communicaGon	
  protocols– Web	
  Services	
  (WS)	
  Architecture– Protocols	
  established	
  in	
  the	
  Open	  Grid	  Forum• Network	
  Measurement	
  Working	
  Group	
  (NM-­‐WG)• Network	
  Measurement	  Control	
  Working	
  Group	
  (NMC-­‐WG)• Network	
  Markup	
  Language	
  Working	
  Group	
  (NML-­‐WG)• Several	
  interoperable	
  soAware	
  implementaGons– perfSONAR-­‐MDM– perfSONAR-­‐PS• A	
  Deployed	
  Measurement	
  infrastructure

15	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

perfSONAR	  IncepGon• perfSONAR	
  originated	
  from	  discussions	
  between	  Internet2’s	
  End-­‐to-­‐End	
  Performance	
  IniGaGve	
  (E2Epi),	
  and	  the	
  Géant2	
  project	
  in	
  September	
  2004.	
  	
  • Members	
  of	
  the	
  OGF’s	
  (then	
  GGF)	
  NM-­‐WG	
  provided	
  guidance	
  on	
  
the	
  encoding	
  of	
  network	
  measurement	
  data.
• AddiGonal	
  network	
  partners,	
  including	
  ESnet	  and	
  RNP	
  provided	
  development	
  resources	
  and	  served	
  as	
  early	
  adopters.	
  	
  • The	
  ﬁrst	  release	
  of	
  perfSONAR	
  branded	
  soAware	
  was	  available	
  in	
  
July	
  2006	
  (Java	
  based	
  soAware).
• All	  perfSONAR	
  branded	
  soAware	
  is	
  open	
  source• All	
  products	
  looking	
  to	
  be	
  labeled	  as	
  perfSONAR	
  compliant	
  must	  establish	
  protocol	
  compliance	
  based	  on	
  the	
  public	
  standards	
  of	
  the	
  
OGF	
  

16	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

perfSONAR	  Architecture	
  Overview• Interoperable	
  network	
  measurement	
  middleware	
  designed	
  as	
  a	
  Service	
  Oriented	  Architecture	
  (SOA):– Each	  component	
  is	
  modular– All	  are	
  Web	
  Services	
  (WS)	
  based– The	
  global	  perfSONAR	
  framework	
  as	
  well	  as	
  individual	
  deployments	
  are	  decentralized– All	  perfSONAR	
  tools	  are	
  Locally	
  controlled– All	  perfSONAR	
  tools	  are	
  capable	
  of	
  federa'ng	
  locally	
  and	
  globally• perfSONAR	
  Integrates:– Network	
  measurement	
  tools	
  and	
  archives	
  (e.g.	
  stored	
  measurement	  results)– Data	  manipula'on– Informa'on	
  Services• Discovery• Topology– Authen'ca'on	
  and	
  authoriza'on
17	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

perfSONAR	  Architecture	
  Overview• The	
  key	  concept	
  of	
  perfSONAR	
  is	
  that	  each	  enGty	
  (e.g.	
  “services”)	
  performs	
  a	
  funcGon– Each	  service	
  provides	
  a	
  limited	
  set	  of	
  func'onality,	
  e.g.	
  collec'ng	  measurements	
  between	
  arbitrary	
  points	
  or	
  managing	
  the	
  registra'on	
  and	  loca'on	  of	
  distributed	
  services– The	  service	
  is	
  a	
  self	
  contained	
  and	
  provides	
  func'onality	
  on	
  its	
  own	  as	
  well	  as	
  when	  deployed	
  with	  the	
  remainder	
  of	
  the	
  framework• Services	
  interact	
  through	
  exchanges– Standardized	
  message	
  formats– Standardized	
  exchange	
  paSerns	
  (e.g.	
  a	
  communica'on	
  protocol)• A	
  collecGon	
  of	
  perfSONAR	
  services	
  within	
  a	
  domain	
  is	
  a	
  deployment– Deploying	
  perfSONAR	
  can	  be	
  done	  À	
  la	
  carte,	
  or	
  through	  a	
  complete	
  solu'on• Services	
  federate	
  with	  each	  other,	
  locally	
  and	
  globally– Services	
  are	  designed	
  to	
  automa'cally	
  discover	
  the	  presence	
  of	
  other	
  perfSONAR	
  components– Clients	
  are	  designed	
  with	  this	  distributed	
  paradigm	
  in	
  mind18	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

perfSONAR	  Architecture	
  Overview

19	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

perfSONAR	  Architecture	
  Overview• A	
  perfSONAR	
  deployment	
  can	  be	
  any	
  combinaGon	
  of	
  services– An	
  instance	
  of	
  the	
  Lookup	
  Service	
  is	
  required	
  to	
  share	  informaGon– Any	
  combinaGon	
  of	
  data	  services	
  and	
  analysis	
  and	
  visualizaGon	
  tools	
  is	
  possible• perfSONAR	
  services	
  have	  the	
  ability	
  to	
  federate	
  globally– The	
  Lookup	
  Service	
  communicates	
  with	  a	
  confederated	
  group	  of	
  directory	
  services	
  (e.g.	
  the	
  Global	
  Lookup	
  Service)– Global	
  discovery	
  is	
  possible	
  through	
  APIs• perfSONAR	
  is	  most	  eﬀecGve	
  when	  all	
  paths	
  are	  monitored– Debugging	
  network	
  performance	
  must	  be	
  done	  end-­‐to-­‐end– Lack	  of	
  informaGon	
  for	
  speciﬁc	
  domains	
  can	
  delay	  or	
  hinder	  the	
  debug	
  process20	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Many collaborations are
inherently multi-domain, so foran end-to-end monitoring tool
to work everyone must
participate in the monitoring
infrastructureuser
performance GUI

m1

m1
m4

measurement
archive

m1

m3

m4

GEANT (AS20965)[Europe]m4

measurement
archive

m1

m4

m3
m1

m4

DESY (AS1754)[Germany]m3

ESnet (AS293)
[US]
21	
  –	
  07/27/10,	
  ©	
  2010	  Internet2

2010年8月16日月曜日

measurement
archive

m3

m3

FNAL (AS3152)[US]

Analysis tool

measurement
archivemeasurement
archive

DFN (AS680)[Germany]

21

Example	
  perfSONAR	
  Use	
  Case• perfSONAR	
  should	
  be	
  used	  to	
  diagnose	
  an	
  end-­‐to-­‐end	
  performance	
  problem– User	  is	
  ayempGng	
  to	
  download	
  a	
  remote	  resource– Resource	
  and	
  user	  are	
  separated	
  by	
  distance– Both	  are	
  assumed	
  to	
  be	
  connected	
  to	
  
high	
  speed	
  networks• OperaGon	
  does	  not	  go	  as	
  planned,	
  where	  to	
  start?

22	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Example	
  perfSONAR	
  Use	
  Case• Simple	
  tools	
  like	  traceroute	
  can	  be	
  used	  to	
  determine	
  the	
  path	  traveled• There	
  could	
  be	  a	  performance	
  problem	
  anywhere	
  in	
  here• The	
  problem	
  may	  be	
  something	
  we	
  could	
  ﬁx,	
  but	  the	
  chances	
  are	
  greater	
  that	
  it	
  is	
  
not

23	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Example	
  perfSONAR	
  Use	
  Case• Each	  segment	
  of	
  the	
  path	  is	
  controlled	
  by	
  
a	
  diﬀerent	
  domain.
• Each	  domain	
  will	  have	  network	
  staﬀ	
  that	  could	
  help	
  ﬁx	
  the	
  problem,	
  but	
  how	  to	
  contact	  them?
• All	  we	
  really	
  want	  is	
  some	  informaGon	
  regarding	  performance

24	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Example	
  perfSONAR	
  Use	
  Case• Each	  domain	
  has	
  made	  measurement	
  data	
  available	
  via	  perfSONAR• The	
  user	  was	
  able	  to	
  discover	
  this	  automaGcally• Automated	
  tools	
  such	  as	
  visualizaGons	
  and	
  analyzers	
  can	  be	
  powered	  by	
  this	  network	
  data25	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Example	
  perfSONAR	
  Use	
  Case• In	  the	
  end,	
  the	
  problem	
  is	
  isolated	
  based	  on	
  tesGng.
– May	  have	  gone	  unnoGced	
  in	
  some	  cases	  (e.g.	  a	
  “soA	  failure”)– Could	
  have	  been	
  observed	
  by	
  many	  others	
  …	
  that	  didn’t	
  think	  to	
  report	
  it• The	
  user	  (or	
  operaGons	
  staﬀ)	
  can	  contact	
  the	
  domain	  in	
  quesGon	
  to	
  inquire	
  about	
  this	
  performance	
  problem• When	  ﬁxed	  the	  transfer	
  should	
  progress	
  as	
  intended

26	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Who	
  is	
  perfSONAR?
• The	  perfSONAR	
  ConsorGum	
  is	
  a	
  joint	
  collaboraGon	
  between	  – ESnet– Géant– Internet2– Rede	  Nacional	
  de	
  Ensino	
  e	
  Pesquisa	  	
  (RNP)• Decisions	
  regarding	
  protocol	
  development,	
  soAware	
  branding,	
  and	  interoperability	
  are	
  handled	
  at	
  this	  organizaGon	
  level• There	  are	  at	
  least	
  two	  independent	
  eﬀorts	
  to	
  develop	
  soAware	
  frameworks	
  that	
  are	  perfSONAR	
  compaGble.	
  	
  – perfSONAR-­‐MDM– perfSONAR-­‐PS– Others?	
  	
  The	
  beauty	
  of	
  open	
  source	
  soAware	
  is	
  we	
  will	  never	
  know	
  the	
  full	  extent!

• Each	  project	
  works	
  on	
  an	
  individual	
  development	
  roadmap	
  and	
  works	
  with	  the	
  consorGum	
  to	
  further	
  protocol	
  development	
  and	
  insure	
  compaGbility27	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Who	
  is	
  perfSONAR-­‐MDM?
• perfSONAR-­‐MDM	
  is	
  made	  up	
  of	
  parGcipants	
  in	
  the	
  Géant	
  project:	
  •Arnes•Belnet•Carnet•Cesnet•CYNet•DANTE•DFN•FCCN•GRNet

•GARR
•ISTF
•PSNC
•Nordunet (Uninett)•Renater•RedIRIS•Surfnet•SWITCH• perfSONAR-­‐MDM	
  is	
  wriyen	  in	
  Java	  primarily	
  and	  was	  designed	
  to	
  serve	  as	
  the	  monitoring	
  soluGon	
  for	  the	  Large	  Hadron	
  Collider	
  (LHC)	
  project.	
  	
  • perfSONAR-­‐MDM	
  is	
  available	
  as	
  Deb	  (Debian	  CompaGble)	
  and	
  RPM	  (Red	  Hat	  CompaGble)	
  packages.
28	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Who	
  is	
  perfSONAR-­‐PS?
• perfSONAR-­‐PS	
  is	
  comprised	
  of	
  several	
  members:– ESnet– Fermilab– Georgia	
  Tech– Indiana	
  University– Internet2– SLAC– The	
  University	
  of	
  Delaware• perfSONAR-­‐PS	
  products	
  are	
  wriyen	  in	
  the	  perl	  programming	  language	
  and	  are	  available	
  for	
  installaGon	
  via	
  source	
  or	
  RPM	
  (Red	  Hat	
  CompaGble)	
  packages• perfSONAR-­‐PS	
  is	
  also	
  a	
  major	
  component	
  of	
  the	  pS	  Performance	  Toolkit	  –	
  A	
  bootable	
  Linux	
  CD	
  containing	
  measurement	
  tools.
29	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

pS-performance ToolKit• easy to start perfSONAR• CentOS is required.
• perfSONAR• regular monitoring tools• WEB visualizing tools• http://psps.internet2.edu/toolkit	  2010年8月16日月曜日

perfSONAR	
  AdopGon• perfSONAR	
  is	
  gaining	
  tracGon	
  as	
  an	
  interoperable	
  and	
  extensible	
  monitoring	
  soluGon• AdopGon	
  has	
  progressed	
  in	
  the	
  following	
  areas:– R&E	  networks	
  including	
  backbone,	
  regional,	
  and	
  exchange	
  points– UniversiGes	
  on	
  a	
  naGonal	
  and	
  internaGonal	
  basis– Federal	
  labs	  and	
  agencies	
  in	
  the	
  United	
  States	  (e.g.	
  JET	  nets)– ScienGﬁc	
  Virtual	
  OrganizaGons,	
  notably	
  the	
  LHC	
  project• Recent	
  interest	
  has	
  also	  accrued	
  from:– InternaGonal	
  R&E	  network	
  partners	
  and	
  exchange	
  points– Commercial	
  Providers	
  in	
  the	
  United	
  States– Hardware	
  manufactures

30	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

perfSONAR	
  AdopGon• Networks– US	
  NaGonal	
  R&E	  Networks• ESnet,	
  Internet2,	
  NLR,	
  NOAA– InternaGonal	
  R&E	  Networks/Exchange	  Points• APAN	
  NOC,	
  CSTNET,	
  GEANT	
  (and	  NREN	  Partners),	
  Gloriad,	
  JGN2,	
  JPNet,	
  KRNET,	
  MANLAN,	
  Starlight,	
  TransPac2,	
  RNP

– US	
  Regional	
  R&E	  Networks• CENIC,	
  CIC	
  OmniPoP,	
  Florida	
  LambdaRail,	
  Front	  Range	
  GigaPoP,	
  GPN,	
  Indiana	  GigaPoP,	
  LEARN,	
  LONI,	
  MAX,	
  MCNC,	
  Merit,	  MOREnet,	
  Northern	
  Lights	
  GigaPoP,	
  NOX,	
  NYSERNet,	  OARnet,	
  Paciﬁc	
  Northwest	
  GigaPoP,	
  SoX,	
  UEN

• US	
  Based	
  Federal	
  Labs/FaciliGes– ANL,	
  BNL,	
  DOE	
  Headquarters,	
  FNAL,	
  LBNL,	
  LLNL,	
  MIT	  Lab	
  for	
  Nuclear	
  Science,	
  NaGonal	
  Library	
  of	
  Medicine,	
  NCAR,	
  NCSA,	
  NERSC,	
  PNNL,	
  PPPL,	
  PSC,	
  SLAC31	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

perfSONAR	
  AdopGon• InternaGonal	
  Sites– North	  America• Simon	  Frazier	
  University	
  (Canada),	
  University	
  of	
  Western	  Ontario	
  (London,	
  Ontario,	  Canada),	
  Camosun	
  College	
  (Canada)– South	  America• MonIPE	
  -­‐	
  RNP	
  (Rio	  de	
  Janeiro,	
  Brazil),	
  Universidade	
  Federal	
  De	
  Santa	
  Catarina	  (Brazil),	
  SPRACE	
  (Brazil),	
  UERJ	  (Brazil),	
  Innova-­‐Red	
  (Buenos	
  Aires,	
  ArgenGna),	
  UFSC	  (Florianopolis,	
  Brazil),	
  REUNA	
  (SanGago,	
  Chile),	
  PUCP	
  (Lima,	  Peru),	
  MRREE	
  (Lima,	  Peru),	
  RAGIE2	
  (Universidad	
  Mariano	
  Galvez	
  –	
  Guatemala),	
  UDESC	
  (Brazil),	
  UNIFACS	
  (Salvador,	
  Bahia,	
  Brazil)– Europe/Middle	  East• Universitat	  Politecnica	
  de	
  Catalunya	  (Barcelona	
  Spain),	
  GEANT	
  Aﬃliated	
  NRENs	  (Europe),	
  Northwestern	  (Doha,	  Qatar),	
  Georgetown	  University	
  (Doha,	  Qatar)– Africa• African	
  Network	
  InformaGon	  Center,	
  KRNET,	
  UbuntuNet– Asia/Oceana• Chinese	
  University	
  of	
  Hong	  Kong,	
  Chonnam	
  NaGonal	
  University	
  (South	
  Korea),	
  KISTI	
  (Korea),	
  Teritary	  EducaGon	
  Commission	
  (Wellington,	
  NZ),	
  IHEP	
  (China),	
  Advanced	
  Network	
  Lab.	
  at	
  JNU	
  (South	
  Korea),	
  Monash	
  University	
  (Melbourne,	
  Australia),	
  USTC	
  ATLAS	
  Tier-­‐3	
  in	
  Hefei	  (AnHui	
  Province,	
  China),	
  NCHC	  (Taiwan),	
  NICT	
  (Japan),	  Thaisarn	  Nectec	  (Bangkok,	
  Thailand)32	
  –	
  07/27/10,	
  ©	
  2010	  Internet2

2010年8月16日月曜日

perfSONAR	
  AdopGon• Universities
	
Boise State University
	
Boston University *	
California Institute of Technology **	
The College of William and Mary	Colorado School of Mines	
Colorado State University
	
Florida Atlantic University	George Mason University	Georgia Tech University	Georgetown University
	
Hat Creek Radio Observatory	
Harvard University *	Hope College
	Indiana University *	Indiana Purdue University *	Iowa State University
	Johns Hopkins University
	
Leeward Community College	Louisianna State University
	
Massachusetts Institute of Technology (MIT) **	
Merced County Office of Education	Michigan State University *	Middle Tennessee State University
	North Dakota State University
	Northwestern University	
Oregon State University	Penn State University	Portland Community College
	
Purdue University **	Scripps College	Southern Methodist University *	Syracuse University	Texas A&M University
	
Tufts *• Universities
	
University of California Irvine *	
University of California Los Angles
	University of California San Diego **	University of California Santa Barbara *	University of California Santa Cruz *	University of Chicago *	University of Connecticut
	
University of Delaware
	University of Florida **	University of Hawaii
	University of Houston	
University of Illionios *	University of Maryland
	
University of Michigan *	University of Minnesota
	
University of Nebraska **	University of Northern Iowa
	
University of Oklahoma *	University of Oregon	
University of Pennsylvania
	
University of South Dakota	University of South Florida
	University of Texas *	University of Utah	
University of Washington *	University of Wisconsin * **	Vanderbilt University **	Washington University in St. Louis	
Wayne State University* USATLAS** USCMS33	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

perfSONAR	
  AdopGon• Commercial	
  Networks/OrganizaGons– AboveNet,	
  BCNet,	
  BBN,	
  Cobham,	
  EagleNet,	  KDL	  Inc.,	
  Northrop	
  Gruman,	  Ocala	  Electric	  Company,	
  Philadelphia	
  Orchestra,	
  Steelville	  Telephone	
  Exchange,	
  Viawest,	
  Verizon• Virtual	
  OrganizaGons– GENI,	
  LIGO,	
  LSST,	
  MeasurementLab,	
  REDDnet,	
  USATLAS,	
  USCMS• Live	
  pS	
  Status:– Services:	
  hyp://www.perfsonar.net/acGveServices– LocaGons:	
  hyp://www.perfsonar.net/acGveServices/IS

34	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

What	  is	
  currently	
  available
Services	
  useful	
  for	
  solving	
  end-­‐to-­‐end	
  performance	
  problems• Link	  UGlizaGon	
  (and	  errors,	
  and	  discards)– SNMP	  or	
  TL1– RRD/SQL	  Based• Interface	  Status– Useful	
  for	
  mul'-­‐domain	  monitoring• Ping/Traceroute	
  beacons– Sources	
  to	
  test	  against	
  for	  a	
  given	
  network• ”looking	
  glass”es– Run	
  commands	
  on	
  network	
  devises.	
  	
  Authen'ca'on	
  will	  vary	  naturally• AcGve	
  measurements	
  and	
  acGve	
  measurement	
  results– Latency	
  (and	  loss,	
  duplicates):	
  one-­‐way	
  and	
  round-­‐trip– Throughput	  (Iperf	
  for	  now)– NDT	
  and	
  NPAD	  server	
  loca'ons– ‘Universal’	
  MP
35	
  –	
  07/27/10,	
  ©	
  2010	  Internet2• Run	  many	  commands	
  like	  ping,	  traceroute2010年8月16日月曜日

How	
  might	
  perfSONAR	
  be	
  useful	
  to	
  you?
• A	
  data	  source	
  for	
  your	  research• AcGve	
  deployments	
  (as	  in	
  the	
  ones	  just	  described)	
  are	  all	
  over.	
  	
  • Easy	  to	
  locate	
  using	  the	
  ‘Lookup	
  Service’• Data	  policy	
  varies,	
  but	
  most	  are	
  open• A	  place	  to	
  try	
  out	  new	
  algorithms	
  and	
  visualizaGons,	
  and	
  feed	  them	
  
back	
  to	
  producGon	
  networks• Open	  Source	
  Development	
  model,	  your	  help	
  is	
  more	  than	
  welcome!

• A	  place	  to	  install	
  new	
  data	  sources	
  with	  the	
  cooperaGon	
  of	
  producGon	
  networks• Integrate	  new	
  tools	
  into	  the	
  framework• Construct	  new	
  data	  representaGons

36	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

For	
  more	  informaGon• General	
  and	
  MDM	
  impelmentaGon:	
  www.perfsonar.net• The	  PS	  implementaGon:	
  hyp://psps.perfsonar.net	
  • perfSONAR-­‐PS	
  tools	
  and	
  soAware:	
  hyp://soAware.internet2.edu	
  • A	  hook	  to	
  the	
  global	
  lookup	
  service:	  	  	
  hyp://stats1.es.net/cgi-­‐bin/tree.cgi	  hyp://www.perfsonar.net/acGveServices/IS/	
  • More	  human-­‐readable	
  list	  of	
  services:	
  hyp://stats1.es.net/cgi-­‐bin/directory.cgi	  hyp://www.perfsonar.net/acGveServices/	
  
37	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Mailing	
  Lists• Development	
  (by	
  approval	
  of	
  the	
  project)– hyps://lists.internet2.edu/sympa/subscribe/perfsonar-­‐dev• User	  Support– hyps://lists.internet2.edu/sympa/subscribe/perfsonar-­‐user– hyps://lists.internet2.edu/sympa/subscribe/perfsonar-­‐ps-­‐users– hyps://lists.internet2.edu/sympa/subscribe/performance-­‐node-­‐users• Announcements– hyps://lists.internet2.edu/sympa/subscribe/perfsonar-­‐announce– hyps://lists.internet2.edu/sympa/subscribe/perfsonar-­‐ps-­‐announce– hyps://lists.internet2.edu/sympa/subscribe/performance-­‐node-­‐announce• Working	
  Groups– hyps://lists.internet2.edu/sympa/subscribe/performance-­‐wg– hyps://lists.internet2.edu/sympa/subscribe/is-­‐wg– hyp://www.ogf.org/mailman/lisGnfo/nm-­‐wg– hyp://www.ogf.org/mailman/lisGnfo/nmc-­‐wg– hyp://www.ogf.org/mailman/lisGnfo/nml-­‐wg38	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

perfSONAR	
  PreliminariesJuly	
  7th	  2010,	  perfSONAR	  Workshop	
  –	
  perfSONAR	  Tutorial
Jason	
  Zurawski	
  -­‐	  Network	
  SoAware	
  Engineer,	
  Research	
  LiasonFor	
  more	  informaGon,	
  visit	
  hyp://www.perfsonar.net

40	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

July	
  7th	  2010,	  perfSONAR	  Workshop	
  –	
  perfSONAR	  Tutorial
Jason	
  Zurawski,	  Network	
  SoAware	
  Engineer,	
  Research	
  Liason

perfSONAR	  Architecture

2010年8月16日月曜日

Outline• MoGvaGon	
  and	
  Goals• Service	
  Oriented	
  Architecture	
  (SOA)• Architecture	
  Overview– Service	
  Components– Client	
  Examples• Framework	
  InteracGons– Deploying	  a	
  Service– Simple	
  Client/Service	
  InteracGon• Echo• Metadata	  Request• Data	  Request– Lookup	
  Service	
  InteracGon2	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

MoGvaGon	
  and	
  Goals• Thus	
  far	  we	
  have	  seen	  that	  the	  perfSONAR	
  framework	
  can	  be	
  used	  to	
  solve	
  end-­‐to-­‐end	
  performance	
  problems	
  for	
  mulG-­‐domain	  paths.
• The	
  framework	
  is	
  made	  up	
  of	
  several	
  unique	
  components	
  and	
  design	  consideraGons,	
  all	  of	
  which	  operate	
  in	
  a	
  cooperaGve	
  yet	  independent	
  manner– Each	  funcGonality	
  is	
  separated	
  into	  a	
  speciﬁc	
  funcGon– Clients	
  and	  servers	
  interact	
  through	  scripted,	
  XML	
  Based	
  protocols– Measurement	
  data	  is	
  encoded	
  in	
  expressive	  XML	
  formats3	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Service	  Oriented	
  Architecture	
  (SOA)• Interoperable	
  network	
  measurement	
  middleware	
  (SOA):– Modular– Web	  services-­‐based– Decentralized– Locally	
  controlled• Integrates:• Network	
  measurement	
  tools	
  and	
  data	  archives• Data	  manipula'on• Informa'on	
  Services• Discovery• Topology• Authen'ca'on	
  and	
  authoriza'on• Based	
  on:• Open	  Grid	  Forum	
  Network	
  Measurement	
  (NM)	
  Working	
  Group	
  schema• Currently	
  aSemp'ng	
  to	
  formalize	
  speciﬁca'on	
  of	
  perfSONAR	
  protocols	
  in	
  the	
  Network	
  Measurement	  Control	
  (NMC)	
  working	
  group.
• Network	
  topology	
  descrip'on	
  being	
  deﬁned	
  in	
  the	  Network	
  Markup	
  Language	  (NML)	  Working	
  Group4	
  –	
  07/27/10,	
  ©	
  2010	  Internet2

2010年8月16日月曜日

Service	  Oriented	
  Architecture	
  (SOA)• Measurement	
  Point	
  (MP)	
  Service– Enables	
  the	
  iniGaGon	
  of	
  performance	
  tests• Measurement	
  Archive	
  (MA)	
  Service– Stores	
  and	
  publishes	
  performance	
  monitoring	
  results• TransformaGon	
  Service– Transform	
  the	
  data	  (aggregaGon,	
  concatenaGon,	
  correlaGon,	
  translaGon,	  etc)• Resource	
  protector– Arbitrate	  the	  consumpGon	
  of	
  limited	
  resources– Other	  services	
  delegate	
  a	
  limited	
  porGon	
  of	
  the	
  authorizaGon	
  decision	  here

These	  services	
  are	
  speciﬁcally	
  concerned	
  with	  the	
  job	  of	
  network	
  performance	  measurement	
  and	
  analysis5	  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Service	  Oriented	
  Architecture	
  (SOA)• Lookup	
  Service– Allows	
  the	
  client	
  to	
  discover	
  the	
  exisGng	
  services	
  and	
  other	
  LS	
  services.
– Dynamic:	
  services	
  registraGon	
  themselves	
  to	
  the	  LS	  and	
  menGon	
  their	
  capabiliGes,	
  they	  can	
  also	  leave	
  or	
  be	
  removed	
  if	
  a	
  service	
  goes	  down.

• Topology	
  Service– Make	  the	  network	
  topology	
  informaGon	
  available	
  to	
  the	
  framework.
– Find	  the	
  closest	  MP,	
  provide	  topology	
  informaGon	
  for	
  visualisaGon	
  tools• AuthenGcaGon	
  Service– AuthenGcaGon	
  &	
  AuthorizaGon	
  funcGonality	
  for	  the	
  framework– Users	
  can	  have	  several	
  roles,	
  the	
  authorizaGon	
  is	
  done	  based	  on	
  the	
  user	  role.
– Trust	  relaGonship	
  between	
  networks

These	  services	
  are	  the	
  infrastructure	
  concerned	
  with	  discovering	
  federaDng	
  the	  available	
  network	
  services6	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

6Service	  Oriented	
  Architecture	
  (SOA)

Where can I get more about
gLS network Doman
B/IPd,e,f andUsefulgraph
Domain A/IP a,b,c?
LS A, LS B Client
Where is link utilization for – IPs a,b,c?
a,b,c : Network A, MA A
linkutilization
utilization d,e,fWhere Get
is link
Here you go
Get Link utilization a,b,c
for :- Network
IPs d,e,f?B,d,e,fHere you go
LS A

MA A

a

b
c

MA B
LS B
MA B

d

Network A

Network B7	
  –	
  07/27/10,	
  ©	
  2010	  Internet2

2010年8月16日月曜日

efArchitecture	
  Overview8	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

perfSONAR	  Architecture	
  Overview

9	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Architecture	
  Overview	  -­‐	  MP
• Measurement	
  Point	
  (MP)	
  form	  the	
  lowest	
  layer	
  of	
  the	  monitoring	
  infrastructure– Directly	
  interacts	
  with	  the	
  measurement	
  tool– Can	
  oﬀer	
  WS	
  control	
  over	  on-­‐demand	
  measurement– Can	
  oﬀer	  interface	
  to	
  a	
  regular	
  scheduled	
  measurements• 	
  Rolls	
  of	
  the	
  Measurement	
  Point:– U'lize	
  well	  known	
  tools	
  to	
  perform	
  measurements– Oﬀer,	  at	
  a	
  minimum,	
  cache	  storage	
  of	
  recently	
  performed	
  measurements– Interact	
  with	  Measurement	
  Archives	  (MAs)	
  to	
  archive	  stored	  measurements• Examples:– perfSONAR-­‐BUOY	
  (OWAMP	  and	
  BWCTL	
  Tes'ng)– PingER	
  (Ping	  Tes'ng)– Command	  Line	
  MP
10	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Architecture	
  Overview	  -­‐	  MA
• Measurement	
  Archive	
  (MA)	
  stores	
  the	  results	
  of	
  network	
  and	
  performance	  measurements– WS	
  interface	
  for	
  storage	  and	
  query– Interacts	
  with	  backend	
  databases	
  (e.g.	
  SQL,	
  RRD)• 	  Rolls	
  of	
  the	
  Measurement	
  Archive:– Expose	
  historical	
  and	
  current	
  measurements	
  of	
  diverse	
  types– Draw	
  data	  queries	
  away	  from	  the	
  Measurement	
  Points	
  (MPs)• Examples:– perfSONAR-­‐BUOY	
  (OWAMP	  and	
  BWCTL	
  Data)– PingER	
  (Ping	
  Data)– SNMP	
  MA/RRD	
  MA– Status	
  MA

11	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Architecture	
  Overview	  -­‐	  TrS
• The	  TransformaGon	
  Service	
  (TrS)	
  performs	
  operaGons	
  on	
  data	  sets	  (e.g.	
  aggregaGon,	
  correlaGon).	
  	
  
– WS	
  interface– PotenGal	
  to	
  store	
  well	  known	
  operaGons,	
  and	
  replay	
  later• 	
  Rolls	
  of	
  the	  TransformaGon	
  Service	  :– Draw	  complex	  staGsGcal	
  queries	
  from	  Measurement	
  Archives– Provide	  a	
  conduit	
  for	
  popular	
  operaGons	
  (e.g.	
  running	
  staGsGcs	
  over	  several	
  changing	
  dataset).	
  	
  • Examples	
  (Planned):– Path	  diagnosGcs	
  tools• Combining	
  mulGple	
  metrics	  (network	
  path,	
  uGlizaGon,	
  latency,	
  bandwidth)– Data	  presentaGon• StaGsGcal	
  results	  for	
  raw	
  measurements.	
  	
  
12	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Architecture	
  Overview	  -­‐	  RP
• The	
  Resource	  Protector	
  (RP)	
  monitors	
  the	
  relaGve	
  performance	
  and	  availability	
  of	
  the	  monitoring	
  infrastructure	  – Knowledge	
  of	
  the	  services	
  in	
  a	
  given	
  deployment– Deﬁned	
  policy	
  regarding	  access	
  and	  resources• Rolls	
  of	
  the	  TransformaGon	
  Service	  :– Protects	
  the	
  Gme	
  and	  resources	
  of	
  services	
  from	  being	
  overrun• Too	  many	  queries	
  from	  a	
  single	
  source• Too	  much	  data	  for	  a	
  given	
  query– Cooperate	
  with	  the	
  AuthenGcaGon	
  and	
  AuthorizaGon	
  (AA)	
  enGGes• Examples	
  (Planned):– Data	  ProtecGon• Limits	
  the	
  size,	
  duraGon,	
  or	
  frequency	
  of	
  a	
  query– Service	
  ProtecGon• Limits	
  access	
  to	
  funcGonality	
  of	
  the	  service13	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Architecture	
  Overview	  -­‐	
  LS
• The	
  Lookup	
  Service	
  (LS)	
  is	
  a	
  general	
  name	  for	  the	  service	
  and	
  and	  data	
  discovery	
  infrastructure– Facilitates	
  service	
  and	
  data	  discovery	
  through	
  the	  concept	
  of	
  registraGon– “Summarizes”	
  and	  distributes	
  the	
  job	  of	
  locaGon	
  across	
  layers	
  of	
  lookup• Home	  Lookup	
  Services	
  –	
  Local	
  cache	  of	
  data	  for	  several	
  services• Global	
  Lookup	
  Services	  –	
  Works	
  similar	
  to	
  DNS	
  for	
  locaGng	
  informaGon	
  through	
  general	
  queries• Rolls	
  of	
  the	
  Lookup	
  Service	  :– Draws	
  speciﬁc	
  queries	
  about	  the	
  data	  and	  services	
  away	  from	  the	
  Measurement	
  Points	  and	
  Archives– Distribute	
  informaGon	
  globally	
  based	  on	
  local	  condiGons– Assure	
  the	
  ‘freshness’	
  of	
  informaGon	
  in	
  a	
  dynamic	
  infrastructure14	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Architecture	
  Overview	  -­‐	  hLS
• The	
  Home	  Lookup	
  Service	
  (hLS)	
  interacts	
  directly	
  with	  the	
  other	  porGons	
  of	
  the	  perfSONAR	
  framework– Recommended	
  deployment	
  is	
  per	
  domain• Accepts	
  RegistraDon	
  informaGon	
  from	  around	  the	
  framework– E.g.	
  An	
  MA	
  will	  register	
  its	
  name,	
  locaGon,	
  and	  available	
  metadata– Metadata	
  =	
  staGc	
  porGon	
  of	
  a	
  measurment	
  (‘subject’,	
  not	  results)• Responds	
  to	
  Queries	
  about	
  services	
  and	
  data– Services	
  looking	
  for	
  each	  other	
  (e.g.	
  MP	
  looking	
  for	  an	
  MA)
– Client	
  applicaGons	
  looking	
  for	
  data15	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Architecture	
  Overview	  -­‐	  gLS
• The	
  Global	
  Lookup	
  Service	
  (gLS)	
  serves	
  as	
  the	
  oracle	
  of	
  the	
  perfSONAR	
  framework– Global	
  cloud	
  of	
  services	
  cooperaGng	
  together	
  to	
  distribute	
  informaGon– Manage	
  the	
  hLSs	
  at	
  the	
  lower	  layer• Accepts	
  RegistraDon	
  informaGon	
  from	  hLSs	  only(!)– E.g.	
  An	
  hLS	
  will	  register	
  its	
  name,	
  locaGon,	
  and	  a	
  summary	
  of	
  the	
  services	
  and	
  data	  it	
  contains– Summary	
  =	
  condensed	
  list	  of	
  domains,	
  ip	
  addresses,	
  data	  types• Responds	
  to	
  Queries	
  about	
  services	
  and	
  data– Similar	  to	
  hLS	
  queries,	
  but	  more	  focused	
  on	
  where	  instead	
  of	
  what– Answer	
  is	
  typically	
  an	
  hLS	  to	
  contact,	
  not	  a	
  direct	  result16	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Architecture	
  Overview	
  –	
  Lookup	
  Service17	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Architecture	
  Overview	  -­‐	  CommuniGes• CommuniGes	
  =	
  Web	
  2.0	
  Content	
  Tagging– Think	
  Flicker	
  (tag	  your	  pictures	
  with	  a	
  category)– Think	
  iTunes	
  (tag	  your	  music	
  with	  a	
  genre)• How	
  does	  this	  help	  measurement	
  lookup	
  and	  discovery?
– One	  more	  axis	  to	
  search	
  on– More	  human	
  readable	
  and	
  understandable	
  than	  IP	
  address	  or	  hostnames• Use	  as	
  many	  (or	  as	
  few)	
  as	
  required:– Networks	
  (e.g.	
  Campus,	  Regional,	
  Network)
– VO	
  or	
  Project	  (e.g.	
  USATLAS,	
  eVLBI,	
  etc.)– OrganizaGon	
  (DOE)– Other?

18	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Architecture	
  Overview	  -­‐	  CommuniGes• Example:	
  Some	  VO	
  is	
  sepng	
  up	
  monitoring.
– All	
  sites	
  want	  to	
  test	  with	  each	  other– Not	
  everyone	
  is	
  coming	
  online	
  at	
  once,	  and	  VO	  membership	
  may	  be	
  volaGle.	
  	  – Strategy	
  1:• Central	
  VO	
  coordinator	
  maintains	
  a	
  list	  of	
  parGcipants	
  (and	  must	  update	  it	
  oAen)• All	  monitoring	
  is	
  manual:	
  add/remove	
  test	  hosts	
  when	
  the	
  list	  changes– Strategy	
  2:	
  
• VO	
  recommends	
  a	
  tag	
  for	  all	  new	
  hosts• All	  VO	  members	
  search	
  for	
  test	  hosts	  (periodically)	
  that	  share	  this	
  
tag	
  –	
  N.B.	
  the	  GUIs	
  on	
  the	  disk	  can	  organize	
  this	
  automaGcally

19	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Architecture	
  Overview	  -­‐	  CommuniGes• Screenshot	
  from	  the	
  toolkit	
  (when	
  sepng	
  up	
  the	
  host):• Top:	
  CommuniGes	
  the	
  host	  has	
  chosen	  to	
  associate	
  with• Boyom:	
  ‘Popular’	
  communiGes– The	
  word	  cloud	
  is	
  based	  on	
  what	  we	
  found	  in	
  the	  GLS	  –	  the	
  larger	
  the	
  word	  =	
  the	  more	  people	  that	  are	
  using	
  this	  classiﬁca'on

20	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Architecture	
  Overview	  -­‐	  CommuniGes• List	  of	
  hosts	
  from	  the	
  LHC	
  community:

21	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Architecture	
  Overview	  -­‐	
  TS
• The	
  Topology	
  Service	
  (TS)	
  gathers	
  and	
  stores	  network	
  topology	
  informaGon	
  similar	
  to	
  the	
  Lookup	
  Service	
  (LS)– Interfaces	
  with	  external	
  network	
  tools	
  (Dynamic	
  Circuits,	
  NOC	
  databases)– Provides	
  a	
  query	
  interface• Rolls	
  of	
  the	  TransformaGon	
  Service	  :– Gather	
  network	
  topology	
  from	  various	
  sources– Correlate	
  informaGon	
  found	  in	
  other	  TS	
  sources	
  to	
  provide	  a	
  complete	
  view	
  of	
  network	
  availability– Interface	
  with	  measurement	
  tools	
  to	
  associate	
  measurements	
  with	  speciﬁc	
  porGons	
  of	
  the	
  infrastructure

22	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Architecture	
  Overview	  -­‐	  AS
• The	
  AuthenGcaGon	
  and	
  AuthorizaGon	
  (AS)	
  service	
  serves	
  as	
  a	
  front	
  end	
  for	
  idenGty	
  management.
– IdenGty	
  management	
  relies	
  on	
  assigning	
  roles	
  to	
  a	
  given	
  user	  via	
  a>ributes,	
  e.g.	
  permission	
  to	
  do	
  something– The	  AS	
  will	  communicate	
  via	  WS	
  with	  a	
  client	
  and	
  pass	  along	  credenDals	
  in	
  order	
  to	
  validate	  an	
  acGon	  or	
  task– The	  AS	
  will	  protect	
  access	
  to	
  services	
  and	
  data• Rolls	
  of	
  the	
  AuthenGcaGon	
  Service	
  :– Validate	
  services	
  and	
  clients	
  given	
  credenGals– Act	  on	
  behalf	
  of	
  the	
  users	
  to	
  acquire	
  the	
  necessary	  permissions

23	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Framework	
  InteracGon• The	
  following	
  examples	
  will	  illustrate	  the	
  mechanics	
  of	
  services	
  and	
  protocols:– Deploying	  a	
  Service– Simple	
  Client/Service	
  InteracGon• Echo• Metadata	  Request• Data	  Request– Lookup	
  Service	
  InteracGon

24	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Deploying	  a	
  Service• A	
  perfSONAR	
  service	
  is	
  deployed	
  alongside	
  the	
  measurement	
  infrastructure• InteracGons	
  with	  the	
  lookup	
  service	
  and	
  clients	
  are	  described

25	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Deploying	  a	
  Service26	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Deploying	  a	
  Service27	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Deploying	  a	
  Service28	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Deploying	  a	
  Service29	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Deploying	  a	
  Service30	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Deploying	  a	
  Service31	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Deploying	  a	
  Service32	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Deploying	  a	
  Service33	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Deploying	  a	
  Service34	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Deploying	  a	
  Service35	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Deploying	  a	
  Service36	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Deploying	  a	
  Service37	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Deploying	  a	
  Service38	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Client/Service	
  InteracGon• EchoRequest– Sent	  to	
  a	
  service	  to	
  test	  connecGvity– Can	  be	
  made	  arbitrarily	
  complex	
  by	
  the	  service	
  designer• Test	  backend	
  storage• Test	  internal	
  self-­‐checks– Minimum	
  is	
  an	
  ‘are	  you	
  alive’	
  ping39	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Client/Service	
  InteracGon

40	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Client/Service	
  InteracGon

41	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Client/Service	
  InteracGon• MetadataKeyRequest– For	  a	
  given	
  (parGal)	
  metadata,	
  ask	  the	  service	  to	  verify	
  that	  it	
  does	
  or	
  
does	
  not	
  exist– Return	  a	
  ‘key’,	
  e.g.	
  replayable	
  token,	  to	
  access	
  the	
  data42	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Client/Service	
  InteracGon

43	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Client/Service	
  InteracGon

44	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Client/Service	
  InteracGon

45	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Client/Service	
  InteracGon

46	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Client/Service	
  InteracGon• SetupDataRequest– Given	
  a	
  key	  or	
  (parGal)	
  metadata,	
  return	
  measurement	
  informaGon.
– Can	  be	
  ‘ﬁltered’	
  by	
  Gme	  to	
  prevent	
  gepng	
  more	  results	
  than	
  necessary.

47	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Client/Service	
  InteracGon

48	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Client/Service	
  InteracGon

49	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Client/Service	
  InteracGon

50	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Client/Service	
  InteracGon

51	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Client/Service	
  InteracGon

52	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Client/Service	
  InteracGon

53	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Lookup	
  Service	
  InteracGon• Services	
  register	
  with	  an	
  hLS• hLSs	  summarize	
  what	  they	  know	
  and	
  pass	  to	
  the	
  gLSs• gLSs	  exchange	
  the	
  informaGon	
  as	
  needed• Clients	
  will	  need	  a	
  mulG-­‐step	
  process	
  to	
  ﬁnd	
  informaGon– Query	
  the	  gLS
– Query	
  the	
  appropriate	
  hLS– Query	
  the	
  appropriate	
  services

54	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Lookup	
  Service	
  InteracGon

55	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Lookup	
  Service	
  InteracGon

56	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Lookup	
  Service	
  InteracGon

57	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Lookup	
  Service	
  InteracGon

58	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Lookup	
  Service	
  InteracGon

59	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Lookup	
  Service	
  InteracGon

60	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Lookup	
  Service	
  InteracGon

61	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Lookup	
  Service	
  InteracGon

62	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Lookup	
  Service	
  InteracGon

63	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Lookup	
  Service	
  InteracGon

64	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Lookup	
  Service	
  InteracGon

65	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Conclusions• We	
  have	  seen	
  the	
  Architecture• We	
  have	  seen	
  how	  it	
  works	
  together• Whats	
  next?
– SoAware	
  Availability– Protocols– Client	  Design– Service	
  Design66	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

perfSONAR	  ArchitectureJuly	
  7th	  2010,	  perfSONAR	  Workshop	
  –	
  perfSONAR	  Tutorial
Jason	
  Zurawski	
  -­‐	  Network	
  SoAware	
  Engineer,	
  Research	
  LiasonFor	
  more	  informaGon,	
  visit	
  hyp://www.perfsonar.net

67	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

July	
  7th	  2010,	  perfSONAR	  Workshop	
  –	
  perfSONAR	  Tutorial
Jason	
  Zurawski,	  Network	
  SoAware	
  Engineer,	
  Research	
  Liason

perfSONAR	
  Use	
  Cases

2010年8月16日月曜日

Outline• MoGvaGon• How	  it	
  Should	  Work• How	  it	  Probably	
  Works• Use	
  Cases– Cisco	  Telepresence• QuesGons	
  for	
  Later	
  in	
  the	  Workshop…
• Reference	  Material	
  (Not	  Reviewed)– Why	  is	
  Science	  Data	  Diﬀerent?
– Iden'fying	  Common	
  Network	
  Problems– Addi'onal	
  Use	
  Cases• Georgetown	
  InternaGonal	
  Campus• USATLAS• REDDnet2	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

MoGvaGon• Now	
  that	  we	
  have	  seen	
  the	
  purpose	
  and	
  makeup	
  of	
  the	  perfSONAR	
  infrastructure,	
  it’s	  Gme	  to	
  see	
  what	  it	
  can	  do	
  in	
  the	
  real	  world• perfSONAR	
  is	
  used	  by	
  network	
  engineers	
  to	
  idenGfy	
  many	  types	  of	
  performance	
  problem– A	  Divide	  and	
  Conquer	
  strategy	
  is	
  necessary	
  to	
  isolate	
  problems– A	
  structured	
  methodology	
  helps	
  to	
  eliminate	
  duplicate	  or	
  useless	
  steps– perfSONAR	
  works	
  best	  when	
  everyone	  parGcipates,	
  holes	  in	
  deployment	
  lead	  to	
  holes	
  in	
  the	
  problem	
  solving	
  phase• The	
  following	
  secGons	
  will	  outline	
  the	
  proper	
  deployment	
  strategy	  and	
  describe	
  some	  real	  work	  use	
  cases3	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

How	
  it	
  Should	  Work• To	
  accurately	
  and	
  swiAly	
  address	
  network	
  performance	
  problems	
  the	
  following	
  steps	
  should	
  be	
  undertaken– IdenGfy	
  the	
  problem:	
  if	
  there	  a	
  user	  in	
  one	
  locaGon	
  is	
  complaining	
  about	  performance	
  to	
  another,	
  get	  as	
  much	  informaGon	
  as	
  possible• Is	
  the	
  problem	
  uni-­‐direcGonal?	
  Bi-­‐direcGonal?
• Does	  the	
  problem	
  occur	
  all	  the	
  Gme,	
  frequently,	
  or	
  rarely?
• Does	  the	
  problem	
  occur	
  for	
  only	  a	
  speciﬁc	
  applicaGon,	
  many	  applicaGons,	
  or	
  
only	
  some	  applicaGons?
• Is	
  the	
  problem	
  reproducible	
  on	
  other	  machines?

– Gather	
  informaGon	
  about	  the	
  environment• Hosts• Network	
  Path• ConﬁguraGon	
  (where	
  applicable)• Resources	
  available4	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

How	
  it	
  Should	  Work• Cont.
– Methodically	
  approach	
  the	
  problem• Test	  using	  the	
  same	  tool	  everywhere,	
  gather	  results• Before	
  moving	
  on	
  to	
  the	
  next	  tool,	
  did	
  you	
  gather	
  everything	
  of	
  value?
• Are	  the	  results	
  consistent?

– AAer	  proceeding	
  through	  all	
  tools	
  and	
  approaches,	
  form	  theories• Can	  the	
  problem	
  be	
  isolated	
  to	
  a	
  speciﬁc	
  resource	
  or	
  component?
• Can	
  tesGng	
  be	
  performed	
  to	
  eliminate	
  dead	  ends?

• Consider	
  the	
  following	
  example:– InternaGonal	
  path– Problems	
  noted– We	
  know	
  the	
  path– We	
  have	  tools	
  available5	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Scenario:	
  MulG-­‐domain	  InternaGonal	
  Path6	  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Desirable	
  Case:	
  Expected	
  Performance7	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Typical:	
  Poor	  Performance	
  …	
  Somewhere8	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Typical:	
  Poor	  Performance	
  …	
  Somewhere
But where?

9	  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

SoluGon:	
  Test	  Points	
  +	
  Regular	
  Monitoring

10	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

perfSONAR:	
  Backbone	
  and	
  Exchanges

11	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

perfSONAR:	
  Regional	
  Networks

12	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

perfSONAR:	
  Campus13	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Path	  DecomposiGon	
  –	
  Isolate	
  the	
  Problem
Step by step: test
between points

14	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Path	  DecomposiGon	
  –	
  Isolate	
  the	
  Problem
1st Segment - noproblems found

15	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Path	  DecomposiGon	
  –	
  Isolate	
  the	
  Problem
2nd Segment – Problem
Identified …

16	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Path	  DecomposiGon	
  –	
  Isolate	
  the	
  Problem
2nd Segment – Problem
Identified … and fixed!

17	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Path	  DecomposiGon	
  –	
  Isolate	
  the	
  Problem
But end to end
performance still
poor

18	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Path	  DecomposiGon	
  –	
  Isolate	
  the	
  Problem
3rd Segment – Noproblems19	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Path	  DecomposiGon	
  –	
  Isolate	
  the	
  Problem
4th Segment – Noproblems20	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Path	  DecomposiGon	
  –	
  Isolate	
  the	
  Problem
5th Segment – Lastproblem found …

21	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Path	  DecomposiGon	
  –	
  Isolate	
  the	
  Problem
5th Segment – Lastproblem found …
and fixed!

22	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Lessons	
  Learned• Problem	
  resoluGon	
  requires	
  proper	
  tools– Specialized	
  to	
  given	
  task	  (e.g.	
  Bandwidth,	
  Latency)– Widely	
  available	
  where	
  the	
  problems	
  will	  be• IsolaGng	
  a	
  problem	
  is	
  a	
  well	  deﬁned,	
  mulG-­‐step	
  process– Rigid	
  set	  of	
  steps	
  –	
  systemaGc	
  approach	
  to	
  prevent	
  causing	  new	
  problems• DiagnosGcs,	
  as	
  well	  as	
  regular	
  monitoring,	
  can	  reveal	
  true	
  network	
  performance

23	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

How	
  it	
  Probably	
  Works• If	
  the	
  suggested	
  steps	  aren’t	
  taken	
  (or	
  followed	
  in	
  an	
  ad-­‐hoc	
  manner),	
  results	
  will	  vary.
– Skipping	
  steps	
  leads	
  to	
  missing	
  clues• Deployment	
  and	  parGcipaGon	
  may	
  vary,	  this	  leads	
  to	
  some	  gaps	
  in	
  
the	
  debugging	
  process• Consider	
  the	
  following	
  example:– InternaGonal	
  path– Problems	
  noted– We	
  know	
  the	
  path– We	
  have	  tools	
  available	
  -­‐	
  almost	  everywhere

24	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

39
2010年8月16日月曜日

Scenario:	
  MulG-­‐domain	  InternaGonal	
  Path25	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Desirable	
  Case:	
  Expected	
  Performance

26	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Typical:	
  Poor	  Performance	
  …	
  Somewhere

27	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Typical:	
  Poor	  Performance	
  …	
  Somewhere

But where?

28	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

SoluGon:	
  Test	  Points	
  +	
  Regular	
  Monitoring

29	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

SoluGon:	
  Test	  Points	
  +	
  Regular	
  Monitoring

Key Point: End to end monitoring
Requires participation from all
domains30	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Typical:	
  Poor	  Performance	
  …	
  Somewhere

Internet2 – Available on
the backbone

31	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Typical:	
  Poor	  Performance	
  …	
  Somewhere

The Campus isparticipating too

32	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Typical:	
  Poor	  Performance	
  …	
  Somewhere

The exchange point
makes statistics available

33	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Typical:	
  Poor	  Performance	
  …	
  Somewhere

A regional network may
not participate…

34	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Typical:	
  Poor	  Performance	
  …	
  Somewhere

Complete end to end
Monitoring is notpossible.

35	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Lessons	
  Learned• Missing	
  part	  of	
  the	
  path	  leaves	
  us	
  with	  a	
  huge	  disadvantage• May	  discover	
  some	  problems	
  through	
  isolaGon	
  on	
  
the	
  path	  we	
  know,	
  could	
  miss	  something– Most	  network	
  problems	
  occur	
  on	
  the	
  demarcaGon	
  between	
  networks– TesGng	
  around	  the	
  problem	
  won’t	
  work	  (we	
  sGll	  have	  to	
  transit	
  this	  network)

36	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Use	
  Cases• The	
  following	
  use	
  cases	
  demonstrate	
  use	  of	
  perfSONAR	
  tools	  to	
  solve	
  someGmes	
  complex	  performance	
  problems– Cisco	  Telepresence• MulG-­‐domain	  path	  where	  performance	
  guarantees	
  dictate	
  use	  of	
  a	
  speciﬁc	
  applicaGon• AddiGonal	
  Use	
  Cases	  (Not	  Presented)– Georgetown	
  InternaGonal	
  Campus• Assuring	
  quality,	
  from	  one	
  end	  of	
  the	
  world	
  to	
  another– USATLAS• Enabling	
  Big	
  Science	  through	
  diagnosGc	
  checks	
  and	
  regular	
  monitoring– REDDnet• Assuring	
  clean	
  paths	
  for	
  datamovement

37	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

Cisco	  TelePresence	
  Demo• 2	
  LocaGons• Harvard	
  University	
  (Boston,	
  MA)• Spring	
  Member	  MeeGng	
  (Arlington,	
  VA)• Must	  meet	  or	
  exceed	
  performance	
  expectaGons
• <	
  10	
  ms	
  Jiyer	
  (Packet	
  Arrival	
  VariaGon)• <	
  160	  ms	  End-­‐to-­‐End	
  Delay• <	
  0.05%	
  Packet	
  Loss• Network	
  Path	  spanned:• ~450	  Miles• 4	
  DisGnct	
  Domains• Internet2• Mid	
  Atlan'c	
  Crossroads	
  (MAX)• Northern	
  Crossroads	
  (NOX)38	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日

DemonstraGon	
  Overview
Internet2
POP

Harvard
Northern Crossroads

Mid-Atlantic Crossroads

Hotel
39	
  –	
  07/27/10,	
  ©	
  2010	  Internet2
2010年8月16日月曜日