Comparison of Two Dimension-Redu

M th d f N t k Si l ti M M

eth

o

ds 

for 

N

e

twor

k Simulation 

M

o

image generated using http:/

Kevin Mills (ANTD) & Jim Filliben (SED)

Complex Systems Study Group Complex Systems Study Group

NIST SED/ANTD Seminar 9/22/11 222/A3

millsjjfsed
Comparison of Two Dimension-Redu

M th d f N t k Si l ti M M

eth

o

ds 

for 

N

e

twor

k Simulation 

M

o

image generated using http:/

Kevin Mills (ANTD) & Jim Filliben (SED)

Complex Systems Study Group Complex Systems Study Group

NIST SED/ANTD Seminar 9/22/11 222/A3

millsjjfsed
Motivation for this Talk

1. Generate useful internet MesoNet mode

conclusions & insight conclusions

& insight

2. Show stat framework/approach &

methodology + beginning-to-”end” dem

3. Show dimension reduction de

p y endenc

y o

Design of Experiment & Sensitivity An
Outline

• CxS: Complex System IMS Project• Goal – Problem – Solution• St t F k Stat Framework• Overview of Candidate MesoNet Factors & Re• Experiment Design• Sensitivity Analysis• Dimension Reduction

via Correlation Analysis with Clustering via Correlation Analysis with Clusteringvia Principal Components Analysis

• Comparison of Dimension Reduction Techniqu• C li onc usions
IMS Project: Measurement Science for Com

Information System Information System

http://www.nist.gov/itl/antd/emergent_behavior.cfm

This project aims to develop and evaluate a coherent set of mto understand to understand behavior in behavior in complex complex information information systems such 

systems, such 

Internet, computational grids and computing clouds.

Such large distributed systems exhibit global behavior arising 

independent decisions made by many simultaneous actors, wadapt their behavior based on local measurements of system 

Actor adaptations Actor adaptations shift the global system state influencing the global system state, influencing

subsequent measurements, leading to further adaptations.

This continuous cycle of measurement and adaptation drives 

varying global behavior. For this reason, proposed changes in actor decision algorithmbe examined/understood at large spatiotemporal scale in ordebe examined/understood at large spatiotemporal scale in orde

predict ( and control) system behavior.
CxS Project

What is the problem? No one understands how to measure, predict or control macroscopic behavior icomplex information systems: (1) threatening our nation’s security and (2) costing billions of dollars.“[Despite] society’s profound dependence on netwofundamental knowledge about them is primitive. [G

communication ... networks have quite advanced 

technological implementations but their behavior ustress still cannot be predicted reliably.... There is science today that offers the fundamental knowledgscience today that offers the fundamental knowledg

necessary to design large complex networks [so] th

their behaviors can be predicted prior to building th

(above quote from Network Science 2006, a Nation(above quote from Network Science 2006, a Nation

Research Council report)
Project Goal – Problem – Solutio

• Goal – understand internet congestion and compare pro

Internet congestion control algorithms under a wide rancontrolled,p , y 

 re

peatable conditions, as simulated b

y selec

combinations of parameter values for MesoNet, a 11- to parameter network simulator.

• Problem – how to determine which MesoNet core respo

analyze when characterizing model behavior.

• Solution

– apply experiment design techniques to gene

affordable but representative data sample, and carry ousubsequent response variable evaluation via three data 

approaches:

1. sensitivity analysis 2. correlation analysis with clustering & 3 principal components analysis 3. principal components analysis
Abilene Network (3-Tier MesoNet Topol
General Problem-Solving Frame

Expert

1

2

5

Problem Solu

3

4

Data DEX = g(k,n) 1. Graphical

2. Quantitative
General Problem-Solving Frame

Expert

P bl

1

2

5

Problem

2 Solu

3

4

1 Ch t i i 1 # D

Data

1-FAT

GraphicalQuantitative

1. Characterizing

2. Sensitivity3. Optimizing

4 Modeling

1. #, D

2. List: 3. Vect

R lit 4 f

DEX = g(k,n)

Monte CarloLatin HCOrthogonal

4. Modeling

5. Comparing6. Predicting 

7 Uncertainty

4. f5. Y/N 6 # 7 SD(#

Realit

y

LabComputational

Orthogonal

RespSurface

7. Uncertainty8. Verifying9. Validating

7. SD(#8. Y/N, ...,xk)9. Y/N, 
General Problem-Solving Frame

Expert

1

2

5

Problem Solu

3

4

Data DEX = g(k,n) 1. Graphical

2. Quantitative

Q1. Response Dimension?Q2. Important Factors?Q3 Best Factor Settings?

A1. # & Set {A2. List (Ran

Q3. Best Factor Settings? A3 Vector (xQ4. Improvement over TCP?

A3. Vector (xA4. Y/N & Be
The Starting Point: Generic Mode

System Behavior Y = f(X1, X2, ..., Xk

1. Y = f(X1, X2, ..., Xk) Com2. Y = f(X1, X2, ..., Xk) Scre3. Y = f(X1, X2, ..., Xk) Regr4. Y = f(X1, X2, ..., Xk) Optim5. Y = f(X1, X2, ..., Xk) = c Cons6 Y = f(X1 X2 Xk) Dime6. 

Y

= f(X1, X2, ..., Xk

) Dime
The Starting Point: Generic Mode

System Behavior Y = f(X1, X2, ..., Xk

1. Y = f(X1, X2, ..., Xk) Com2. Y = f(X1, X2, ..., Xk) Scre



3. Y = f(X1, X2, ..., Xk) Regr4. Y = f(X1, X2, ..., Xk) Optim5. Y = f(X1, X2, ..., Xk) = c Cons6 Y = f(X1 X2 Xk) Dime



6. 

Y

= f(X1, X2, ..., Xk

)

 Dime
The Starting Point: Generic Model (P

System Behavior Y = f(X1, X2, ..., Xk)

System Behavior Yi = fi(X1, X2, ..., Xk) (i = 1

Unknowns: (k=?,n=?,m
Factor Groups Affecting System Behav

1 Network Factors 1. Network Factors2. User Factors3. Source & Receiver Factors4. Protocol Factors
Factors Xi Affecting System Behavio

Yi = fi(X1 X2 Xk)

Network

x1 Propagation delay

x2 Network speed

Yi fi(X1, X2, ..., Xk)

Factors x2 Network speed

x3 Buffer sizing

User

x4 Average file size for web pagesx5 Average think time between web clicks Factors x5 Average think time between web clicksx6 Probability a user opts to transfer a largx7 Probability a source or receiver is on 

host

Source & Receiver Factors

x8 Scaling factor for number of sou

receivers

x9 Distribution of sources

x10 Distribution of receivers

ProtocolFactors x11 Initial TCP slow-start threshold

(k=11,n=?,m
n <= 100

Affordable Number of Runs n = ?

n <= 100

4 Ways to Reduce DEX Full Factorial Des

1. Reduce # Factors (but scope reduc

2 R d N b f L l ( 2?) 2. Reduce Number of Levels (=> 2?)3. Reduce Number of Reps4 Fractional Factorial Design 4. Fractional Factorial Design

(k=11,n <= 100,m
n <= 100

Affordable Number of Runs n = ?

n <= 100

4 Ways to Reduce DEX Full Factorial Des

1. Reduce # Factors (but scope reduc

2 R d N b f L l (2?) 2. Reduce Number of Levels (2?)3. Reduce Number or Reps4 Fractional Factorial Design 4. Fractional Factorial Design

(k=11,n <= 100,m
Affordable Number of Runs n = ?

Additional Desirable Feature of the Design

Good Estimates for (at least) the Good Estimates for (at least) theMain Effects & 2-Term Interactions(Resolution)11 + 11-choose-2 = 11 + 55 = 66

(66+1) = 67

 64 



26 



211-5

Fi l D i 2 Final Design: 

211

5 O th l 2 L l F 

11

-

5

O

rthogonal 2

-

Level Fra

Factorial Design (k=11,n=64)

(k=11,n=64,m
MesoNet Factors (k=11) & Levels (2

Category Factor Code Definition Level 1: - L

x1 PDM Propagation delay 1 

NetworkFactors

x1 PDM Propagation delay 1 x2 BRS (s) Network speed 800 p/ms 4x3 QSA Buffer sizing RTTx

C/SQRT(

n) 

UserFactors

x4 AvFSWO Average file size for web pages 50 packets 10

x5 AvThT Average think time between web clicks 2000 ms 5

Probabilit

a ser opts x6 PrLF Probabilit

y

a

user opts

to transfer a larger file 0.02 

x7 PrFH

Probability a sourceor receiver is on a fasthost

0.4 

Source &Receiver Factors

x8 SFSR

Scaling factor fornumber of sources &receivers

2 

x9 SDist Distribution of

sources

WEB 

x10 RDist Distribution of receivers WEB 

Protocol

Factors x11 SST Initial TCP slow-start threshold 43 packets 1Factors threshold p
211-5 Orthogonal Fractional Factorial De

(k = 11 n,= 64)

Generators:

X7 = X3*X4*X5X8 = X1*X2*X3*X4X9 = X1 X2 X6 * *X10 = X2*X4*X5*X6X11 = X1*X4*X5*X6

Resolution IV

Reference: Box, Hunter, & Hunter, “Statistics for

Experimenters”, 2nd Edition, 2005, Wiley, p. 272
211-5 Fractional Factorial Design (k=11,n=64) 

(2to11m5.xls) 

Index X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X

1 -1 -1 -1 -1 -1 -1 -1 +1 -1 +1 +2 +1 -1 -1 -1 -1 -1 -1 -1 +1 +1 -3 -1 +1 -1 -1 -1 -1 -1 -1 +1 -1 +4 +1 +1 -1 -1 -1 -1 -1 +1 -1 -1 -5 -1 -1 +1 -1 -1 -1 1 -1 -1 +1 +6 +1 -1 +1 -1 -1 -1 1 +1 +1 +1 -7 -1 +1 +1 -1 -1 -1 1 +1 +1 -1 +8 +1 +1 +1 -1 -1 -1 1 -1 -1 -1 -9 -1 -1 -1 +1 -1 -1 1 -1 -1 -1 -

10 +1 -1 -1 +1 -1 -1 1 +1 +1 -1 +

11 -1 +1 -1 +1 -1 -1 1 +1 +1 +1 -12 +1 +1 -1 +1 -1 -1 1 -1 -1 +1 +13 -1 -1 +1 +1 -1 -1 -1 +1 -1 -1 -14 +1 -1 +1 +1 -1 -1 -1 -1 +1 -1 +15 -1 +1 +1 +1 -1 -1 -1 -1 +1 +1 -16 +1 +1 +1 +1 -1 -1 -1 +1 -1 +1 +17 -1 -1 -1 -1 +1 -1 1 +1 -1 -1 -18 +1 -1 -1 -1 +1 -1 1 -1 +1 -1 +19 -1 +1 -1 -1 +1 -1 1 -1 +1 +1 -20 +1 +1 -1 -1 +1 -1 1 +1 -1 +1 +21 -1 -1 +1 -1 +1 -1 -1 -1 -1 -1 -22 +1 -1 +1 -1 +1 -1 -1 +1 +1 -1 +23 -1 +1 +1 -1 +1 -1 -1 +1 +1 +1 -24 +1 +1 +1 -1 +1 -1 -1 -1 -1 +1 +25 -1 -1 -1 +1 +1 -1 -1 -1 -1 +1 +26 +1 -1 -1 +1 +1 -1 -1 +1 +1 +1 -27 -1 +1 -1 +1 +1 -1 -1 +1 +1 -1 +28 +1 +1 1 +1 +1 1 1 1 1 1 28 +1 +1 -1 +1 +1 -1 -1 -1 -1 -1 -29 -1 -1 +1 +1 +1 -1 1 +1 -1 +1 +30 +1 -1 +1 +1 +1 -1 1 -1 +1 +1 -31 -1 +1 +1 +1 +1 -1 1 -1 +1 -1 +32 +1 +1 +1 +1 +1 -1 1 +1 -1 -1 -
33 -1 -1 -1 -1 -1 +1 -1 +1 +1 -1 

34 +1 -1 -1 -1 -1 +1 -1 -1 -1 -1 35 -1 +1 -1 -1 -1 +1 -1 -1 -1 +1 36 +1 +1 -1 -1 -1 +1 -1 +1 +1 +1 37 -1 -1 +1 -1 -1 +1 1 -1 +1 -1 38 +1 -1 +1 -1 -1 +1 1 +1 -1 -1 39 -1 +1 +1 -1 -1 +1 1 +1 -1 +1 40 +1 +1 +1 -1 -1 +1 1 -1 +1 +1 41 -1 -1 -1 +1 -1 +1 1 -1 +1 +1 42 +1 -1 -1 +1 -1 +1 1 +1 -1 +1 43 -1 +1 -1 +1 -1 +1 1 +1 -1 -1 44 +1 +1 -1 +1 -1 +1 1 -1 +1 -1 45 -1 -1 +1 +1 -1 +1 -1 +1 +1 +1 46 +1 -1 +1 +1 -1 +1 -1 -1 -1 +1 47 -1 +1 +1 +1 -1 +1 -1 -1 -1 -1 48 +1 +1 +1 +1 -1 +1 -1 +1 +1 -1 49 -1 -1 -1 -1 +1 +1 1 +1 +1 +1 50 +1 -1 -1 -1 +1 +1 1 -1 -1 +1 51 -1 +1 -1 -1 +1 +1 1 -1 -1 -1 52 +1 +1 -1 -1 +1 +1 1 +1 +1 -1 53 -1 -1 +1 -1 +1 +1 -1 -1 +1 +1 54 +1 -1 +1 -1 +1 +1 -1 +1 -1 +1 55 -1 +1 +1 -1 +1 +1 -1 +1 -1 -1 56 +1 +1 +1 -1 +1 +1 -1 -1 +1 -1 57 -1 -1 -1 +1 +1 +1 -1 -1 +1 -1 58 +1 -1 -1 +1 +1 +1 -1 +1 -1 -1 59 -1 +1 -1 +1 +1 +1 -1 +1 -1 +1 60 +1 +1 -1 +1 +1 +1 -1 -1 +1 +1 61 -1 -1 +1 +1 +1 +1 1 +1 +1 -1 62 1 1 1 1 1 1 1 1 1 1 62

+1 -1 +1 +1 +1 +1 1 -1 -1 -1 

63 -1 +1 +1 +1 +1 +1 1 -1 -1 +1 64 +1 +1 +1 +1 +1 +1 1 +1 +1 +1 

345 1234 126 2456 1
What does this design look like? Why use i

(k=11,n=64) 

 (k=7,n=8) 

(k=7,n=8) 27-4Orthogonal Fractional Factorial 

X5

+

X5

+

X4

+

X1

X2

X3

+

+

_

_

_

_

+ X4

+

X1

X2

X3

+

+_

_

_

_

+

+ +

X5

+

X1

X2

X3

+

+

_

_

_

X5

+

X1

X2

X3

+

+_

_

_

X4 _ + X4 _ +
What does this design not look like? 

(k=7,n=8) 1FAT Fractional Factorial Desig

X5

+

X5

+

X4

+

X1

X2

X3

+

+

_

_

_

_

+ X4

+

X1

X2

X3

+

+_

_

_

_

+

+ +

X5

+

X1

X2

X3

+

+

_

_

_

X5

+

X1

X2

X3

+

+_

_

_

X4 _ + X4 _ +
Measures of System Behavior

(Response Variables) (Response Variables)

Yi = fi(X1, X2, ..., Xk)

1. Characterizing Macroscopic Behavio2. Characterizing Instantaneous Throug

for Active Flows by Flow Class (User

(k=11,n=64,m
16 Responses Characterizing Macroscopic 

Response Definition

y

1 Active Flows – flows attemptin

g to transfer data

y2 Proportion of potential flows that were active: Active Flowy3 Data packets entering the network per measurement intervy4 Data packets leaving the network per measurement intervay

5 Loss Rate: 

y (y y ) 4/(y3+

y

4)

y6 Flows Completed per measurement intervaly7 Flow-Completion Rate: y6/(y6+y1)y8 Connection Failures per measurement intervaly9 Connection-Failure Rate: 

y (y y ) 8/(y8+

y

1)

y10 Retransmission Rate (ratio)

y11 Congestion Window per Flow (packets)

y12 Window Increases per Flow per measurement intervaly13 Negative Acknowledgments per Flow per measurement int

Negative Acknowledgments per Flow per measurement int

y14 Timeouts per Flow per measurement intervaly15 Smoothed Round-Trip Time (ms)y16 Relative queuing delay: y15/(x1x41)
6 Responses Characterizing Instantaneous T

for Active Flows by Flow Class for Active Flows by Flow Class

Response Definition (Throughput in packets/second)

y17 Average Throughput for Active DD Flows

y18 Average Throughput for Active

D

F Flows

y19 Average Throughput for Active

D

N Flows

y20 Average Throughput fo

r

Active FF Flows

y21 Average Throughput for Active

F

N Flows

y22 Avera

ge Throu

g ph

put fo

r

Active NN Flows

Router Type SpeedBackbone

2

s

PoP 25 % of 

s

D-class Access 25 % of 

s

F-class Access 5 % of 

s

N-class Access 2.5 % of 
MesoNet 22 Responses: 16 Macro + 6 Thro

Response Definition

y1 Active Flows Active Flows – flows attempting to transfer data flows attempting to transfer datay2 Proportion of potential flows that were active: Active Flows/Aly3 Data packets entering the network per measurement intervaly4 Data packets leaving the network per measurement intervaly

5 L R t 4/( 3+ 4) Loss 

R

a

te: y4/(

y3+

y4)

y6 Flows Completed per measurement intervaly7 Flow-Completion Rate: y6/(y6+y1)y8 Connection Failures per measurement intervaly

9 C ti onnection-F il R t 8/( 8 1) F

ailure 

R

a

te: y8/(

y

8+y1)

y10 Retransmission Rate (ratio)

y11 Congestion Window per Flow (packets)

y12 Window Increases per Flow per measurement intervaly13 Negative Acknowledgments per Flow per measurement intervay14 Timeouts per Flow per measurement intervaly15 Smoothed Round-Trip Time (ms)y16 Relative queuing delay: y15/(x1x41)

y17 Average Throughput for Active DD Flows

y18 Average Throughput for Active

D

F Flows

y19 Average Throughput for Active

D

N Flows

y20 Average Throughput for Active FF Flowsy21 Average Throughput for Active

F

N Flows

y22 Average Throughput for Active NN Flows

(k=11
General Problem-Solving Frame

Expert

1

2

5

Problem Solu

3

4

Data DEX = 

g(k=11,n=64)

1. Graphical2. Quantitative

Q1. Response Dimension?Q2. Important Factors?Q3 Best Factor Settings?

A1. # & Set {A2. List (Ran

Q3. Best Factor Settings? A3 Vector (xQ4. Improvement over TCP?

A3. Vector (xA4. Y/N & Be
Data: 64 x 22 Multivariate Data Set Resultin

211

-

5 Orthogonal Fractional Factorial Experime

Orthogonal Fractional Factorial Experime

Run y1 y2

... y21 y22

1 4680.619 0.168126

... 92.034 89.78

2 6654.512 0.239371

... 72.596 57.73

3 9431 405 9431.405 0 339259 0.339259

... 29 569 . 13 96

.

4 11565.81 0.415439

... 23.427 19.88

... ... ...... ... ...

61 10319.55 0.247471

... 87.969 41.57

62 1738.469 0.093668

... 159.298 161.60

63 1783.509 0.096094

... 148.395 161.3

64 21467.6 0.514811

... 26.159 9.98
General Problem-Solving Frame

Expert

1

2

5

Problem Solu

3

4

Data DEX = 

g(k=11,n=64)

1. Graphical2. Quantitative

Q1. Response Dimension?Q2. Important Factors?Q3 Best Factor Settings?

A1. # & Set {A2. List (Ran

Q3. Best Factor Settings? A3 Vector (xQ4. Improvement over TCP?

A3. Vector (xA4. Y/N & Be
Sensitivity Analysis
Sensitivity Analysis

Q1. Of the 11 factors, what are most/leas

important (including interactions)?

Q2 Robust over the 22 responses? Q2. Robust over the 22 responses?
Analysis: For each of the 22 responses ...

Response Definition Example 1: Y10 = Retransmission Ra

y1 Active Flows – flows attempting to transfer datay2 Proportion of potential flows that were active: Active Flowsy3 Data packets entering the network per measurement intervy4 Data packets leaving the network per measurement intervay5 Loss Rate: y4/(y3+y4)y6 Flows Completed per measurement intervaly7 Flow-Completion Rate: y6/(y6+y1)y

8 Connection Failures 

per measurement interval

y9 Connection-Failure Rate: y8/(y8+y1)

y10 Retransmission Rate (ratio)

y11 Congestion Window per Flow (packets)

y12 Window Increases 

p p er Flow 

per measurement interval

y13 Negative Acknowledgments per Flow per measurement inty14 Timeouts per Flow per measurement intervaly15 Smoothed Round-Trip Time (ms)y16 Relative queuing delay: y15/(x1x41 Relative queuing delay: y15/(x1x41)

y17 Average Throughput for Active DD Flows

y18 Average Throughput for Active

D

F Flows

y19 Average Throughput for Active

D

N Flows

y19 Average Throughput for Active

D

N Flows

y20 Average Throughput for Active FF Flowsy21 Average Throughput for Active

F

N Flows

y22 Average Throughput for Active NN Flows

(k=11
Y10: Retransmission Rate

Main Effects Plot (Augmented)
Y10: Retransmission Rate

Main Effects Plot (Augmented)
Y10: Retransmission Rate

Main Effects Plot (Augmented)

Means: (+ - + - + + + -
Y10: Retransmission Rate

Interaction Effects Matrix
Y10: Retransmission Rate

Interaction Effects Matrix
Y10: Retransmission Rate

Interaction Effects Matrix

http://www.itl.nist.gov/div898/handbook/pri/section5/pri59.h
Y10: Retransmission Rate

Ordered Data Plot
Y10: Retransmission Rate

Ordered Data Plot

+ 

‐ +

7 13 x1

15 5 x2

4 16 x3

16 4 x4 

6 14 x5

10 10 x6

9 11 x7

13 7 x8

14

6

9 

14

6

x

9

10 10 x1010 10 x11
Y10: Retransmission Rate

Ordered Data Plot

+ 

‐ +

7 13 x1

15 5 x2

4 16 x3

16 4 x4

Left:(+

+

+

+

)

6 14 x5

10 10 x6

9 11 x7

13 7 x8

14

6

9

Means:(+ - + - + + + -

- + -)

Left:(+

-

+ -

+ . 

+ -

-

. .

)

Right:(+ - + 

-

+ . + -

- + .)

14

6

x

9

10 10 x1010 10 x11
Response Definition

Example 2: Y17 = Ave. TP for Active DD 

y1 Active Flows – flows attempting to transfer datay2 Proportion of potential flows that were active: Active Flowsy3 Data packets entering the network per measurement intervy4 Data packets leaving the network per measurement intervay5 Loss Rate: y4/(y3+y4)y6 Flows Completed per measurement intervaly7 Flow-Completion Rate: y6/(y6+y1)y8 Connection Failures per measurement intervaly9 Connection-Failure Rate: y8/(y8+y1)

y10 Retransmission Rate (ratio)

y11 Congestion Window per Flow (packets)

y12 Window Increases p p er Flow per measurement intervaly13 Negative Acknowledgments per Flow per measurement inty14 Timeouts per Flow per measurement intervaly15 Smoothed Round-Trip Time (ms)y16 Relative queuing delay: y15/(x1x41 Relative queuing delay: y15/(x1x41)

y17 Average Throughput for Active DD Flows

y18 Average Throughput for Active DF Flows

y19 Average Throughput for Active DN Flowsy19 Average Throughput for Active DN Flowsy20 Average Throughput for Active FF Flowsy21 Average Throughput for Active FN Flowsy22 Average Throughput for Active NN Flows

(k=11
Y17: Average Throughput for Active DD Flows 

Main Effects Plot (Augmented)
Main Effects Plot (Augmented)

Y17: Average Throughput for Active DD Flows 
Main Effects Plot (Augmented)

Y17: Average Throughput for Active DD Flows 

Means: (

- - + 

+ - - - + 
Y17: Average Throughput for Active DD Flows 

Interaction Effects Matrix
Y17: Average Throughput for Active DD Flows 

Interaction Effects Matrix
Y17: Average Throughput for Active DD Flows 

Interaction Effects Matrix
Ordered Data Plot

Y17: Average Throughput for Active DD Flows 
Ordered Data Plot

Y17: Average Throughput for Active DD Flows 

‐ + 

0 20 x1

10 10 x211 9 x3

16 4 x416 4 x4 11 9 x5

9 11 x6

10 10 x711 9 x811 9 x8 11 9 x910 10 x1012 8 x11
Ordered Data Plot

Y17: Average Throughput for Active DD Flows 

‐ + 

0 20 x1

10 10 x211 9 x3

16 4 x416 4 x4 11 9 x5

9 11 x6

10 10 x711 9 x8

Left:( + +) 11 9 x8 11 9 x9

10 10 x1012 8 x11

Means:(- - + + - - - + . + +) 

Left:(- . . + . . . . . . +)

Right:(- . . + . - . . . + .) 
Robustness Assessment: Stacked Main Effects Plot
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 

Robustness Assessment: (1-Way) ANOVA CDF Values (uno

X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 

PDM BRS QSA AvFSW AVThT PrLF PrFH SFSR SDist RDist 

Y1 51.16 98.66 37.27 99.83 100 20.6 13.47 99.93 99.98 15.78 

Y2 62.1 99.84 35.47 99.99 100 31.51 18.87 91.42 96.85 22.06 Y3 42.48 100 15.96 99.97 100 28.33 7.84 97.49 87.44 33.63 Y4 23.87 100 30.78 99.88 99.99 27.37 9.48 94.56 74.98 42.21 Y5 86.91 99.99 99.98 96.28 97.66 13.56 23.51 94.83 98.87 54.01 Y6 14.99 100 47.49 99.99 99.99 29.52 19.27 92.44 71.04 36.43 Y7 84.55 99.99 41.43 100 99.82 24.79 16.99 94.31 99.37 27.9 Y8 83.44 98.98 99.06 70.34 95.79 45.54 18.13 95.79 99.25 44.83 Y9 91.84 99.57 99.89 49.3 95.69 20.05 13.19 88.83 99.21 62.88 

Y10 86.67 99.97 99.97 95.67 95.87 21.7 29.64 94.35 99.21 48.61 Y11 22.45 99.94 99.09 17.5 98.91 45.27 48.81 80.41 98.37 62.46 Y12 87 12 . 99 99 . 71 44 . 96 85 . 99 91 . 3 49 . 23 44 . 87 87 . 99 4. 38 95 . Y13 99.47 96.76 100 93.93 95.28 31.3 42.08 55.53 83.6 22.11 Y14 99.68 99.32 100 70.85 95.1 2.42 44.48 81.68 95.31 30.49 Y15 100 88.52 100 83.64 76 18.17 8.34 71.77 69.49 5.28 Y16 81.89 91.56 100 87.83 82.66 22.07 4.41 76.31 79.34 13.34 Y17 100 16.78 3.28 100 21.28 27.89 24.66 34.01 2.89 16.24 Y18 100 99.09 67.06 99.45 94.98 47.51 11.33 84.16 99.41 42.36 Y19 95.05 100 70.38 43.16 99.94 10.71 30.51 95.02 99.94 66.59 Y20 99.98 99.71 70.05 95.48 98.11 33.15 0.96 85.65 99.85 47.06 Y21 93 100 73 21 59 53 99 98 17 79 32 17 97 03 98 34 34 56 Y21 93 100 73.21 59.53 99.98 17.79 32.17 97.03 98.34 34.56 Y22 83.79 100 69.13 69.1 99.96 12.49 30.32 95.01 99.95 59.86 

Sum 7 19 9 13 18 0 0 11 15 0 
X2 X5 X9 X4 X8 X3 X1 X11 X10 X7 

Robustness Assessment: (1-Way) ANOVA CDF Values (ord

X2 X5 X9 X4 X8 X3 X1 X11 X10 X7 

BRS AVThT SDist AvFSW SFSR QSA PDM SST RDist PrFH 

Y1 98.66 100 99.98 99.83 99.93 37.27 51.16 14.39 15.78 13.47 

Y2 99.84 100 96.85 99.99 91.42 35.47 62.1 14.69 22.06 18.87 Y3 100 100 87.44 99.97 97.49 15.96 42.48 38.25 33.63 7.84 Y4 100 99.99 74.98 99.88 94.56 30.78 23.87 42.83 42.21 9.48 Y5 99.99 97.66 98.87 96.28 94.83 99.98 86.91 18.52 54.01 23.51 Y6 100 99.99 71.04 99.99 92.44 47.49 14.99 36.74 36.43 19.27 Y7 99.99 99.82 99.37 100 94.31 41.43 84.55 57.22 27.9 16.99 Y8 98.98 95.79 99.25 70.34 95.79 99.06 83.44 42.18 44.83 18.13 Y9 99.57 95.69 99.21 49.3 88.83 99.89 91.84 45.21 62.88 13.19 

Y10 99.97 95.87 99.21 95.67 94.35 99.97 86.67 26.76 48.61 29.64 Y11 99.94 98.91 98.37 17.5 80.41 99.09 22.45 98.93 62.46 48.81 Y12 99 99 . 99 91 . 99 4. 96 85 . 87 87 . 71 44 . 87 12 . 98 02 . 38 95 . 23 44 . Y13 96.76 95.28 83.6 93.93 55.53 100 99.47 43.18 22.11 42.08 Y14 99.32 95.1 95.31 70.85 81.68 100 99.68 2.75 30.49 44.48 Y15 88.52 76 69.49 83.64 71.77 100 100 8.59 5.28 8.34 Y16 91.56 82.66 79.34 87.83 76.31 100 81.89 0.82 13.34 4.41 Y17 16.78 21.28 2.89 100 34.01 3.28 100 27.46 16.24 24.66 Y18 99.09 94.98 99.41 99.45 84.16 67.06 100 62.51 42.36 11.33 Y19 100 99.94 99.94 43.16 95.02 70.38 95.05 53.33 66.59 30.51 Y20 99.71 98.11 99.85 95.48 85.65 70.05 99.98 73.11 47.06 0.96 Y21 100 99 98 98 34 59 53 97 03 73 21 93 61 62 34 56 32 17 Y21 100 99.98 98.34 59.53 97.03 73.21 93 61.62 34.56 32.17 Y22 100 99.96 99.95 69.1 95.01 69.13 83.79 63.94 59.86 30.32 

Sum 19 18 15 13 11 9 7 200
Robustness Assessment: Multiplot of (1-Way) ANOVA CD
Robustness Assessment: Multiplot of (1-Way) ANOVA CD
Robustness Assessment: Multiplot of (1-Way) ANOVA CD

1
Robustness Assessment: Multiplot of (1-Way) ANOVA CD

1

2
Robustness Assessment: Multiplot of (1-Way) ANOVA CD

1

23
Robust Sensitivity Analysis Ranking (Criterio

Major Factors (ordered) influencing MesoNet behavior:

X2: Network Speed

X5: Think Time X5: Think TimeX9: Distribution of SourcesX4: File SizeX8: Number of Sources

Minor Factor influencing MesoNet behavior:

X3: Buffer Size – small buffer sizes reduces delay va

lar

g g e buffer size has 

greater effect 

high network speed

X1: Propagation Delay

Non

-Factors

X11: Initial TCP Slow-Start ThresholdX10: Distribution of ReceiversX7: Probability a Source or Receiver is on a Fast HosX6: Probability a User Opts to Transfer a Larger File
Robust Sensitivity Analysis Ranking (Criterio

Major Factors (ordered) influencing MesoNet behavior:

X2: Network Speed

X4: File Size X4: File SizeX5: Think TimeX8: Number of SourcesX1: Propagation DelayX9 Distrib tion of So rces X9: Distrib

ution of So

urces

Minor Factor influencing MesoNet behavior:

X3: Buffer Size – small buffer sizes reduces dela

y 

 va

large buffer size has greater effect high network speed

Non-Factors

X11: Initial TCP Slow X11: Initial TCP Slow

-Start Threshold Start Threshold

X10: Distribution of ReceiversX7: Probability a Source or Receiver is on a Fast HosX6: Probability a User Opts to Transfer a Larger File
D fi iti L l1 L 

Robust Sensitivity Analysis Ranking (Criterio

Category Factor Code Definition Level 1: - L

NetworkFactors

x1 PDM Propagation delay 1 

x2 BRS (s) Network speed 800 p/ms 4

x3 QSA B ff u er si i z ng RTTxC/SQRT(n) 

User

x4 AvFSWO Average file size for web pages 50 packets 10

x5 AvThT Average think time 2000 ms 5

Factors x5 AvThT g between web clicks 2000 ms 5

x6 PrLF Probability a user opts to transfer a larger file 0.02 x7 PrFH

Probability a sourceor receiver is on a fast 0 4 

Source &Receiver 

x7 PrFH or receiver is on a fast

host

0.4 

x8 SFSR

Scaling factor fornumber of sources &receivers

2 

Factors x9 SDist Distribution of

sources

WEB 

x10 RDist Distribution of receivers WEB 

ProtocolFactors x11 SST Initial TCP slow-start threshold 43 packets 1

p
Di i R d ti Dimension 

R

e

duction

Anal

ysis
We Applied Two Different Techniq

Principal

Components

A li

m responses (y

1, ...

y

m)

Analysis

m

–

d

1 responses Domain

Expertise

m responses (y

1, ...

y

m)

Analysis

m

–

d

1 responses Domain

Expertise

m responses (y

1, ...

y

m)

Analysis

m

–

d

1 responses Domain

Expertise

Correlation

m

–

d

2 responses Expertise

m

–

d

3

responses

Correlation

m

–

d

2 responses Expertise

m

–

d

3

responses

Correlation

m

–

d

2 responses Expertise

m

–

d

3

responses Analysis

DATA SCIENTIFIC SCIENTIFIC

& Clustering

DOMAIN

EXPERTISE

ANALYSIS

EXPERTISE

SCIENTIFIC

DOMAIN

EXPERTISE

ANALYSIS

EXPERTISE
Abilene Network (3-Tier MesoNet Topol
22 Responses: 16 Macro + 6 Through

Response Definition

y1 Active Flows – flows attempting to transfer datay2 Proportion of potential flows that were active: Active Flowsy3 Data packets entering the network per measurement intervy4 Data packets leaving the network per measurement intervay5 Loss Rate: y4/(y3+y4)y6 Flows Completed per measurement intervaly7 Flow-Completion Rate: y6/(y6+y1)y

8 Connection Failures 

per measurement interval

y9 Connection-Failure Rate: y8/(y8+y1)

y10 Retransmission Rate (ratio)

y11 Congestion Window per Flow (packets)

y12 Window Increases 

p p er Flow 

per measurement interval

y13 Negative Acknowledgments per Flow per measurement inty14 Timeouts per Flow per measurement intervaly15 Smoothed Round-Trip Time (ms)y16 Relative queuing delay: y15/(x1x41 Relative queuing delay: y15/(x1x41)

y17 Average Throughput for Active DD Flows

y18 Average Throughput for Active

D

F Flows

y19 Average Throughput for Active

D

N Flows

y19 Average Throughput for Active

D

N Flows

y20 Average Throughput for Active FF Flowsy21 Average Throughput for Active

F

N Flows

y22 Average Throughput for Active NN Flows

(k=11
Data: 64 x 22 Multivariate Data Set Resultin

211

-

5 Orthogonal Fractional Factorial Experime

Orthogonal Fractional Factorial Experime

Run y1 y2

... y21 y22

1 4680.619 0.168126

... 92.034 89.78

2 6654.512 0.239371

... 72.596 57.73

3 9431 405 9431.405 0 339259 0.339259

... 29 569 . 13 96

.

4 11565.81 0.415439

... 23.427 19.88

... ... ...... ... ...

61 10319.55 0.247471

... 87.969 41.57

62 1738.469 0.093668

... 159.298 161.60

63 1783.509 0.096094

... 148.395 161.3

64 21467.6 0.514811

... 26.159 9.98
Method 1: Correlation

Analysis & Clustering
Matrix of Pair-wise Scatter Plots & Correlation Coeffic

Red 80 > |r|x100 < 100 Blue 30 > |r|x100 < 80 Green |r|x100 < 30 
Sorted Correlations
Sorted Correlations
Matrix of Pair-wise Scatter Plots & Correlation Coefficients 

Red 80 > |r|x100 < 100 Blue 30 > |r|x100 < 80 Green |r|x100 < 30 
R de 80 > | | 100 |r|x100 < 100 Blue 30 > | | 100 80 |r|x100 < 80 Green | | 100 30 |r|x100 < 30

(a) Pair-wise Correlation Matrix (b) Histogram: bins where |r| > 0.
Frequency Distribution of Absolute Value of Corr

Coefficients for All Response Pairs

Select a threshold for |r| such that correlations above that threshold will be further 

We chose |r| > 0.65
Response Index-Index Plot where |ri,j| > 0.6

Clustered into Mutual Correlations
Response Index-Index Plot where |ri,j| > 0.6

Clustered into Mutual Correlations

Plot suggests MesoNet exhibits 

7 distinct behaviors
22 Responses: 16 Macro + 6 Through

Response Definition

y1 Active Flows Active Flows – flows attempting to transfer data flows attempting to transfer datay2 Proportion of potential flows that were active: Active Flowsy3 Data packets entering the network per measurement intervy4 Data packets leaving the network per measurement intervay

5 L R t 4/( 3+ 4) Loss 

R

a

te: y4/(

y3+

y4)

y6 Flows Completed per measurement intervaly7 Flow-Completion Rate: y6/(y6+y1)y8 Connection Failures per measurement intervaly

9 C ti onnection-F il R t 8/( 8 1) F

ailure 

R

a

te: y8/(

y

8+y1)

y10 Retransmission Rate (ratio)

y11 Congestion Window per Flow (packets)

y12 Window Increases per Flow per measurement intervaly13 Negative Acknowledgments per Flow per measurement inty14 Timeouts per Flow per measurement intervaly15 Smoothed Round-Trip Time (ms)y16 Relative queuing delay: y15/(x1x41)

y17 Average Throughput for Active DD Flows

y18 Average Throughput for Active

D

F Flows

y19 Average Throughput for Active

D

N Flows

y20 Average Throughput for Active FF Flowsy21 Average Throughput for Active

F

N Flows

y22 Average Throughput for Active NN Flows

(k=11
Matrix of Pair-wise Scatter Plots & Correlation Coefficients 

Red 80

> |r|x100

< 100 Blue 30

> |r|x100 < 80 Green |r|x100 < 30

Red 80

> |r|x100

< 100 Blue 30

> |r|x100

< 80 Green |r|x100

< 30
22 Responses: 16 Macro + 6 Through

Response Definition

y1 Active Flows Active Flows – flows attempting to transfer data flows attempting to transfer datay2 Proportion of potential flows that were active: Active Flowsy3 Data packets entering the network per measurement intervy4 Data packets leaving the network per measurement intervay

5 L R t 4/( 3+ 4) Loss 

R

a

te: y4/(

y3+

y4)

y6 Flows Completed per measurement intervaly7 Flow-Completion Rate: y6/(y6+y1)y8 Connection Failures per measurement intervaly

9 C ti onnection-F il R t 8/( 8 1) F

ailure 

R

a

te: y8/(

y

8+y1)

y10 Retransmission Rate (ratio)

y11 Congestion Window per Flow (packets)

y12 Window Increases per Flow per measurement intervaly13 Negative Acknowledgments per Flow per measurement inty14 Timeouts per Flow per measurement intervaly15 Smoothed Round-Trip Time (ms)y16 Relative queuing delay: y15/(x1x41)

y17 Average Throughput for Active DD Flows

y18 Average Throughput for Active

D

F Flows

y19 Average Throughput for Active

D

N Flows

y20 Average Throughput for Active FF Flowsy21 Average Throughput for Active

F

N Flows

y22 Average Throughput for Active NN Flows

(k=11
Matrix of Pair-wise Scatter Plots & Correlation Coefficients 

Red 80

> |r|x100

< 100 Blue 30

> |r|x100 < 80 Green |r|x100 < 30

Red 80

> |r|x100

< 100 Blue 30

> |r|x100

< 80 Green |r|x100

< 30
22 Responses: 16 Macro + 6 Through

Response Definition

y1 Active Flows Active Flows – flows attempting to transfer data flows attempting to transfer datay2 Proportion of potential flows that were active: Active Flowsy3 Data packets entering the network per measurement intervy4 Data packets leaving the network per measurement intervay

5 L R t 4/( 3+ 4) Loss 

R

a

te: y4/(

y3+

y4)

y6 Flows Completed per measurement intervaly7 Flow-Completion Rate: y6/(y6+y1)y8 Connection Failures per measurement intervaly

9 C ti onnection-F il R t 8/( 8 1) F

ailure 

R

a

te: y8/(

y

8+y1)

y10 Retransmission Rate (ratio)

y11 Congestion Window per Flow (packets)

y12 Window Increases per Flow per measurement intervaly13 Negative Acknowledgments per Flow per measurement inty14 Timeouts per Flow per measurement intervaly15 Smoothed Round-Trip Time (ms)y16 Relative queuing delay: y15/(x1x41)

y17 Average Throughput for Active DD Flows

y18 Average Throughput for Active

D

F Flows

y19 Average Throughput for Active

D

N Flows

y20 Average Throughput for Active FF Flowsy21 Average Throughput for Active

F

N Flows

y22 Average Throughput for Active NN Flows

(k=11
Matrix of Pair-wise Scatter Plots & Correlation Coefficients 

Red 80

> |r|x100

< 100 Blue 30

> |r|x100 < 80 Green |r|x100 < 30

Red 80

> |r|x100

< 100 Blue 30

> |r|x100

< 80 Green |r|x100

< 30
22 Responses: 16 Macro + 6 Through

Response Definition

y1 Active Flows – flows attempting to transfer datay2 Proportion of potential flows that were active: Active Flowsy3 Data packets entering the network per measurement intervy4 Data packets leaving the network per measurement intervay5 Loss Rate: y4/(y3+y4)y6 Flows Completed per measurement intervaly7 Flow-Completion Rate: y6/(y6+y1)y8 Connection Failures per measurement intervaly9 Connection-Failure Rate: y8/(y8+y1)

y10 Retransmission Rate (ratio)

y11 Congestion Window per Flow (packets)

y12 Window Increases 

p p er Flow 

per measurement interval

y13 Negative Acknowledgments per Flow per measurement inty14 Timeouts per Flow per measurement intervaly15 Smoothed Round-Trip Time (ms)y16 Relative 

q g yy ( ueuin

g dela

y: y15/(x1x41)

y17 Average Throughput for Active DD Flows

y18 Average Throughput for Active

D

F Flows

y19 Average Throughput for Active

D

N Flows

y19 Average Throughput for Active

D

N Flows

y20 Average Throughput for Active FF Flowsy21 Average Throughput for Active

F

N Flows

y22 Average Throughput for Active NN Flows

(k=11
Matrix of Pair-wise Scatter Plots & Correlation Coefficients 

Red 80

> |r|x100

< 100 Blue 30

> |r|x100 < 80 Green |r|x100 < 30

Red 80

> |r|x100

< 100 Blue 30

> |r|x100

< 80 Green |r|x100

< 30
22 Responses: 16 Macro + 6 Through

Response Definition

y1 Active Flows – flows attempting to transfer datay2 Proportion of potential flows that were active: Active Flowsy3 Data packets entering the network per measurement intervy4 Data packets leaving the network per measurement intervay5 Loss Rate: y4/(y3+y4)y6 Flows Completed per measurement intervaly7 Flow-Completion Rate: y6/(y6+y1)y8 Connection Failures per measurement intervaly9 Connection-Failure Rate: y8/(y8+y1)

y10 Retransmission Rate (ratio)

y11 Congestion Window per Flow (packets)

y12 Window Increases 

p p er Flow 

per measurement interval

y13 Negative Acknowledgments per Flow per measurement inty14 Timeouts per Flow per measurement intervaly15 Smoothed Round-Trip Time (ms)y16 Relative 

q g yy ( ueuin

g dela

y: y15/(x1x41)

y17 Average Throughput for Active DD Flows

y18 Average Throughput for Active

D

F Flows

y19 Average Throughput for Active

D

N Flows

y19 Average Throughput for Active

D

N Flows

y20 Average Throughput for Active FF Flowsy21 Average Throughput for Active

F

N Flows

y22 Average Throughput for Active NN Flows

(k=11
Matrix of Pair-wise Scatter Plots & Correlation Coefficients 

Red 80

> |r|x100

< 100 Blue 30

> |r|x100 < 80 Green |r|x100 < 30

Red 80

> |r|x100

< 100 Blue 30

> |r|x100

< 80 Green |r|x100

< 30
22 Responses: 16 Macro + 6 Through

Response Definition

y1 Active Flows – flows attempting to transfer datay2 Proportion of potential flows that were active: Active Flowsy3 Data packets entering the network per measurement intervy4 Data packets leaving the network per measurement intervay5 Loss Rate: y4/(y3+y4)y6 Flows Completed per measurement intervaly7 Flow-Completion Rate: y6/(y6+y1)y8 Connection Failures per measurement intervaly9 Connection-Failure Rate: y8/(y8+y1)

y10 Retransmission Rate (ratio)

y11 Congestion Window per Flow (packets)

y12 Window Increases 

p p er Flow 

per measurement interval

y13 Negative Acknowledgments per Flow per measurement inty14 Timeouts per Flow per measurement intervaly15 Smoothed Round-Trip Time (ms)y16 Relative 

q g yy ( ueuin

g dela

y: y15/(x1x41)

y17 Average Throughput for Active DD Flows

y18 Average Throughput for Active

D

F Flows

y19 Average Throughput for Active

D

N Flows

y19 Average Throughput for Active

D

N Flows

y20 Average Throughput for Active FF Flowsy21 Average Throughput for Active

F

N Flows

y22 Average Throughput for Active NN Flows

(k=11
Matrix of Pair-wise Scatter Plots & Correlation Coefficients 

Red 80

> |r|x100

< 100 Blue 30

> |r|x100 < 80 Green |r|x100 < 30

Red 80

> |r|x100

< 100 Blue 30

> |r|x100

< 80 Green |r|x100

< 30
Summary: Response Index-Index Plot where |ri,j

Clustered into Mutual Correlations

2 responses uncorrelated

(1) throughput on DD flows

(2) flow completion rate

flow completion rate

14 correlationpairs reflectingpacket losses packet losses

3 pair-wise correlations:

(1) throughput on flows constrained by 

F-class 

(2) net ork dela net

work dela

y

(3) packets entering and leaving the network

Plot suggests MesoNet exhibits 7 distinct behaviors
Summary of Correlation Results

Correlation Analysis

Dimension Responses

Congestion y1, y2, y7, y11, y12, 

y19, y21, y22

Losses y5, y8, y9, y10, y13, 

y14

Delay y15, y16F-class TP y18, y20D-class TP y17Packet TP y3, y4Flow TP Flow TP y6
Summary of Correlation Results

Correlation Analysis

Dimension Responses

Congestion y1, y2, y7, y11, y12, 

y19, y21, y22

Losses y5, y8, y9, y10, y13, 

y14

Delay y15, y16F-class TP y18, y20D-class TP y17Packet TP y3, y4Flow TP Flow TP y6
22 Responses: 16 Macro + 6 Through

Response Definition

y1 Active Flows – flows attempting to transfer datay2 Proportion of potential flows that were active: Active Flowsy3 Data packets entering the network per measurement intervy4 Data packets leaving the network per measurement intervay5 Loss Rate: y4/(y3+y4)y6 Flows Completed per measurement intervaly7 Flow-Completion Rate: y6/(y6+y1)y8 Connection Failures per measurement intervaly9 Connection-Failure Rate: y8/(y8+y1)

y10 Retransmission Rate (ratio)

y11 Congestion Window per Flow (packets)

y12 Window Increases 

p p er Flow 

per measurement interval

y13 Negative Acknowledgments per Flow per measurement inty14 Timeouts per Flow per measurement intervaly15 Smoothed Round-Trip Time (ms)y16 Relative 

q g yy ( ueuin

g dela

y: y15/(x1x41)

y17 Average Throughput for Active DD Flows

y18 Average Throughput for Active

D

F Flows

y19 Average Throughput for Active

D

N Flows

y19 Average Throughput for Active

D

N Flows

y20 Average Throughput for Active FF Flowsy21 Average Throughput for Active

F

N Flows

y22 Average Throughput for Active NN Flows

(k

m
Correlation Analysis & Clustering Suggests MesoNe

Reflected in Only 7 Responses Reflected in Only

7 Responses

Response Definition

Average number of packet output per measurement int

y4 Average number of packet output per measurement int

(network throughput in packets/sec)

y6 Average number of flows completed per measurement 

(network throughput in flows/sec)

y10 Average Average retransmission retransmission rate (packet (packet loss)y15 Average smoothed round-trip time (network delay)

y17 Average instantaneous throughput for DD flows (throughput in packets/sec for the most advantaged us

A erage instantaneo s thro ghp t for FF flo s y20Average instantaneo

u

s thro

ughp

u

t for FF flo

w

s

(throughput in packets/sec for 2nd most advantaged us

y22 Average instantaneous throughput for NN flows (network congestion)
Q. Why is the Scatter Plot of Y7 vs Y22 Bifurca
Q. Why is the Scatter Plot of Y7 vs Y22 Bifurca
Method 2: Principal 

Components

Analysis
Principal Components Analysis of 22 MesoNet Re

Most response variance appears to be accounted for by the first 4 c
Weight Vectors for the first 4 Compo
|Weight| Vectors for the first 4 Compo

0.75

1

caled MAX = 1 PC1 0.75

1

PC2

scaled MAX = 1

0.25

0.5

ardized Weight| sc 0.25 0.5

dardized Weight| s

0

0 2 4 6 8 10 12 14 16 18 20 22

Response Identifier (y1-y22)

|Standa

0

0 2 4 6 8 10 12 14 16 

Response Identifier (y1-y2

|Stan

1

PC3

AX = 1

1 AX = 1PC4

0.5

0.75

Weight| scaled MA 0.5 0.75

d Weight| scaled MA

0

0.25

0 2 4 6 8 10 12 14 16 18 20 22

Response Identifier (y1-y22)

|Standardized 

0

0.25

|Standardize 0 2 4 6 8 10 12 14 16 1d

Response Identifier (y1-y2
Significant Responses in PC1 (congestion) Significant Responses in PC2 

Response Definition

y1 Average number of active flowsy2 Proportion of possible flows that are activey5 Loss rate

7 Fl l ti t

Significant Responses in PC1 (congestion)

Response Definition

y15 Smoothed round-trip timy16 Relative queuing delay

Significant Responses in PC2 

y7 Flow-completion rate

y8 Connection failuresy9 Connection-failure rate

y10 Retransmission ratey11 Average congestion windowy12 Window-increase rate

y13 Negative-acknowledgment ratey14 Timeout ratey19 Average instantaneous throughput for DN flowsy21 Average instantaneous throughput for FN flowsy22 Average instantaneous throughput for NN flows

Response Definition

Significant Responses in PC3 (throughput for advantaged users)

Response Definition

Significant Responses in 

(network throughput in flo

y3 Packets inputy4 Packets output

y17 Average instantaneous throughput for DD flows

y18 Average instantaneous throughput for DF flows

y20 Average instantaneous throughput for FF flows

y3 Packets inputy4 Packets outputy6 Flows completed per meas
Summary of PCA Results

PCA Analysis

Dimension Responses

125789 PC1:

Congestion

y

1, y

2, y

5, y

7, y

8, y

9, 

y10, y11, y12, y13, y14, y19, y21, y22

PC2: Delay PC2: Delay y15 y16 y15, y16PC3:D-class &F class TP

y3, y4, y17, 18, y20

-class TP

PC4: Flow TP y3, y4, y6

Comparing Correlation & PCA Res

Correlation Analysis PCA

Dimension Responses Dimension Resp

Congestion y1, y2, y7, y11, y12, y19, y21, 

y22 PC1:

Congestion

y1, y2, y5, y7, y11, y12, y13, 

Losses y5, y8, y9, y10, y13, y14 y22Delay y15, y16 PC2: Delay y15, y16F-class TP y18, y20 PC3:D-class TP y17 D-class & y3, y4, y17, 18

F-class TP Packet TP y3, y4

Flow TP y6 PC4: Flow TP y3, y4, y6

The results show good alignment:

PCA1 merges congestion + losses;PCA2 & Correlation identical for delay;

PCA3 merges D-class & F-class Throughput;PCA4 splits Packet TP acrosstwo dimensions (D- & F-class T
Identifyg g p ing Significant Resp

Dimensions for MesoNe

4 or 7 or something betwe4 or 7 or something betwe
HIGHER CONGESTION IS

LOWER TP: -X2, -X5, +X9, +X

y22 – NN TP Note X2 is miscoded so I reverse +/

PC+ IS: -X2, -X5, +X9, 

PC1

Note that PC interpretation is poresorting to cross-mapping with 

y10 – Retransmission Rate

I THINK LOSS & CONGESTBE SEPARATE – SIMILAR C

HIGHER IS -X2, -X3, +X9, +X4, -X5, +X8, -X1

SUBTLE DIFFERENCES
HIGHER TP: -X1, +X9, +X2, +

y20 – FF TP 

PC- IS: +X4, +X2, -X1, 

PC3

y17 – DD TP

I THINK D-class & F-class TH

SHOULD BE SEPARATE – O

HIGHER TP IS –X1, +X4

INPUT FACTORS INFLUENCD-class THROUGHPUT
HIGHER PO: +X2, -X5, +X4

y4 – Packet Output Rate 

PC- IS: +X2, -X5, -X4, -X

PC4

THIS SEEMS A BETTER FLOW COMPLETION R

y6 – Flow Completion Rate

I THINK FLOW COMPLETE I THINK FLOW COMPLETE PACKET THROUGHPUT RABE KEPT SEPARATE BECACOMPLETE IS HIGHER WIT

HIGHER FC IS +X2, -X4, -X5

FILE SIZE & PACKET OUTPWITH LARGE FILE SIZE 
Note: The Domain Analyst Sides With 

the Correlation Analysis Results the Correlation Analysis Results

Dimension Definition

1 Congestion2 Loss3 Delay4 Throughput Throughput for the most advantaged advantaged users5 Throughput for the somewhat advantaged user6 Network-wide Packet Throughput7 Network-wide Flow Throughput
Pros & Cons of the 2

Dimension Reduction

Techniques
Pros/Cons of Correlation Analysis & Cl

• Provided effective dimension reduction (22

→

7) through

Pros

• Provided effective dimension reduction (22

→

7

) through

correlations that could be vetted by a domain expert

• Examining response correlations helped to validate MesoN

• Uncovered nuanced differences Uncovered nuanced differences between flow and packet between flow and packet

throughput rates in a network

Cons

• A second 211-5 OFF experiment with different level settings 

some (valid) differences in correlations – thus separate co

f ff

Cons

analyses must be conducted 

for different level settings
Pros/Cons of Principal Components A

• Provided greater dimension reduction (22 

→ 4) than corre

analysis & clustering

Pros

analysis

& clustering

There is no specific domain interpretation of even the top 2 

Cons

• There is no specific domain interpretation of even the top

2 

principal components – in the case shown here we were aarrive at a reasonable interpretation; in other cases, we we

• Principal components take on + and Principal components take on

+ and

– values which prese

values, which prese

analysts with difficulty assigning meaning – we had to infe

of components by comparing them with meaning derived f

analyzing individual responses analyzing individual responses

• Principal components proved coarser than corresponding 

generated by clustering mutual correlations

• A second 2•

A second

211-5 OFF experiment with different level settings 

11-5 OFF experiment with different level settings 

some differences in principal components – such differenc

difficult to understand without assistance from other analys
Summary: Correlation Analysis or P

• If limited to one technique, correlation analysis provides re

easier for a domain analyst to comprehend

• Pi i l t t k + d P

rincipal componen

ts 

t

a

ke on 

+ an

d

– val hi h 

lues, whi

c

h prese

analysts with difficulty assigning meaning – we had to infe

by comparing main effects plots of principal components w

effects plots from responses chosen from groupings estabeffects plots from responses chosen from groupings estab

correlation analysis

• Principal components proved coarser than corresponding 

generated by clustering mutual correlations generated by clustering mutual correlations

• PCA provides a reasonable complement to correlation ana

giving a separate view of the data, which should be consis

correlation results thus helping to validate a model correlation results, thus helping to validate

a model
MesoNet Conclusions

• We investigated correlation and PC analyses as 

techniques to reduce the dimension of responseMesoNet, a network simulato

r

• We demonstrated that both techniques can sign

reduce the dimension of response data

• We also showed that We also showed that both techniques could be u

techniques could be u

validate a model, but that PCA is more suited as 

complement to correlation analysis

• We found that PCA results are difficult for a dom

analyst to interpret without comparison to analysindividual responses

• We also found that results from correlation and 

analyses with one set of parameter values cannnecessaril

y be extra

polated to a different set of v
Stat Conclusions

1. Stat Framework/App g

roach & Methodolog

Demo beginning-to-”end”

2. Critical importance of domain expert 2. Critical importance of domain expert3. Dimension Reduction dependency on

DEX & Sensitivity Analysis DEX

& Sensitivity Analysis

4. Internet Modeling Conclusions & Insight
Methodology Applications

1. MesoNet Analysis #1 (k=11,n = 64,m=22 

 7) 

S iti it & Di i Sensiti

vity 

& Dimension-R d ti A l i t d 

R

e

duction 

Analysis <

t

o

d

2. MesoNet Analysis #2 (k=20,n=256,m=22)

Sensitivity Analysis Sensitivity Analysis

3. MesoNet TCP Congestion/Control Alg. Comparison (k=

6,n

4. Cloud Computing Analysis (k=11,n=64,m=42 => 8) (Koala

Sensitivity & Dimension-Reduction Analysis

5. Cloud Computing VM Placement Alg. Comparison (k=

6,n=

(Koala)
Graphical Methods

1. Main Effects Plots

2. Interaction Effects Matrix Interaction Effects Matrix3. Ordered Data Plots4. Pairwide Scatter Plot Matrix (Unordered)5. Pairwise Scatter Plot Matrix (Ordered) Scatter Plot Matrix (Ordered)6. Stacked Main Effects Plot7. Multiplot of (1-Way) ANOVA CDF Values8. Index-Index Cluster Plot9. Character Plots

10. PCA Weights Plot 10. PCA Weights Plot
Presentations

J. Filliben, "Sensitivity Analysis Methodology for a Complex System Computational M

Symposium on the Interface: Computing Science and Statistics, Philadelphia, PA, Ma

K Mill d J Fillib "A Effi i t S iti it A l i M th d f K. Mills an

d

J. Filliben, "An Effi

cient Sensiti

vity 

Analysis 

M

eth

o

d for M i esoscopic N t 

N

etw

Complex Systems Study Group, NIST, February 2, 2010.K. Mills and J. Filliben, "Comparing Two Dimension-Reduction Methods for Network SModels", ( ), , y , , Winter Simulation Conference (WSC 2010), Baltimore, Mar

yland, Dec. 6, 20

K. Mills and J. Filliben, "Using Sensitivity Analysis to Identify Significant Parameters iSimulation", Winter Simulation Conference (WSC 2010), Baltimore, Maryland, Dec. 6K Mills J Filliben D K. Mills, J. Filliben, D.-Y Cho and E Schwartz Y. Cho and

E. Schwartz, "Predicting Macroscopic Dynamics in 

Predicting Macroscopic Dynamics in 

Distributed Systems", LSN Seminar on Complex Networks and Information Systems, Gaithersburg, Maryland, June 30, 2011.K. Mills, J. Filliben and C. Dabrowski, "An Efficient Sensitivity Analysis Method for La

Simulations", IEEE Cloud 2011, Washington, D.C., July 8, 2011.K. Mills, J. Filliben, D.-Y. Cho and E. Schwartz, "Predicting Macroscopic Dynamics in Distributed Systems", American Society of Mechanical Engineers2011 Conference on Pressure Vessels & Piping Baltimore MD July 21 20112011 Conference on Pressure Vessels

& Piping, Baltimore, MD, July 21, 2011.
References

K. Mills, “Measurement Science for Complex Information Systems”, NIST/Page for the Complex Systems Project: http://www.nist.gov/itl/antd/emergent_behavior.cfmK. Mills, J. Filliben, D. Cho, E. Schwartz and D. Genin, "Study of Proposed Congestion Control Mechanisms“, NIST Special Publication 500-282, May 

534 pages. http://www.nist.gov/itl/antd/Congestion_Control_Study.cfm

K. Mills, J. Filliben and C. Dabrowski, "An Efficient Sensitivity Analysis MetLarge Cloud Simulations“, Proceedings of the 4th International Cloud ComConference, IEEE, Washington, D.C., July 5-9, 2011.K. Mills, J. Filliben, D-Y. Cho and E. Schwartz, "Predicting Macroscopic DyLarge Distributed Systems“, Proceedings of ASME 2011 Conference on PVessels & Piping, Baltimore, MD, July 17-22, 2011.K. Mills and J. Filliben, "Comparison of Two Dimension-Reduction MethodNetwork Simulation Models“, Journal of Research of the National Institute Standards and Technology, 116-5, September-October 2011, in press.K. Mills, J. Filliben and C. Dabrowski, "Comparing VM-Placement AlgorithmDemand Clouds“, (submitted to IEEE CloudCom 2011, under review.
Web References

Complex Systems Projecthttp://www.nist.gov/itl/antd/emergent_behavior.cfmNIST SP 500-282 (534 pages) http://www.nist.gov/itl/antd/Congestion_Control_Study.cfmNIST/SEMATECH Engineering Statistics Handbookhttp://www.itl.nist.gov/div898/handbook/Dataplothttp://www.itl.nist.gov/div898/software/dataplot/This Talkhttp://stat.nist.gov/~filliben/fillibenmillsnistsedtalk092211.pdfhttp://www.nist.gov/itl/antd/upload/millsjjfsedtalk092211.pdf