Predicting Global Failure Regimes in 

Complex Information Networks

July 12, 2012 SFI Workshop on Measurement of Complex Networks

Chris Dabrowski, Jim Filliben and Kevin Mills 

A C

D E G

F

J K

M

P

B L

N

O

A1 A1b

A1c

A1a A2a A2b A2c

C1 C1b

C1c

C1a

C2 C2b C2c C2a H2b

H2c

H2a H1

H1c H1b H1a

E1 E1b E1c E1a

E2b E2a E2c

H1d

H1e

H1f

H2f H2e H2d

G1b

G1c

G1a

G1e G1d G2a G2b G2c

G2e G1f G2d

G2f P1 P2 P1a

P1b

P1c

P1d P2aP2b P2c P2d

P2e

P2f

P2g

K1 K2 K0a

K1a

K1b K1c K1d K2a K2b

K2c

K2d

L2

L1

L1a L1b L1c

L1d

L2a

L2b

L2c L2d

L0a

L0b J1

J2

J1a J1b J1c J1d

J1e

J1f J2a J2b

J2c

J2d J2e J2f N1a N1b

N1c

N1d

N1e N1f N2 N2a

N2b N2c N2d N2e

N2f

M2b

M2c

M2d M2e M2f M2 M2g M1a M2a M M1 1b

M1c

M1dM1e M1f M1g O0a

O1a

O1b

O1c

O1c

O1

O2a

O2

O2b O2c O2d

O2e

O2f O2g

I2 I2a I1 I I0a

I1a

I1b I1c I1d

I2g I2f

I2e

I2d I2b I2c

F2

F2a F1 F1a

F1b F1c F1d

F2g F2f

F2d

F2c F2b F0a F2e

D2 D2a

D1a D1 D1b

D1c D1d

D2g

D2f

D2d D2b D2c D2e

D0a

B0a B B1 1a

B1b

B1c B1d

B2 B2a

B2b

B2c B2d B2e

B2f

B2g H

G1 G2 E2

H2

A2

N1

ISP Topology

A

B

D

C

F

G

H

I K

J

A1 A2 A1a A1b

A1c

A1d

A1e E

A2b

A2g

Access Router n x Normal Speed

A1f

A2a

Access Router m x Normal Speed

B0a

B1 B1b

B2g

B2f

B1a

B2

B2a B2b

B2c

B2d

B2e D1 D1b D2g

F1d D1a D2 D2a

D2b D2c D2d

D2e

C1 C1a

C1b C1c C1d C1e

C2

C1f

C2f C2e C2d C2c C2b

C2a E1

E1b

C2g

E2f

E1a

E E2 2a E2b

E2c

E2d E2e G1b

G1c G1d

G1e

G2e G2c

G2b

G2a

CG1 G1a

G1f

G2

G2d G2f

F0a F1 F1b

F1a

F1c F2

F2a

F2b F2c F2d

F2e

H2 H2a

H2b

H2c H2d H2e H1 H1b H2g H1a

F2f

J1 J1e J1c J1d J1b J1a J1f

J2

J2a

J2f

J2b

J2c

J2d

J2e I0a

I1c

I1b

I1a

I1 I1d I2

I2c I2a I2b

I2d K1

K1b

J2g

K1a

I2f K2

K2b G2g

H2f

K2a I2e

D2f

E0a

K0a

J1g

I1e I1f

G1g

I1g

I2g

F2g

C1g E2g

F1g F1e F1f

A2d A2c

B1c

B1d

D1c D1d H1c

E1c E1d

H1d

K1d K1c

K2d K2c

C0a

Access Router Normal Speed 102 to 103 Sources (not shown) and Receivers (not shown) under each Access Router

A

B

D

C

F

G

H

I K

J

A1 A2 A1a A1b

A1c

A1d

A1e E

A2b

A2g

Access Router n x Normal Speed

A1f

A2a

Access Router m x Normal Speed

B0a

B1 B1b

B2g

B2f

B1a

B2

B2a B2b

B2c

B2d

B2e D1 D1b D2g

F1d D1a D2 D2a

D2b D2c D2d

D2e

C1 C1a

C1b C1c C1d C1e

C2

C1f

C2f C2e C2d C2c C2b

C2a E1

E1b

C2g

E2f

E1a

E E2 2a E2b

E2c

E2d E2e G1b

G1c G1d

G1e

G2e G2c

G2b

G2a

CG1 G1a

G1f

G2

G2d G2f

F0a F1 F1b

F1a

F1c F2

F2a

F2b F2c F2d

F2e

H2 H2a

H2b

H2c H2d H2e H1 H1b H2g H1a

F2f

J1 J1e J1c J1d J1b J1a J1f

J2

J2a

J2f

J2b

J2c

J2d

J2e I0a

I1c

I1b

I1a

I1 I1d I2

I2c I2a I2b

I2d K1

K1b

J2g

K1a

I2f K2

K2b G2g

H2f

K2a I2e

D2f

E0a

K0a

J1g

I1e I1f

G1g

I1g

I2g

F2g

C1g E2g

F1g F1e F1f

A2d A2c

B1c

B1d

D1c D1d H1c

E1c E1d

H1d

K1d K1c

K2d K2c

C0a

Access Router Normal Speed 102 to 103 Sources (not shown) and Receivers (not shown) under eI2 Topology ach Access Router
 Overview of our Past & Ongoing Research – with application to 

complex information networks, e.g., Internet, Clouds, Grids

 What is the problem and why is it hard?

 Four Approaches we are investigating:

1. Sensitivity Analysis + Correlation Analysis & Clustering

2. Combine Markov Models, Graph Analysis & Perturbation Analysis

3. Anti-Optimization + Genetic Algorithm

4. Measuring Key System Properties such as Critical Slowing Down

 Example of Sensitivity Analysis + Correlation Analysis & 

Clustering applied to a TCP/IP Network Model – closely related to 

the theme of this topical event: Measurement of Complex Networks

2

 Today’s Topics

July 12, 2012 SFI Workshop
Our Past Research: How can we understand the influence of distributed 

control algorithms on global system behavior and user experience?

3

 Mills, Filliben, Cho, Schwartz and Genin, Study of Proposed 

Internet Congestion Control Mechanisms, NIST SP 500-282 (2010).

 Mills and Filliben, "Comparison of Two Dimension-Reduction 

Methods for Network Simulation Models", Journal of NIST 

Research 116-5, 771-783 (2011).

 Mills, Schwartz and Yuan, "How to Model a TCP/IP Network using 

only 20 Parameters", Proceedings of the Winter Simulation 

Conference (2010). 

 Mills, Filliben, Cho and Schwartz, "Predicting Macroscopic 

Dynamics in Large Distributed Systems", Proceedings of ASME 

(2011).

 Mills, Filliben and Dabrowski, "An Efficient Sensitivity Analysis 

Method for Large Cloud Simulations", Proceedings of the 4th

International Cloud Computing Conference, IEEE (2011).

 Mills, Filliben and Dabrowski, "Comparing VM-Placement 

Algorithms for On-Demand Clouds", Proceedings of IEEE CloudCom, 

91-98 (2011).

For more see: http://www.nist.gov/itl/antd/emergent_behavior.cfm

 Past Research

http://www.nist.gov/itl/antd/Congestion_Control_Study.cfm

July 12, 2012 SFI Workshop
 Our Ongoing & Planned Research: How can we help to 

increase the reliability of complex information systems?

 Research Goals: (1) develop design-time methods that system engineers 

can use to detect existence and causes of costly failure regimes prior to 

system deployment and (2) develop run-time methods that system 

managers can use to detect onset of costly failure regimes in deployed 

systems, prior to collapse.

 Ongoing: investigating

a. Sensitivity Analysis + Correlation Analysis & Clustering

b. Markov Chain Modeling + Cut-Set Analysis + Perturbation 

Analysis (MCM+CSA+PA) (e.g., Dabrowski, Hunt and Morrison, 

“Improving the Efficiency of Markov Chain Analysis of Complex

Distributed Systems”, NIST IR 7744, 2010).

c. Anti-Optimization + Genetic Algorithm (AO+GA) 

 Planned: investigate run-time methods based on approaches that may 

provide early warning signals for critical transitions in large systems (e.g., 

Scheffer et al., “Early-warning signals for critical transitions”, NATURE, 461, 53-59, 2009).

 Ongoing Research

4

http://www.nist.gov/itl/antd/upload/NISTIR7744.pdf

July 12, 2012 SFI Workshop
 What is the Problem? 

 Problem: Given a complex information network (represented using a 

simulation model), how can one identify conditions that could cause 

global system behavior to degenerate, leading to costly system outages?

5

 Why is it Hard? – Reason 1

Determining causality is difficult – in a complex system, global behavior is not easily

predictable, even if behavior of the components is understood completely 

Koala Cloud

Simulator

July 12, 2012 SFI Workshop

See: K. Mills and C. Dabrowski, "Investigating Global 

Behavior in Computing Grids", Self-Organizing Systems, 

Lecture Notes in Computer Science, Volume 4124 ISBN 978-

3-540-37658-3, pp. 120-136. 
6

 Why is it Hard? – Reason 2

 y1, ..., ym = f( x1|[1,...,k], ..., xn|[1,...,k] )

Model Response Space Model Parameter Space

For example, the NIST Koala simulator of IaaS Clouds has about n = 125 

parameters with average k = 6.6 values each, which leads to a model 

parameter space of ~10100 (note that the visible universe has ~1080 atoms) and the 

Koala response space ranges from m = 8 to m = 200, depending on the 

specific responses chosen for analysis (typically m 45). ͌

Size of the search space!!

July 12, 2012 SFI Workshop
7

 Sensitivity Analysis: Determine which parameters most significantly influence 

model behavior. Reduces parameter search space and identifies conditions 

under which alternate control algorithms should be compared. 

 Correlation Analysis & Clustering: Determine response dimension of a model.

See: Mills, Filliben and Dabrowski, "An Efficient Sensitivity Analysis Method for Large Cloud Simulations", Proceedings of the 4th

International Cloud Computing Conference, IEEE (2011) and Mills and Filliben, "Comparison of Two Dimension-Reduction 

Methods for Network Simulation Models", Journal of NIST Research 116-5, 771-783 (2011).

Use 2-level, orthogonal fractional factorial (OFF) 

experiment design and main effects & interaction 

analyses to identify significant model parameters

Use correlation analysis and clustering to identify

unique behavior dimensions of your model

 Sensitivity Analysis + CAC

July 12, 2012 SFI Workshop
Innovations in Measurement Science 

Example: Markov simulation and 

perturbation of a minimal s-t cut set 

of a Markov chain graph:

Using simulated failure scenarios in a 

Markov chain model to predict failures in a Cloud

• Corresponds to software failure 

scenario involving multiple 

faults/attacks.

• Simulation identifies threshold 

beyond which increased failure 

incidence causes drastic 

performance collapse

 Verified in target system being 

modeled (i.e., Koala, a large-scale 

simulation of a Cloud)

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0 Decrease in Probabilities of Transition Proportion of Requests Granted

Total Grants (Large Scale 

Simulation)

Increase in Probability of Transition from Allocating_Partial

state (11) to Transferring_Failure_Estimate state (10). 

Increase in Probability of Transition from Allocating_ Maximum state (9) to Allocating_Partial state (11). 

Decrease in probability of 

transition from Allocating_Maximum

state (9) to Recording_Allocation (12) state. 

Decrease in Probability of Transition 

from Allocating_Partial state (11) to 

Recording_Allocation state (12). 

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1 

Proportion of Requests Granted (both Full and Partial)

Increase in Probability of Transition from Selecting 

Next Cluster state (14) to Selection_Failing state (15)

(a) Total Grants (Markov Simulation)

(b) Total Grants (Large Scale Simulation)

Decrease in probability of transition from

Selecting_Next_Clusterstate (14) to Transferring_

Implementation_Request (F) state (17)

Decrease in probability of transition from Selecting_ 

Next_Cluster state (14) to Transferring_

Implementation_Request (P) state (18)

8

 Cut-Set + Perturbation Analysis 

July 12, 2012 SFI Workshop
9

Model Parameter 

Specifications Parallel Execution of

Model Simulators

Population of Model

Parameterizations

Selection based on

Anti-Fitness

Recombination

& Mutation

List of parameters 

and for each

parameter a MIN, 

MAX and

precision.

Anti-Fitness Reports

GENETIC ALGORITHM

Principal Components Analysis,

Clustering, ...

MULTIDIMENSIONAL ANALYSIS TECHNIQUES

Growing Collection of Tuples: {Generation, Individual, Fitness, Parameter 1 value,....Parameter N value} {Generation, Individual, Fitness, Parameter 1 value,....Parameter N value} {Generation, Individual, Fitness, Parameter 1 value,....Parameter N value} {Generation, Individual, Fitness, Parameter 1 value,....Parameter N value} {Generation, Individual, Fitness, Parameter 1 value,....Parameter N value} {Generation, Individual, Fitness, Parameter 1 value,....Parameter N value} {Generation, Individual, Fitness, Parameter 1 value,....Parameter N value} {Generation, Individual, Fitness, Parameter 1 value,....Parameter N value} {Generation, Individual, Fitness, Parameter 1 value,....Parameter N value} {Generation, Individual, Fitness, Parameter 1 value,....Parameter N value} ... {Generation, Individual, Fitness, Parameter 1 value,....Parameter N value}

MODEL SIMULATORS

 Anti-Opt. + Genetic Algorithm

July 12, 2012 SFI Workshop
10

 Critical Slowing Down

A simple univariate example predicting power 

grid blackout in a human engineered system*

*From P. Hines, E. Cotilla-Sanchez, and S. Blumsack. Topological Models and Critical Slowing Down: Two Approaches to Power System Risk Analysis. 

Proceedings of the 44th Hawaii Conference on System Sciences. IEEE Computer Society, Washington, DC, USA, pp. 1-10.

Time before critical transition (minutes)

-8 -7 -6 -5 -4 -3 -2 -1 0

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0

-0.1

July 12, 2012 SFI Workshop
11

 SA + CAC Example

July 12, 2012 SFI Workshop

Sensitivity Analysis + Correlation Analysis & Clustering 

applied to a TCP/IP Network Model 

(using an 11-parameter subset of a 20-parameter model*)

*For a discussion of the full 20-parameter TCP/IP model see: Mills, Schwartz and Yuan, "How to Model a TCP/IP Network 

 using only 20 Parameters", Proceedings of the Winter Simulation Conference (2010). 

 y1, ..., ym = f( x1|[1,...,k], ..., xn|[1,...,k] )

Model Response Space Model Parameter Space

In the example that follows, m = 22, n = 11 and k = 2 

The approach is general: as we have demonstrated on a TCP/IP model

with m = 45, n = 20 and k = 2 and on a Cloud Computing model with m = 38,

n = 11 and k = 2 and with m = 45, n = 20 and k = 2

Questions: (1) What responses characterize system behavior? 

 (2) What factors drive system behavior?
 

12 July 12, 2012 SFI Workshop

A

B

D

C

F

G

H

I K

J

A1 A2 A1a A1b

A1c

A1d

A1e E

A2b

A2g

Access Router n x Normal Speed

A1f

A2a

Access Router m x Normal Speed

B0a

B1 B1b

B2g

B2f

B1a

B2

B2a B2b

B2c

B2d

B2e D1 D1b D2g

F1d D1a D2 D2a

D2b D2c D2d

D2e

C1 C1a

C1b C1c C1d C1e

C2

C1f

C2f C2e C2d C2c C2b

C2a E1

E1b

C2g

E2f

E1a

E E2 2a E2b

E2c

E2d E2e G1b

G1c G1d

G1e

G2e G2c

G2b

G2a

CG1 G1a

G1f

G2

G2d G2f

F0a F1 F1b

F1a

F1c F2

F2a

F2b F2c F2d

F2e

H2 H2a

H2b

H2c H2d H2e H1 H1b H2g H1a

F2f

J1 J1e J1c J1d J1b J1a J1f

J2

J2a

J2f

J2b

J2c

J2d

J2e I0a

I1c

I1b

I1a

I1 I1d I2

I2c I2a I2b

I2d K1

K1b

J2g

K1a

I2f K2

K2b G2g

H2f

K2a I2e

D2f

E0a

K0a

J1g

I1e I1f

G1g

I1g

I2g

F2g

C1g E2g

F1g F1e F1f

A2d A2c

B1c

B1d

D1c D1d H1c

E1c E1d

H1d

K1d K1c

K2d K2c

C0a

Access Router Normal Speed 102 to 103 Sources (not shown) and Receivers (not shown) under each Access Router

A

B

D

C

F

G

H

I K

J

A1 A2 A1a A1b

A1c

A1d

A1e E

A2b

A2g

Access Router n x Normal Speed

A1f

A2a

Access Router m x Normal Speed

B0a

B1 B1b

B2g

B2f

B1a

B2

B2a B2b

B2c

B2d

B2e D1 D1b D2g

F1d D1a D2 D2a

D2b D2c D2d

D2e

C1 C1a

C1b C1c C1d C1e

C2

C1f

C2f C2e C2d C2c C2b

C2a E1

E1b

C2g

E2f

E1a

E E2 2a E2b

E2c

E2d E2e G1b

G1c G1d

G1e

G2e G2c

G2b

G2a

CG1 G1a

G1f

G2

G2d G2f

F0a F1 F1b

F1a

F1c F2

F2a

F2b F2c F2d

F2e

H2 H2a

H2b

H2c H2d H2e H1 H1b H2g H1a

F2f

J1 J1e J1c J1d J1b J1a J1f

J2

J2a

J2f

J2b

J2c

J2d

J2e I0a

I1c

I1b

I1a

I1 I1d I2

I2c I2a I2b

I2d K1

K1b

J2g

K1a

I2f K2

K2b G2g

H2f

K2a I2e

D2f

E0a

K0a

J1g

I1e I1f

G1g

I1g

I2g

F2g

C1g E2g

F1g F1e F1f

A2d A2c

B1c

B1d

D1c D1d H1c

E1c E1d

H1d

K1d K1c

K2d K2c

C0a

Access Router Normal Speed 102 to 103 Sources (not shown) and Receivers (not shown) under each Access Router Backbone of Topology based on Internet 2

(including propagation delays & routing paths)

Router Type

Base Speed 

p/ms

Backbone 400

POP 100

Typical Access 10

Fast Access 20

Directly Connected Access 100

 Simulated Topology
 11-Parameter Subset

13 July 12, 2012 SFI Workshop

Factor Name Plus (+1) Setting Minus (-1) Setting

Network

Factors

x1 Propagation Delay Multiplier 2 1

 x2 [1] Network Speed Multiplier 1 2

x3 Buffer Sizing Algorithm RTTxC RTTxC/SQRT(n)

User

Factors

x4 Average File Size 100 packets 50 packets

x5 Average Think Time 5000 ms 2000 ms

 x6 [1] Probability User Downloads 10x File 0.01 0.02

Source &

Receiver

Factors

 x7 [1] Probability of a Fast Host Connection 0.2 0.4

x8 Multiplier for Number of Sources & 

Receivers per Access Router 3 2

x9 Distribution Pattern of Sources P2P WEB

x10 Distribution Pattern of Receivers P2P WEB

Protocol

Factors x11 Initial TCP Slow-Start Threshold 1.07x109 packets 43 packets

[1] Unfortunately, we coded these settings backwards from the usual convention of higher value for the Plus setting,

 so care must be taken when interpreting the results for these factors – mainly the network speed factor. Sorry.
 22-Response Subset

14 July 12, 2012 SFI Workshop

Response Name Definition

y1 # Sending Flows Active Flows – flows attempting to transfer data

y2 % Sources Sending Proportion of potential flows that were active: Active Flows/All Sources

y3 # Packets Entering Data packets entering the network per measurement interval

y4 # Packets Exiting Data packets leaving the network per measurement interval

y5 Loss Rate Loss Rate: y4/(y3+y4)

y6 # Flow Completions Flows Completed per measurement interval

y7 Flow Completion Rate Flow-Completion Rate: y6/(y6+y1)

y8 # Connection Failures Connection Failures per measurement interval

y9 Connection Failure Rate Connection-Failure Rate: y8/(y8+y1)

y10 Retransmission Rate Retransmission Rate

y11 Average Congestion Window Congestion Window per Flow

y12 # Window Increases Window Increases per Flow per measurement interval

y13 # NAKs Negative Acknowledgments per Flow per measurement interval

y14 # Timeouts Timeouts per Flow per measurement interval

y15 Average Round-Trip Time Smoothed Round-Trip Time

y16 Queuing Delay Relative queuing delay: y15/(x1x41)

Response Definition

y17 Average Throughput for Active DD Flows

y18 Average Throughput for Active DF Flows

y19 Average Throughput for Active DN Flows

y20 Average Throughput for Active FF Flows

y21 Average Throughput for Active FN Flows

y22 Average Throughput for Active NN Flows

16 Responses Characterizing Macroscopic Network Behavior

+ 6 Responses Characterizing Instantaneous Throughput for Active Flows by Class 

User Experience

Global Behavior
 OFF Experiment Design

15 July 12, 2012 SFI Workshop

Template for a 2-Level 211-5 Orthogonal Fractional Factorial 

(OFF) experiment design specifying the combination of

parameter level settings for 64 simulation runs

(p = 20, n = 256)

FAT = 1.4142...

128

0.125 2

1

1

1

( )

n n

SD Bi



X 4

X 5

+

X1

X2

X3

+

+

_

_

_

_

+

+ X 4

X 5

+

X1

X2

X3

+

+

_

_

_

_

+

+

X 4

X 5

+

X1

X2

X3

+

+

_

_

_

_

+

+ X 4

X 5

+

X1

X2

X3

+

+

_

_

_

_

+

+

X 4

X 5

+

X1

X2

X3

+

+

_

_

_

_

+

+ X 4

X 5

+

X1

X2

X3

+

+

_

_

_

_

+

+

X 4

X 5

+

X1

X2

X3

+

+

_

_

_

_

_

+

+ X 4

X 5

+

X1

X2

X3

+

+

_

_

_

_

_

+

+

OFF Design 1-FAT Design

X5

X4

X6

X7

OFF Design Benefit #2:

Minimizes Variation in 

Effect Estimates

OFF Benefit #1:

Superior Coverage 

& Robustness as

compared with

1-Factor-at-a-Time 

Designs

- +

32 32

Xi

Balance

- +

-

+

8

8 8

Xi

Xj

8

Orthogonality
 Pairwise Corr. Matrix 

16 July 12, 2012 SFI Workshop

Red 80 > |r|x100 < 100 Blue 30 > |r|x100 < 80 Green |r|x100 < 30 

Each

response

observed

under the same

64 combinations

of parameter

settings
 Histogram of |r|

17 July 12, 2012 SFI Workshop

Select a threshold for |r| such that correlations above that threshold will be further considered
 Response Index-Index Plot

18 July 12, 2012 SFI Workshop

2 responses uncorrelated

(1) throughput on DD flows

(2) flow completion rate

25 correlation

pairs reflecting 

congestion

14 correlation

pairs reflecting

packet losses

3 pair-wise correlations:

(1) throughput on flows constrained by F-class routers

(2) network delay

(3) packets entering and leaving the network

where |ri,j| > 0.65 are clustered into mutual correlations
 7-D Model Behavior

19 July 12, 2012 SFI Workshop

Representative

Response Dimension and a Characterizing Measurement

y4 D1 - network throughput in packets/sec

measured by average number of packets output per measurement interval

y6 D2 - network throughput in flows/sec

measured by average number of flows completed per measurement interval

y10 D3 - packet loss

measured by average retransmission rate

y15 D4 - network delay

measured by average smoothed round-trip time

y17 D5 - throughput in packets/sec for the most advantaged users

measured by average instantaneous throughput for DD flows

y20 D6 - throughput in packets/sec for 2

nd most advantaged users

measured by average instantaneous throughput for FF flows

y22 D7 - network congestion

measured by average instantaneous throughput for NN flows

 Question 1: What responses characterize system behavior?

 Answer: As shown below, measuring only 7 of the 22

 responses suffices to characterize behavior
 Main Effects Analyses

20

For each response, compare mean at 32 Plus settings with mean at 32 Minus settings

and conduct t-test to determine statistical significance (response here is TP on NN flows)

Faster 

Network

Longer

Think Time

Fewer

Sources

Web-oriented

Traffic Flow

Patterns

July 12, 2012 SFI Workshop

Four factors

lead to

higher

throughput

for typical users;

inverse settings

lead to lower

throughput

The other seven

factors have

little effect

on this response
 Summary of Main Effects

21 July 12, 2012 SFI Workshop

x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 x11

y4 Packet TP -** +** -** +* +*

y6 Flow TP -** -** -** +* +*

y10 Packet Loss +** -** +* -* +* +*

y15 Delay +** +* +** +* -* +* +*

y17 DD TP -** +**

y20 FF TP -** -** +* +* -* +**

y22 NN TP -* -** +* -* -**

Y Net Effect 50 79 29 64 57 0 0 43 50 0 0

Y = 100 (|{y | p < 0.01}| + 1⁄2 |{y | p < 0.05}|) / |{y}| 

• * p < 0.05 and ** p < 0.01 

• - means minus value caused response increase

• + means plus value cause response increase

• please remember that network speed x2 was miscoded, so – means higher speed and + lower

This chart reveals much

about model behavior
 6 Factors Drive Behavior

22 July 12, 2012 SFI Workshop

Relative

Influence Factors

1 Network Speed

2 File Size

3

User Duty Cycle

Propagation Delay

Distribution of Sources

4 Number of Sources

5 Buffer Sizing

6

Proportion of Hosts with Fast Attachments

Probability of 10x File Sizes

Receiver Distribution

Initial Slow-Start Threshold

Question 2: What factors drive system behavior?

Answer: Network speed mainly followed by file size, then by user duty cycle, 

 propagation delay and source distribution, and finally by number of sources.

The information generated

here can be used in two ways:

1. Compare model behavior to 

experiences of operational 

networks, as a validation 

step.

2. Select parameter 

combinations to explore 

when asking what if 

questions, such as what if TCP 

were replaced by any of 7 

competing congestion control 

algorithms?

For more information see: Mills, Filliben, Cho and Schwartz, "Predicting Macroscopic Dynamics in Large Distributed 

Systems", Proceedings of ASME (2011)
 Reviewed our Past & Ongoing Research – with application to complex 

information networks, e.g., Internet, Clouds, Grids

 Defined the problem underlying our ongoing research and 

identified two reasons why the problem is difficult

 Described 4 approaches to address the problem:

1. Sensitivity Analysis + Correlation Analysis & Clustering

2. Combine Markov Models, Graph Analysis & Perturbation Analysis

3. Anti-Optimization + Genetic Algorithm

4. Measuring Key System Properties such as Critical Slowing Down

 Discussed an example of Sensitivity Analysis + Correlation 

Analysis & Clustering applied to a TCP/IP network model

23

 Brief Recap

July 12, 2012 SFI Workshop
Questions?

24

For more information see: http://www.nist.gov/itl/antd/emergent_behavior.cfm

and/or http://www.nist.gov/itl/cloud/index.cfm

Contact information about studying Complex Information Systems:

{cdabrowski, jfilliben, kmills@nist.gov} 

Contact information about Information Visualization:

sressler@nist.gov

Suggestions?

Ideas?

July 12, 2012 SFI Workshop