Sensitivity Analysis Methodology 

a Complex System Computational M

James J. Filliben

National Institute of Standards and Technolog

Gaithersburg, MD 20899

39th Symposium on the Interface:

Computing Science and Statistics

11:30-12:00, Saturday, May 26, 2007

Doubletree Hotel, Philadelphia, PA

philadelphiaint
Sensitivity Analysis Methodology for a Complex System Computational Model 

James J. Filliben 

National Institute of Standards and Technology 

100 Bureau Drive, Gaithersburg, MD, USA 20899-8980 

Complex systems are of many types—biological (the human brain), physical (the World

Trade Center collapse), social (the U.S. domestic aircraft transportation network), andinformational (the Internet). Many such complex systems are successfully advanced by

the judicious construction of computational models to serve as predictive surrogates for the system. The use of such models increasingly serves as the backbone forcharacterizing, modeling, predicting and ultimately optimizing the systems themselves. The results of such efforts gain credence only after a proper V&V (verification and validation) of such computational models. This task itself is extremely difficult and canfrequently be achieved only in a matter of degrees. In practice, an essential component inthis V&V is an appropriate sensitivity analysis of the computational model. Gaining anappreciation of the dominant factors (and the ever-present interactions) of acomputational model for a complex system is always an essential component inaccepting/rejecting such a model, and (after acceptance) in gaining deeper insights and

understanding as to what actually drives the complex system itself. This talk describes the methodology (experiment design and statistical analysis) whichwas brought to bear to carry out a sensitivity analysis for computational models for aspecific complex informational system (a network). Such methodology has application toassessing models of other complex system types. 
March 28, 2007

Innovations in Measurement Science

3

Science for Networks = ∑k 

j j 

N E 

1

Measurement Science for Measurement Science for 

Complex Information System

Complex Information System

K. Mills K. Mills892, C. Dabrowski , C. Dabrowski897, J. Filliben , J. Filliben898, D. 

J. Hagedorn J. Hagedorn891, F. Hunt , F. Hunt891, M. Laverne , M. Laverne^, D. 

, D. 

V. Marbukh V. Marbukh892, Edward Schwartz , Edward Schwartz#, Bert 

, Bert 

J. Terrill J. Terrill891 and J. Yuan and J. Yuan*

March 28, 2007 March 28, 2007Innovations in Measurement Science Innovations in Measurement Science

+NRC post NRC post-doc#NSF SURF student NSF SURF student

^Guest Researcher from ISIMA

*Professor at *Professor at Tsinghua University

NIST 5-Year Competence Project
March 28, 2007

Innovations in Measurement Science

4

Science for Networks = ∑k 

j j 

N E 

1

What is the problem? 

No one understands how to measure, predict or cmacroscopic behavior in complex information sys

“[Despite] society’s profound dependence on networks, fundknowledge about them is primitive. [G]lobal communication networks have quite advanced technological implementationtheir behavior under stress still cannot be predicted reliably.

is no science today that offers the fundamental knowledge nto design large complex networks [so] that their behaviors cpredicted prior to building them.”— Network Science 2006, recently released NRC report 

– threatening our nation’s security– costing billions of dollars
March 28, 2007

Innovations in Measurement Science

5

Science for Networks = ∑k 

j j 

N E 

1

What is the problem? 

No one understands how to measure, predict or cmacroscopic behavior in complex information sys

“[Despite] society’s profound dependence on networks, fundknowledge about them is primitive. [G]lobal communication networks have quite advanced technological implementationtheir behavior under stress still cannot be predicted reliably.

is no science today that offers the fundamental knowledge nto design large complex networks [so] that their behaviors cpredicted prior to building them.”— Network Science 2006, recently released NRC report 

– threatening our nation’s security– costing billions of dollars
March 28, 2007

Innovations in Measurement Science

6

Science for Networks = ∑k 

j j 

N E 

1

How is it solved today?

Analyzing

Spatiotemporal

Properties

Predicting Phase

Transitions

Visualizing

Macroscopic

Evolution

Network Science

Little work

Controlling

Global

Behavior

Network 

Test Facilities

Emulab

DETER

Very active

GENI

TeraGrid

National Lamda Rail

Autonomic 

Computing

Network 

Technology

Very active

IPv6 Transition

Service-Oriented

Architectures

Mobile and

Wireless Devices

Peer-to-Peer

Services

Vis

Top

An

Self-

A

Traffi

Esti

Ne

Con

NetwMeas

Som
March 28, 2007

Innovations in Measurement Science

7

Science for Networks = ∑k 

j j 

N E 

1

Technical Approach

• Establish models and analysis methods 

– Computationally tractable– Reveal macroscopic behavior– Establish predictability and causality

• Characterize distributed control techniques

– Economic mechanisms to elicit desired behaviors– Biological mechanisms to organize components

Leverage models and mathematics from the phys

sciences to define a systematic method to measurunderstand, predict and control macroscopic behavior in the Internet and distributed software systems built on the Internet

Inte

Distribu

Scheduler

Task

Control

Negotiation

Control

Grid Processor 

DSI

Service 

Negotiator

Agreement

Execution 

Control

CLIE

Application

Task 1

D

C

Task 2

Grid Processor

DSI

Service 

Negotiator

Agreeme

Task 3

DRMS 

Front-End

DRMS 

Front-End

DSF

spawns

monitors

monitors

spaw

Supervisory Process 

ProviderSite

Scheduler

Task

Control

Negotiation

Control

Grid Processor 

DSI

Service 

Negotiator

Agreement

Execution 

Control

CLIE

Application

Task 1

D

C

Task 2

Grid Processor

DSI

Service 

Negotiator

Agreeme

Task 3

DRMS 

Front-End

DRMS 

Front-End

DSF

spawns

monitors

monitors

spaw

Supervisory Process 

ProviderSite
March 28, 2007

Innovations in Measurement Science

8

Science for Networks = ∑k 

j j 

N E 

1

Current Activities (FY07)

Micro-scale model of compute grid (Dabrowski and Mills )Meso-scale model of compute grid (Laverne)Agent automata model of Internet (Mills, Schwartz and Yuan)

Model DevelopmentData Repository Development

Configuration & event repository for grid models (Laverne and Hagedorn)

Experiment Designs

Grid model visualization (Filliben, Hagedorn, Laverne and Terrill)

Visualization Methods

Grid model sensitivity & key factors (Filliben, Leber and Laverne)

Abstract Models and Analysis Methods 

Phase-transition models for queuing networks (Hunt and Marbukh)Fluid-flow models of the Internet (Genin and Mills)Time series analysis for Internet models (Rust and Mills)

Theory

Internet behavioral predictions (Marbukh)Grid resource allocation models (Marbukh and Mills)
March 28, 2007

Innovations in Measurement Science

9

Science for Networks = ∑k 

j j 

N E 

1

Current Activities (FY07)

Micro-scale model of compute grid (Dabrowski and Mills )Meso-scale model of compute grid (Laverne)Agent automata model of Internet (Mills, Schwartz and Yuan)

Model DevelopmentData Repository Development

Configuration & event repository for grid models (Laverne and Hagedorn)

Experiment Designs

Grid model visualization (Filliben, Hagedorn, Laverne and Terrill)

Visualization Methods

Grid model sensitivity & key factors (Filliben, Leber and Laverne)

Abstract Models and Analysis Methods 

Phase-transition models for queuing networks (Hunt and Marbukh)Fluid-flow models of the Internet (Genin and Mills)Time series analysis for Internet models (Rust and Mills)

Theory

Internet behavioral predictions (Marbukh)Grid resource allocation models (Marbukh and Mills)
March 28, 2007

Innovations in Measurement Science

10

Science for Networks = ∑k 

j j 

N E 

1

What is the global behavior of the Interne

• Modeling Approach – Agent Automaton • Simulated topology• Agent rules

– backbone routers, backbone links, enterprise routers, rece

department routers

• Model dimensions• Sample model measurements• Next steps – and some comparison with measureme• Issues to consider

– How to improve the model?– What analysis methods to use on the measurements?
March 28, 2007

Innovations in Measurement Science

11

Science for Networks = ∑k 

j j 

N E 

1

Agent Automata Model of Internet-like Net

• A network topology of routers, links and flows (source-rece

– Links have distances (and corresponding propagation delay

– Routers have (finite) queues (and thus varying queuing dela– Routers have forwarding capacities (that vary with router typ– Assume fixed-length shortest path among routers (others assum

– Flows have random ON and OFF durations (various distributions – Flows implement TCP congestion-control algorithm (others pos

• Packets (data and acknowledgments) associated with flow

between sources and receivers following routes through rolinks in the topology

• Every model element updated at each time step• Measurements taken every “measurement” interval (e.g., 200 
March 28, 2007

Innovations in Measurement Science

12

Science for Networks = ∑k 

j j 

N E 

1

Starting with Fixed Topology – original Abil

Router-level network with three tiers: backbone (160 ppts), enterprise (20 ppts), d

200 flow sources/department (1 ppts

< 800 flow receivers/department (1 pp

+

Abilene Network
March 28, 2007

Innovations in Measurement Science

13

Science for Networks = ∑k 

j j 

N E 

1

Rules for Backbone Router

Rules for Backbone Link

Rules for Enterprise Router

Rules for Receiver

Rules for Source

Rules for Department Router (outbound p

Rules for Department Router (inbound data 

Rules for Department Router (inbound AC
March 28, 2007

Innovations in Measurement Science

14

Science for Networks = ∑k 

j j 

N E 

1

Rules for Backbone Router

If there are packets in the queue then lookup the next hop router

If the next hop router is in the same domain

lookup the enterprise router of the next hopif the queue in the enterprise router is not full 

then forward the packetelse drop the packet

endif

else

lookup the backbone link on which to send the packetforward the packet on the backbone link and set the time it will be propag

endif

Rules for Backbone Link

If the first packet on the link has been propagated

if the sink backbone router for the link has room in its queue

then forward the packet from the link to the sink routerelse drop the packet

endif

endif
March 28, 2007

Innovations in Measurement Science

15

Science for Networks = ∑k 

j j 

N E 

1

Model Dimensions

• Elements updated per time step: all backbone routers, back

enterprise routers, department routers, source and active re

– 11 + 28 + 40 +110 +22,000 + 22,000 = 45,000

• Measurement interval = 200 time steps and run for 4000 me

intervals = 800,000 time steps (other configurations possible)

• Model size is about 1100 lines of source code

– MATLAB version (pre-project prototype) required 280 CPU hours to c

– SLX version (developed during project) requires < 3 CPU hours to co– Recently required ~16 CPU hours to simulate 7.2 million time steps

with a measurement granularity of 100 times per interval (to throw aw3.6 million time steps)

• Can model size be expanded and still execute reasonably?

– Concatenate 10 networks (i.e., about 500,000 elements)?
March 28, 2007

Innovations in Measurement Science

16

Science for Networks = ∑k 

j j 

N E 

1

Sample measurements (per interval)

• Packets flowing into and out of the network• Count of packets bound (sort of) from domain x to leaf y• Active flows• Router queue size• Network-wide loss rate• Domain with maximum flowing heading into it and the numbe• Packets transiting selected monitored links• Loss Rate = (In-Out)/In

. . .

Measurements drive memory usage ~ 100 to 200 Mbytes (depending on m
March 28, 2007

Innovations in Measurement Science

17

Science for Networks

=

∑k

j

j 

N

E 

1

Some sample time series for aggregate m

2000 3000 

2 .10

4

2.5 .10

4

3 .10

4

Time

Packets Injected

2000 3000 

2.2 .10

4

2.4 .10

4

2.6 .10

4

2.8 .10

4

3 .10

4

Time

Packets Departing

lr :=

2000 3000 4000

0

0.10.2

Time

Lose Rate

LR

2000 3000 4000

1.2 .10

4

1.25 .10

4

1.3 .10

4

Time

Active Flows

200 simulated seconds
March 28, 2007

Innovations in Measurement Science

18

Science for Networks = ∑k 

j j 

N E 

1

Sample time series of Enterprise Router Q Le

0

20406080

100120140160

1500

Time

 Queue Length for a Selected Subnet Router

10 simulated seconds
March 28, 2007

Innovations in Measurement Science

19

Science for Networks = ∑k 

j j 

N E 

1

Sample time series of Department Router Q 

0

20406080

100120140160

2000

Measurement Interval

Leaf Router Queue Length

2 simulated seconds
March 28, 2007

Innovations in Measurement Science

20

Science for Networks = ∑k 

j j 

N E 

1

Questions ...

• What is a reasonable topology?

– Selected tier 3 sites (e.g., Google, Yahoo) will provide higher capacities 

increased load

– Internet is a network of networks (ASes)

• Where should sources be placed relative to receivers?

– Web surfing puts sources and receivers at different places

– Peer-to-peer interaction provides more even distribution of sources and 

– Content-distribution networks might also provide more even distribution

• What are reasonable distributions for the pattern of flows?

– Web surfing suggests concentration of flow to selected receivers

– Special events suggest hot spots– Collaborative interactions suggest peer-to-peer

• What ON/OFF regimes should flows observe and in what mix?

– File transfers might differ from Web accesses and from interactive gam

voice and video

• What special techniques should simulated routers employ?

– Random-early drop might mitigate congestion– Flow reservations and prioritized queues might provide varying quality o
March 28, 2007

Innovations in Measurement Science

21

Science for Networks = ∑k 

j j 

N E 

1

What analysis techniques will provide ins

Wavelets

Relative 

Eigen analysis

Multi-scale Time Series

Power Spectral Density

Congestio
Scientific 

Problem

Data

Expert/

Theory

Stat Problem-Solving Approach/Frame
Scientific 

Problem

Data

Expert/

Theory

DEX (Pre-Data) Stat (Q/V) (Post-Data)

1 2

Stat Problem-Solving Approach/Frame
Scientific 

Problem

Data

Expert/

Theory

DEX (Pre-Data) Stat (Q/V) (Post-Data)

1 2

Stat Problem-Solving Approach/Frame
Scientific 

Problem

Data

Expert/

Theory

DEX (Pre-Data) Stat (Q/V) (Post-Data)

1 2

Stat Problem-Solving Approach/Frame
Scientific 

Problem

Data

Expert/

Theory

DEX (Pre-Data) Stat (Q/V) (Post-Data)

1 2

Question

Stat Problem-Solving Approach/Frame
Scientific 

Problem

Data

Expert/

Theory

DEX (Pre-Data) Stat (Q/V) (Post-Data)

1 2

Question 

Stat Problem-Solving Approach/Frame
Scientific 

Problem

Data

Expert/

Theory

DEX (Pre-Data) Stat (Q/V) (Post-Data)

1 2

1. Reality2. Lab3. Computational

Question 

Stat Problem-Solving Approach/Frame
Scientific 

Problem

Data

Expert/

Theory

DEX (Pre-Data) Stat (Q/V) (Post-Data)

1 2

1. Reality2. Lab3. Computational

1. Estimation2. Testing3. Insight

1. Translation2. Classification3. Construction (k,n)4. Execution

Question 

Stat Problem-Solving Approach/Frame
Scientific 

Problem

Data

Expert/

Theory

DEX (Pre-Data) Stat (Q/V) (Post-Data)

1 2

1. Reality2. Lab3. Computational

1. Estimation2. Testing3. Insight

1. Translation2. Classification3. Construction (k,n)4. Execution

Question 

Stat Problem-Solving Approach/Frame
Scientific 

Problem

Data

Expert/

Theory

DEX (Pre-Data) Stat (Q/V) (Post-Data)

1 2

1. Reality2. Lab3. Computational

1. Estimation2. Testing3. Insight

1. Translation2. Classification3. Construction (k,n)4. Execution

Question 

Stat Problem-Solving Approach/Frame
Scientific 

Problem

Data

Expert/

Theory

DEX (Pre-Data) Stat (Q/V) (Post-Data)

1 2

1. Reality2. Lab3. Computational

1. Estimation2. Testing3. Insight

1. Translation2. Classification3. Construction (k,n)4. Execution

Question

Stat Problem-Solving Approach/Frame
Scientific 

Problem

Data

Expert/

Theory

DEX (Pre-Data) Stat (Q/V) (Post-Data)

1 2

Q. Insight/Understanding:What node andsystem “parameters”(= factors) are most importantwrt affecting the performanceof the Abilene network simulator? (Sensitivity Analysis)

Stat Problem-Solving Approach/Frame
Scientific 

Problem

Data

Expert/

Theory

DEX (Pre-Data) Stat (Q/V) (Post-Data)

1 2

Q. Insight/Understanding:What node andsystem “parameters”(= factors) are most importantwrt affecting the performance(mean active flows per interval) of the Abilenenetwork simulator? (Sens. Analysis)

D

Stat Problem-Solving Approach/Frame
(k = ?, n = ?)
(k = 10, n = ?)

(10 hours of CPU time)

Î n <= 50

(k = 10, n <= 50)
(k = 10, n <= 50) 

Î

210-5 Fractional Factorial Des

(k = 10, n = 32)

1 + 10 + 45
(k = 10, n = 32)

Scientific 

Problem

R

Sc

S

Data

Expert/

Theory

DEX Stat (Q/V)

Abilene Network Model Sensitivity Analysis

X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 Y

Run Prop Rel Router Rel Router Queue Queue Queue Pareto Pareto Exp Start Average Flows

# Delay Speed (1,2) Speed (2,3) Size (1) Size (2) Size (3) Mean Shape Mean Up Mix per Interval

1 1 8 4 160 160 80 100 1.2 2500 2 181742 2 8 4 160 160 160 50 1.5 5000 2 119813 1 4 4 160 160 160 50 1.5 2500 1 137084 2 4 4 160 160 80 100 1.2 5000 1 119635 1 8 2 160 160 160 50 1.2 5000 1 53756 2 8 2 160 160 80 100 1.5 2500 1 179427 1 4 2 160 160 80 100 1.5 5000 2 147758 2 4 2 160 160 160 50 1.2 2500 2 118829 1 8 4 80 160 160 100 1.5 5000 1 16547

10 2 8 4 80 160 80 50 1.2 2500 1 1549911 1 4 4 80 160 80 50 1.2 5000 2 512012 2 4 4 80 160 160 100 1.5 2500 2 1762113 1 8 2 80 160 80 50 1.5 2500 2 1433614 2 8 2 80 160 160 100 1.2 5000 2 1181715 1 4 2 80 160 160 100 1.2 2500 1 1676116 2 4 2 80 160 80 50 1.5 5000 1 877417 1 8 4 160 80 80 50 1.5 5000 1 1215318 2 8 4 160 80 160 100 1.2 2500 1 1823519 1 4 4 160 80 160 100 1.2 5000 2 1099420 2 4 4 160 80 80 50 1.5 2500 2 1428021 1 8 2 160 80 160 100 1.5 2500 2 1791022 2 8 2 160 80 80 50 1.2 5000 2 550723 1 4 2 160 80 80 50 1.2 2500 1 1363724 2 4 2 160 80 160 100 1.5 5000 1 1370925 1 8 4 80 80 160 50 1.2 2500 2 1542426 2 8 4 80 80 80 100 1.5 5000 2 1639127 1 4 4 80 80 80 100 1.5 2500 1 1801228 2 4 4 80 80 160 50 1.2 5000 1 430029 1 8 2 80 80 80 100 1.2 5000 1 1214230 2 8 2 80 80 160 50 1.5 2500 1 1427431 1 4 2 80 80 160 50 1.5 5000 2 597832 2 4 2 80 80 80 100 1.2 2500 2 17298

1 2 3 

5 6 7 

9 1

12
(k = 10, n = 32)

Scientific 

Problem

R

Sc

S

Data

Expert/

Theory

DEX Stat (Q/V)

Abilene Network Model Sensitivity Analysis

X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 Y

Run Prop Rel Router Rel Router Queue Queue Queue Pareto Pareto Exp Start Average Flows

# Delay Speed (1,2) Speed (2,3) Size (1) Size (2) Size (3) Mean Shape Mean Up Mix per Interval

1 1 8 4 160 160 80 100 1.2 2500 2 181742 2 8 4 160 160 160 50 1.5 5000 2 119813 1 4 4 160 160 160 50 1.5 2500 1 137084 2 4 4 160 160 80 100 1.2 5000 1 119635 1 8 2 160 160 160 50 1.2 5000 1 53756 2 8 2 160 160 80 100 1.5 2500 1 179427 1 4 2 160 160 80 100 1.5 5000 2 147758 2 4 2 160 160 160 50 1.2 2500 2 118829 1 8 4 80 160 160 100 1.5 5000 1 16547

10 2 8 4 80 160 80 50 1.2 2500 1 1549911 1 4 4 80 160 80 50 1.2 5000 2 512012 2 4 4 80 160 160 100 1.5 2500 2 1762113 1 8 2 80 160 80 50 1.5 2500 2 1433614 2 8 2 80 160 160 100 1.2 5000 2 1181715 1 4 2 80 160 160 100 1.2 2500 1 1676116 2 4 2 80 160 80 50 1.5 5000 1 877417 1 8 4 160 80 80 50 1.5 5000 1 1215318 2 8 4 160 80 160 100 1.2 2500 1 1823519 1 4 4 160 80 160 100 1.2 5000 2 1099420 2 4 4 160 80 80 50 1.5 2500 2 1428021 1 8 2 160 80 160 100 1.5 2500 2 1791022 2 8 2 160 80 80 50 1.2 5000 2 550723 1 4 2 160 80 80 50 1.2 2500 1 1363724 2 4 2 160 80 160 100 1.5 5000 1 1370925 1 8 4 80 80 160 50 1.2 2500 2 1542426 2 8 4 80 80 80 100 1.5 5000 2 1639127 1 4 4 80 80 80 100 1.5 2500 1 1801228 2 4 4 80 80 160 50 1.2 5000 1 430029 1 8 2 80 80 80 100 1.2 5000 1 1214230 2 8 2 80 80 160 50 1.5 2500 1 1427431 1 4 2 80 80 160 50 1.5 5000 2 597832 2 4 2 80 80 80 100 1.2 2500 2 17298

3
This is data file mills21.dat DEX #1: 2**(10-5) Kevin Mills, 05/22/07Contents: Results of Abilene Network Designed Experiment #1Number of observations = 32 (= 2**(10-5))Number of variables per line image = 16 (= 6 responses + 10 factors)Order of variables on a line image:

1. Response variable 1 = Computer (= Processor) ID2. Response variable 2 = CPU Time (hours)3. Response variable 3 = Average active flows per interval4. Response variable 4 = Average packets in per interval5. Response variable 5 = Average packets out per interval6. Response variable 6 = Average loss rate per interval7. Factor 1 = Propagation Delay (2 levels: (-,+) = (current,2 x current))8. Factor 2 = (L1/L2) Rel. Router Speed (2 levels: (-,+) = (8 ,4))9. Factor 3 = (L2/L3) Rel. Router Speed (2 levels: (-,+) = (8 ,4))

10. Factor 4 = Level 3 Queue Size (2 levels: (-,+) = (160 ,80))11. Factor 5 = Level 2 Queue Size (2 levels: (-,+) = (160 ,80))12. Factor 6 = Level 1 Queue Size (2 levels: (-,+) = (160 ,80))13. Factor 7 = Packet Block Size Pareto Mean (2 levels: (-,+) = (50 ,100))14. Factor 8 = Packet Block Size Pareto Shape(2 levels: (-,+) = (1.5 ,1.2)15. Factor 9 = "Think Time" Exponential Mean (2 levels: (-,+) = (5000,2500)16. Factor 10 = Start-up Mix (2 levels: (-,+) = (100/0/0/0, 0/0/0/100))

To read this data file into dataplot:

skip 25; read mills21.dat y1 to y6 x1 to x10

The Input Data File
Response 1: Average Active Flows per IntResponse 2: Average Packets In per InterResponse 3: Average Packets Out per InteResponse 4: Average Loss per Interval
Y1 Y2 Y3 Y4 Y5 Y6 X1 X2 X3 X4 X5 X6 X7 X8 X9 X10-----------------------------------------------------------------------

9. 2.7 18174. 30708. 27201. 0.114 -1 -1 -1 -1 -1 1 1 1 1 19. 2.1 11981. 25754. 23760. 0.077 1 -1 -1 -1 -1 -1 -1 -1 -1 19. 2.7 13708. 45354. 41588. 0.083 -1 1 -1 -1 -1 -1 -1 -1 1 -19. 2.6 11963. 40133. 37346. 0.069 1 1 -1 -1 -1 1 1 1 -1 -19. 1.8 5375. 35369. 34098. 0.036 -1 -1 1 -1 -1 -1 -1 1 -1 -19. 3.3 17942. 41631. 37839. 0.091 1 -1 1 -1 -1 1 1 -1 1 -19. 2.6 14775. 38251. 35576. 0.070 -1 1 1 -1 -1 1 1 -1 -1 19. 2.8 11882. 43527. 41083. 0.056 1 1 1 -1 -1 -1 -1 1 1 1

10. 2.5 16547. 29859. 26022. 0.128 -1 -1 -1 1 -1 -1 1 -1 -1 -110. 2.6 15499. 28445. 25530. 0.102 1 -1 -1 1 -1 1 -1 1 1 -110. 2.1 5120. 35889. 34010. 0.052 -1 1 -1 1 -1 1 -1 1 -1 110. 3.4 17621. 44769. 41228. 0.079 1 1 -1 1 -1 -1 1 -1 1 110. 2.7 14336. 42241. 38093. 0.098 -1 -1 1 1 -1 1 -1 -1 1 110. 2.9 11817. 39438. 36931. 0.063 1 -1 1 1 -1 -1 1 1 -1 110. 3.5 16761. 45876. 41996. 0.084 -1 1 1 1 -1 -1 1 1 1 -110. 2.2 8774. 33817. 32476. 0.039 1 1 1 1 -1 1 -1 -1 -1 -1

7. 8.6 12153. 26565. 24133. 0.091 -1 -1 -1 -1 1 1 -1 -1 -1 -17. 10.0 18235. 29763. 26782. 0.100 1 -1 -1 -1 1 -1 1 1 1 -17. 9.7 10994. 44638. 41353. 0.074 -1 1 -1 -1 1 -1 1 1 -1 17. 9.1 14280. 40275. 37192. 0.076 1 1 -1 -1 1 1 -1 -1 1 17. 10.3 17910. 44611. 39225. 0.121 -1 -1 1 -1 1 -1 1 -1 1 17. 7.8 5507. 33339. 32084. 0.038 1 -1 1 -1 1 1 -1 1 -1 17. 10.0 13637. 38776. 36270. 0.064 -1 1 1 -1 1 1 -1 1 1 -17. 10.2 13709. 41547. 38815. 0.066 1 1 1 -1 1 -1 1 -1 -1 -18. 9.4 15424. 29771. 26212. 0.119 -1 -1 -1 1 1 -1 -1 1 1 18. 9.6 16391. 28419. 25299. 0.110 1 -1 -1 1 1 1 1 -1 -1 18. 10.4 18012. 44761. 39966. 0.107 -1 1 -1 1 1 1 1 -1 1 -18. 7.6 4300. 36237. 35304. 0.026 1 1 -1 1 1 -1 -1 1 -1 -18. 9.7 12142. 41541. 37585. 0.095 -1 -1 1 1 1 1 1 1 -1 -18. 10.0 14274. 40689. 37398. 0.081 1 -1 1 1 1 -1 -1 -1 1 -18. 9.7 5978. 40673. 38877. 0.044 -1 1 1 1 1 -1 -1 -1 -1 18. 10.2 17298. 38919. 36318. 0.067 1 1 1 1 1 1 1 1 1 1

Experiment Design(210-5) & Data
Response 1:

Average Active Flow per Interval
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 

Run Prop Rel Router Rel Router Queue Queue Queue Pareto Pareto Exp Start Avera

# Delay Speed (1,2) Speed (2,3) Size (1) Size (2) Size (3) Mean Shape Mean Up Mix per 

1 1 8 4 160 160 80 100 1.2 2500 2 12 2 8 4 160 160 160 50 1.5 5000 2 13 1 4 4 160 160 160 50 1.5 2500 1 14 2 4 4 160 160 80 100 1.2 5000 1 15 1 8 2 160 160 160 50 1.2 5000 1 6 2 8 2 160 160 80 100 1.5 2500 1 17 1 4 2 160 160 80 100 1.5 5000 2 18 2 4 2 160 160 160 50 1.2 2500 2 19 1 8 4 80 160 160 100 1.5 5000 1 1

10 2 8 4 80 160 80 50 1.2 2500 1 111 1 4 4 80 160 80 50 1.2 5000 2 12 2 4 4 80 160 160 100 1.5 2500 2 113 1 8 2 80 160 80 50 1.5 2500 2 114 2 8 2 80 160 160 100 1.2 5000 2 115 1 4 2 80 160 160 100 1.2 2500 1 116 2 4 2 80 160 80 50 1.5 5000 1 17 1 8 4 160 80 80 50 1.5 5000 1 118 2 8 4 160 80 160 100 1.2 2500 1 119 1 4 4 160 80 160 100 1.2 5000 2 120 2 4 4 160 80 80 50 1.5 2500 2 121 1 8 2 160 80 160 100 1.5 2500 2 122 2 8 2 160 80 80 50 1.2 5000 2 23 1 4 2 160 80 80 50 1.2 2500 1 124 2 4 2 160 80 160 100 1.5 5000 1 125 1 8 4 80 80 160 50 1.2 2500 2 126 2 8 4 80 80 80 100 1.5 5000 2 127 1 4 4 80 80 80 100 1.5 2500 1 128 2 4 4 80 80 160 50 1.2 5000 1 29 1 8 2 80 80 80 100 1.2 5000 1 130 2 8 2 80 80 160 50 1.5 2500 1 131 1 4 2 80 80 160 50 1.5 5000 2 32 2 4 2 80 80 80 100 1.2 2500 2 1

Experiment Design(210-5) & Data
1 2 3 

5 6 7 

9 10

10-Step Data Analysis (Dataplot Macro dexplot.
1 2 3 

5 6 7 

9 10

10-Step Data Analysis (Dataplot Macro dexplot.

NIST/SEMATECH e-Handbook of Statistical Met

http://www.itl.nist.gov/div898/handbook/

5. Improve

5. Advanced Topics

9. An EDA Approach to Experiment D

Dataplot

http://www.itl.nist.gov/div898/software/datap

















Response 2:

Average Packets In per Interval











Response 3:

Average Packets Out per Interval











Response 4:

Average Loss per Interval













Conclusion

1. Network Goal: insight & understanding2. Structured Stat approach provides a path for networ3. DEX is critically important component 4. To achieve insight: Step 1= Sensitivity Analysis5. 2k-p fractional factorial designs potent & efficient

6. Insight: Flows from accompanying 10-step graphica