An Automated Tool for Developing Experimental Designs: 

The Computer-Aided Design Reference for Experiments 

(CADRE) 

by Robert C. Williges and Sam E. Middlebrooks 

ARL-TN-0341 January 2009 

Approved for public release; distribution unlimited.
NOTICES 

Disclaimers 

The findings in this report are not to be construed as an official Department of the Army position 

unless so designated by other authorized documents. 

Citation of manufacturer’s or trade names does not constitute an official endorsement or 

approval of the use thereof. 

Destroy this report when it is no longer needed. Do not return it to the originator. 
Army Research Laboratory 

Aberdeen Proving Ground, MD 21005-5069 

ARL-TN-0341 January 2009 

An Automated Tool for Developing Experiments Designs: 

The Computer-Aided Design Reference for Experiments 

(CADRE) 

Robert C. Williges 

Virginia Polytechnic Institute and State University

and 

Sam E. Middlebrooks 

Human Research and Engineering Directorate, ARL 

Approved for public release; distribution unlimited.
ii

REPORT DOCUMENTATION PAGE Form Approved 

OMB No. 0704-0188

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering 

and maintaining the data needed, and completing and reviewing the collection information. Send comments regarding this burden estimate or any other aspect of this collection of information, 

including suggestions for reducing the burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson 

Davis Highway, Suite 1204, Arlington, VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to 

comply with a collection of information if it does not display a currently valid OMB control number. 

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS. 

1. REPORT DATE (DD-MM-YYYY)

January 2009 

2. REPORT TYPE 

Final 

3. DATES COVERED (From - To) 

5a. CONTRACT NUMBER 

5b. GRANT NUMBER 

4. TITLE AND SUBTITLE 

An Automated Tool for Developing Experimental Designs: The Computer-Aided 

Design Reference for Experiments (CADRE) 

5c. PROGRAM ELEMENT NUMBER 

5d. PROJECT NUMBER 

5e. TASK NUMBER 

6. AUTHOR(S) 

Robert C. Williges and Sam E. Middlebrooks 

5f. WORK UNIT NUMBER 

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 

U.S. Army Research Laboratory 

ATTN: AMSRD-ARL-HR-MV 

Aberdeen Proving Ground MD 21005-5069

8. PERFORMING ORGANIZATION 

 REPORT NUMBER 

ARL-TN-0341

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S) 

11. SPONSOR/MONITOR'S REPORT 

 NUMBER(S) 

12. DISTRIBUTION/AVAILABILITY STATEMENT 

Approved for public release; distribution unlimited.

13. SUPPLEMENTARY NOTES 

14. ABSTRACT 

Based on past problems with data collection in experiments, experimental designers need a tool to assist in establishing effective 

experiments in which the data collection method minimizes the amount of data required and the data collection effort while 

maximizing the effectiveness and power of the data. A desktop tool, the Computer-Aided Design Reference for Experiments 

(CADRE), was developed for this purpose. The CADRE tool includes over 850 pages of reference material covering 25 topics, 

divided into five major sections, including an introduction to experimental design; supplemental data collection design and 

analysis; basic analysis of variance (ANOVA) designs; advanced ANOVA designs; and empirical model building. In addition, 

the CADRE tool contains over 200 pages explaining 40 examples of statistical analyses covered in the reference material and is 

hyperlinked to version 9.1.3 of the SAS statistical analysis package. Designers can use the CADRE tool to choose experimental 

design procedures to interrogate and build empirical models in support of complex studies, such as future command and control 

(C2) systems computer simulations. 

15. SUBJECT TERMS

Experimental design 

16. SECURITY CLASSIFICATION OF: 

19a. NAME OF RESPONSIBLE PERSON

Sam E. Middlebrooks 

a. REPORT 

U 

b. ABSTRACT 

U 

c. THIS PAGE 

U 

17. LIMITATION 

 OF 

 ABSTRACT 

UU 

18. NUMBER 

 OF 

PAGES 

26 19b. TELEPHONE NUMBER (Include area code) 

(254) 288-9379 

Standard Form 298 (Rev. 8/98) 

Prescribed by ANSI Std. Z39.18 
iii

Contents 

List of Tables iv

Acknowledgments v

1. Introduction 1

2. Description of the CADRE Tool 1

2.1 CADRE Structure............................................................................................................2 

2.2 CADRE Topics................................................................................................................2 

2.2.1 Topic 1. Research Design Process ......................................................................3 

2.2.2 Topic 2. Experimental Designs ...........................................................................4 

2.2.3 Topic 3. Basic Statistical Concepts .....................................................................4 

2.2.4 Topic 4. Supplemental Data Collection Methods ...............................................4 

2.2.5 Topic 5. Analysis of Nominal Scale Data ...........................................................4 

2.2.6 Topic 6. Analysis of Ordinal Scale Data.............................................................4 

2.2.7 Topic 7. Summary of Supplemental Data ...........................................................5 

2.2.8 Topic 8. Introduction to ANOVA .......................................................................5 

2.2.9 Topic 9. ANOVA Summary Table Components ................................................5 

2.2.10 Topic 10. Between-Subjects ANOVA Designs ..................................................5 

2.2.11 Topic 11. Analysis of Comparisons and Interactions..........................................6 

2.2.12 Topic 12. Within-Subjects ANOVA Designs .....................................................6 

2.2.13 Topic 13. Mixed-Factors ANOVA Designs........................................................6 

2.2.14 Topic 14. Summary of Basic ANOVA ...............................................................6 

2.2.15 Topic 15. Introduction to Advanced ANOVA ....................................................7 

2.2.16 Topic 16. Hierarchical ANOVA Designs............................................................7 

2.2.17 Topic 17. Blocking ANOVA Designs.................................................................7 

2.2.18 Topic 18. Fractional Factorial ANOVA Designs................................................7 

2.2.19 Topic 19. Analysis of Covariance (ANCOVA) ..................................................8 

2.2.20 Topic 20. Summary of Advanced ANOVA ........................................................8 

2.2.21 Topic 21. Introduction to Empirical Models.......................................................8 

2.2.22 Topic 22. Multiple Regression ............................................................................8 

2.2.23 Topic 23. Central-Composite Designs (CCDs)...................................................8 

2.2.24 Topic 24. Sequential Experimentation ................................................................9 

2.2.25 Topic 25. Summary of Empirical Models ...........................................................9 
iv

2.3 CADRE Bookmarks........................................................................................................9 

2.4 CADRE Links and Hyperlinks......................................................................................10 

3. The Focus of CADRE 12

4. Conclusions 13

5. References 14

Acronyms and Abbreviations 16

Distribution List 17 

List of Tables 

Table 1. Major topics in CADRE. ...................................................................................................3 

Table 2. Major bookmark tabs in CADRE. .....................................................................................9 

Table 3. Hyperlinked SAS examples in CADRE. .........................................................................11 
v 

Acknowledgments 

The Computer-Aided Design Reference for Experiments (CADRE) project was conducted at the 

Virginia Polytechnic Institute and State University (Virginia Tech) and was supported by 

funding from the U.S. Army Research Laboratory’s (ARL) Human Research and Engineering 

Directorate (HRED), and the ARL Collaborative Technology Alliance on Advanced Decision 

Architectures (ADA) and its prime contractor, Micro Analysis and Design, managed by Dr. 

Michael Strub, the HRED Manager for the ADA. 
vi 

INTENTIONALLY LEFT BLANK. 
1 

1. Introduction 

Several statistical and experimental design and analysis textbooks cover basic and advanced 

experimental design techniques (Keppel and Wickens, 2004; Montgomery, 2008; Myers and 

Montgomery, 2002; Shadish, Cook, and Campbell, 2002; Winer, Brown, and Michels, 1991) as 

well as attempts at automated experimental design (Mills, Meyer, Rickels, and Hoyland, 1982). 

What they do not provide, and what is needed, is a computer-based reference (using modern 

hardware and software technologies) that incorporates an integrated subset of currently accepted 

basic and advanced experimental design approaches that human factors and ergonomics 

researchers can use in the complex system research environments they are investigating. 

Procedural guidance and reference for experimental design and statistical analysis was often 

summarized in hard copy tables in the covers of textbooks (Siegel and Castellan, 1988). With 

the advent of hyperlink technology that can embed such information into machine independent 

Postscript Definition File (PDF) documents, modern computer technology provides the technical 

basis for the development of an interactive desktop tool to access a large reference resource to 

assist researchers in their quest for efficient and effective designs. 

2. Description of the CADRE Tool 

The research on command and control (C2) systems is a real case example of how the 

availability of an experimental design and analysis tool can significantly assist researchers in 

experimental design development and analysis of the results. Appropriate experimental designs 

for complex analyses can be overwhelming to complete, demanding to structure, and incomplete 

in their results if an ineffective experiment is followed. Considering the needs of this type of 

highly complex study provided the basis for developing procedures and processes to ease the 

burden of experimental design development and ensure that an effective design is selected that 

minimizes the data collection effort while maximizing the validity of the data. 

In 2003 the U.S. Army Research Laboratory (ARL), in collaboration with the Virginia 

Polytechnic Institute and State University (Virginia Tech), sponsored the development of just 

such a tool, the Computer-Aided Design Reference for Experiments (CADRE). This three-year 

effort is now coming to fruition and the resulting interactive experimental design and statistical 

analysis reference resource provides a highly effective aid for researchers to quickly establish 

efficient experimental designs for their work. 
2 

2.1 CADRE Structure 

The CADRE tool is as an interactive desktop computer tool designed to aid human factors and 

ergonomics researchers in experimental design and analysis. This tool, with the full 

documentation papers describing it, is on a CD accompanying this report. As the CADRE tool 

was developed with Government funding it is in the public domain and there are no distribution 

limitations associated with it. 

The human factors experimental design and analysis reference material in CADRE is a 

compilation of two technical reports (Slater and Williges, 2006; Williges, 2006). The Williges 

report provides the primary reference material on experimental design in a PowerPoint 

presentation with accompanying notes. Over 850 pages of experimental design and analysis 

reference material for human factors researchers is provided in this report. The content of the 

reference material is presented in a numbered outline format on PowerPoint slides, and 

descriptive text together with appropriate reference citations are provided for the content of each 

slide. At the end of each topic presentation, references are provided to the current scientific 

literature for supplemental readings that are pertinent to the topic presented. The companion 

report by Slater and Williges is an appendix to Williges’ report and provides over 200 pages in 

SAS of program files (SAS, 2004), output files, and explanations for 40 examples of statistical 

analyses covered by the 2006 Williges report.

The CADRE tool has several interactive features that can be used to rapidly locate reference 

material information. These features include extensive bookmarks within the PDF file; links 

between the reference material and the appendix for each example computational problem; and 

hyperlinks from the CADRE tool to the SAS desktop interactive statistical program for each 

example problem. 

2.2 CADRE Topics 

The CADRE reference material covers 25 topics, divided in to five major sections, including an 

introduction to experimental design; supplemental data collection design and analysis; basic 

analysis of variance (ANOVA) designs; advanced ANOVA designs; and empirical model 

building as shown in table 1. 
3 

Table 1. Major topics in CADRE. 

Overview 

Section 1. Introduction to Experimental Design 

Topic 1. Research Design Process 

Topic 2. Experimental Designs 

Topic 3. Basic Statistical Concepts 

Section 2. Supplemental Data Collection and Analysis 

Topic 4. Supplemental Data Collection Methods 

Topic 5. Analysis of Nominal Scale Data 

Topic 6. Analysis of Ordinal Scale Data 

Topic 7. Summary of Supplemental Data 

Section 3. Basic Analysis of Variance (ANOVA) Design 

Topic 8. Introduction to ANOVA 

Topic 9. ANOVA Summary Table Components 

Topic 10. Between-Subjects ANOVA Designs 

Topic 11. Analysis of Comparisons and Interactions 

Topic 12. Within-Subjects ANOVA Designs 

Topic 13. Mixed-Factors ANOVA Designs 

Topic 14. Summary of Basic ANOVA 

Section 4. Advanced ANOVA Designs 

Topic 15. Introduction to Advanced ANOVA 

Topic 16. Hierarchical ANOVA Designs 

Topic 17. Blocking ANOVA Designs 

Topic 18. Fractional Factorial ANOVA Designs 

Topic 19. Analysis of Covariance (ANCOVA) 

Topic 20. Summary of Advanced ANOVA 

Section 5. Empirical Model Building 

Topic 21. Introduction to Empirical Models 

Topic 22. Multiple Regression 

Topic 23. Central-Composite Designs (CCDs) 

Topic 24. Sequential Experimentation 

Topic 25. Summary of Empirical Models 

2.2.1 Topic 1. Research Design Process 

Topic 1 presents the overall research design process. First, the various stages of research are 

shown in a flow diagram. Next, six critical aspects of this process are highlighted, from the 

research problem to research reports. As with all subsequent topics covered in the reference 

material, this topic concludes with a summary followed by suggestions for supplemental 

readings for in-depth coverage of the material covered in this topic. 
4 

2.2.2 Topic 2. Experimental Designs 

Topic 2 is an introduction to experimental design. It begins with an overview of various 

conditions that can threaten the validity of the results of experiments and then offers a general 

notation for designating designs. Two general experimental design alternatives are presented, 

and the case is developed for always choosing randomized experimental designs, if possible. 

2.2.3 Topic 3. Basic Statistical Concepts 

Topic 3 summarizes basic statistical concepts that are fundamental to experimental design. This 

reference material is not a tutorial on basic statistics, but rather, the material highlights and 

summarizes key statistical concepts. It is assumed that users of this reference material already 

have a background in introductory descriptive and inferential statistics. Consequently, this topic 

is designed as a review of basic concepts without providing detailed descriptions or 

mathematical derivations. Users should refer to a textbook such as Walpole’s latest edition on 

introductory statistics (Walpole, Myers, Myers, and Ye, 2002) for details on the concepts 

summarized in this topic if they are not familiar with them. The concepts of sampling 

distributions, the F-distribution, and statistical hypothesis testing are critical to understanding the 

experimental design topics covered in this reference. 

2.2.4 Topic 4. Supplemental Data Collection Methods 

Topic 4 deals with supplemental data collection that augments data from experimental designs. 

Quantitative supplemental data are very important for understanding and interpreting research 

results. Most supplemental data are analyzed by nonparametric analyses as opposed to the 

parametric analyses used on the major dependent variables manipulated in the experimental 

design. Several types of supplemental data collection techniques can be used. This reference 

concentrates on the graphical rating scales that are most often used to collect quantitative 

supplemental data in human factors research. 

2.2.5 Topic 5. Analysis of Nominal Scale Data 

Topic 5 deals with an overview of major supplemental data analysis alternatives that can be used 

with nominal scale data. Only a sample of nonparametric analyses covering the most common 

techniques used in human factors and ergonomics research is presented in this topic. Siegel 

(Siegel and Castellan, 1988) provides a detailed discussion of all of these techniques, and their 

formulae and notation are used throughout this topic for easy reference. The nonparametric 

techniques in this reference are organized around between-subjects and within-subjects 

techniques to facilitate an easy choice of nonparametric analysis procedures for nominal data. 

2.2.6 Topic 6. Analysis of Ordinal Scale Data 

Topic 6 deals with an overview of four nonparametric analysis alternatives that can be used with 

ordinal scale supplemental data. The four procedures described in this reference material are 

often used in human factors and ergonomics research. Once again, Siegel and Castellan (1988) 
5 

provide a detailed discussion of each of these techniques, and their formulae and notation are 

used throughout this topic for easy reference. Similar to the approach followed in the discussion 

on nominal data analysis, the presentation of this topic is organized around between-subjects and 

within-subjects techniques to facilitate choice of nonparametric analysis procedures for ordinal 

data. 

2.2.7 Topic 7. Summary of Supplemental Data 

Topic 7 summarizes the major points discussed in section 2 of the reference material dealing 

with supplemental data. First, supplemental data collection procedures dealing with self reports, 

questionnaires, and rating scales are reviewed. Second, supplemental data analysis techniques 

using common nominal and ordinal scale nonparametric procedures are reviewed. Third, a 

three-step process for dealing with supplemental data in experimental design is presented. 

2.2.8 Topic 8. Introduction to ANOVA 

Topic 8 introduces ANOVA designs by discussing their advantages, basic terms, and the three 

major categories of ANOVA designs used in human factors research. Next, procedures for 

specifying the underlying statistical model that describes the components of any ANOVA design 

are presented. The introduction ends with a discussion and example for using ANOVA for 

statistical hypothesis testing of the difference between two treatment means. 

2.2.9 Topic 9. ANOVA Summary Table Components 

Topic 9 provides an overview of the computational aspects of any between-subjects, withinsubjects, and mixed-factors ANOVA design used in human factors research. The subsections are 

organized around the five major components of the ANOVA Summary Table used for listing the 

results of an ANOVA. Rather than derive formulae for calculating each component, 

computational procedures and algorithms are provided. First, each component of the summary 

table is discussed separately. Then conventions for stating the complete ANOVA summary table 

are presented for each of the three major categories of ANOVA experimental designs used in 

human factors research. 

2.2.10 Topic 10. Between-Subjects ANOVA Designs 

Topic 10 covers the construction and computational details of the first of the three major 

categories of ANOVA designs used in human factors and ergonomics research. Betweensubjects designs are discussed in terms of one-factor, two-factor, and n-factor designs. The 

procedures for determining the sum of squares computational formulae for any ANOVA are 

presented in the discussion of one-factor designs. The concept of an interaction is presented in 

two-factor designs. Generalizations for constructing and analyzing any between-subjects 

ANOVA design are summarized under n-factor designs. Computational examples are provided 

for both a one-way and a two-way between-subjects design. A summary listing of all the 

ANOVA procedural rules and algorithms for conducting any ANOVA analysis in human factors 

research is provided at the end of this topic. 
6 

2.2.11 Topic 11. Analysis of Comparisons and Interactions 

Topic 11 covers analytical techniques that can be used to isolate the form or nature of the main 

effects and interactions that are significant in the overall ANOVA. Basically, these procedures 

deal with multiple paired comparisons of various treatment means. First, this topic covers 

comparisons and provides examples of both planned and unplanned comparison procedures. 

Second, special analytical procedures in addition to paired comparisons are covered, and 

example computations are provided for the analysis of interactions. Although these procedures 

are demonstrated using between-subjects designs, these same techniques are appropriate for 

within-subjects and mixed-factors designs covered in the next two topics. 

2.2.12 Topic 12. Within-Subjects ANOVA Designs 

Topic 12 covers within-subjects ANOVA designs in which each subject receives every treatment 

condition in the human factors experiment. Basic configurations of one- and two-factor designs 

as well as generalizations to n-factor repeated measures designs are covered. The advantages of 

within-subjects designs are summarized as compared to between-subjects designs. Additional 

considerations in using repeated measures are discussed, including the homogeneity of 

covariance assumption, balancing techniques for controlling possible confounding effects of 

treatment orders, and the effect of differential transfer. 

2.2.13 Topic 13. Mixed-Factors ANOVA Designs 

Topic 13 covers the basic configuration and analytical procedures used in mixed-factors 

ANOVA designs, which comprise the third major category of ANOVA designs. Mixed-factors 

designs are composed of both between-subjects and within-subjects factors. This type of 

ANOVA design is often referred to as split-plot designs in scientific literature. These designs are 

used quite frequently in human factors and ergonomic research due to the nature of the 

independent variables being investigated in the experiment. Consider a training research study 

that investigates training methods and practice trials. The researcher must manipulate the 

training condition variable as a between-subjects variable because subjects cannot return to a 

beginning level of knowledge when provided with alternative training. On the other hand, 

practice trials in this training experiment must be manipulated as a within-subjects factor since 

each subject receives multiple trials. Consequently, a mixed-factors design is needed. This topic 

also provides a list of considerations that the research should address when using mixed-factors 

designs. 

2.2.14 Topic 14. Summary of Basic ANOVA 

Topic 14 provides a brief summary review and roadmap of basic ANOVA in terms of 

fundamental considerations, ANOVA design classification, ANOVA generalizations, and steps 

in the overall ANOVA design and data interpretation process. Details on these issues are 

provided in specific topics referenced. 
7 

2.2.15 Topic 15. Introduction to Advanced ANOVA 

Topic 15 introduces topics in advanced ANOVA that expand upon the basic ANOVA 

procedures, rules, and algorithms already described in section 3 of the reference material. Two 

examples of extending basic ANOVA (i.e., Quasi-F ratios and Randomized Blocks Designs) are 

presented in detail. In addition, this introduction provides an overview of several special purpose 

ANOVA design and analysis procedures that satisfy various constraints present in human factors 

and ergonomics research. Detailed discussions of these procedures are presented in subsequent 

topics. 

2.2.16 Topic 16. Hierarchical ANOVA Designs 

Topic 16 covers hierarchical ANOVA designs that include nested factors of interest. This topic 

describes both the basic layout and the ANOVA computations involved in this class of 

experimental designs. These basic procedures can be generalized to higher-order hierarchical 

designs, and the use of hierarchical designs in human factors and ergonomics research is 

discussed. This topic ends with a general summary of the considerations of using hierarchical 

designs in human factors and ergonomics research. 

2.2.17 Topic 17. Blocking ANOVA Designs 

Topic 17 deals with ANOVA experimental designs that can be used to control nuisance variables 

that are confounded with the factors of interest in an experiment. These confounding factors 

may include the effect of repeated testing sessions or experimenter bias resulting from using 

several experimenters in data collection. Confounding is controlled through blocking designs 

where blocks represent the nuisance variables. Through the use of modular representation, the 

exact nature of the block confounding with specific treatment effects of interest can be 

determined by the experimenter in order to choose an experimental design to avoid confounding 

blocks with effects of major interest in a factorial ANOVA design. The use of blocking is 

demonstrated through 2k

 factorial designs and extended to pseudo-factor blocking. 

2.2.18 Topic 18. Fractional Factorial ANOVA Designs 

Topic 18 deals with fractional replicates of full factorial ANOVA designs. When large-scale 

factorial designs are used in human factors and ergonomics research, constraints such as time, 

money, and equipment availability may make the complete factorial design unfeasible and data 

collection must be restricted to a portion of the design. Obviously, some effects in the full 

factorial design cannot be evaluated and some effects will be confounded with others if data are 

not collected on the entire design. The experimenter must select a subset of the complete design 

that yields the most useful data. Fractional-factorial designs are ANOVA designs in which only 

a fractional portion of the complete factorial designs are observed. Fractional replications of 2k

, 

3k

, and 5k designs can be constructed through modular representation. Most often 2k-p fractional 

replicates are used in human factors research to avoid subsequent confounding of partial effects 

that are present in 3k

 and 5k designs. Alternatively, a subset of fractional-factorial designs, called 
8 

Latin Square designs, can be used to specify the subset of treatment conditions to observe when 

the experimenter is only interested in testing the main effects of three factors of interest. 

Consequently, this topic describes both 2k-p fractional replicates and Latin Square designs. 

2.2.19 Topic 19. Analysis of Covariance (ANCOVA) 

Topic 19 deals with an analytical technique for reducing the effect of a covariate to increase the 

sensitivity of the F-test on effects of interest to the experiment. The covariate is correlated with 

the dependent variable and its effect is removed through simple linear regression. Both 

calculations of correlation and simple regression are described as the basic components of 

ANCOVA. Basic computations in ANCOVA and subsequent interpretations of results are 

described in this topic. 

2.2.20 Topic 20. Summary of Advanced ANOVA 

Topic 20 summarizes the advanced ANOVA techniques described in the reference material. 

This topic provides a summary of design constraints and describes a process for addressing them. 

2.2.21 Topic 21. Introduction to Empirical Models 

Topic 21 provides an overview of quantitative models with an emphasis on empirical model 

building developed through efficient experimental design. Empirical models are descriptive 

models of human behavior based on results obtained through one or more controlled experiments 

that can be used to predict human performance in complex systems. These empirical models can 

assist the human factors specialist in conducting design tradeoffs of critical interface parameters. 

2.2.22 Topic 22. Multiple Regression 

Topic 22 provides an overview of multiple regression procedures used to generate empirical 

models that involve more than one factor. First, multiple linear regression is described to 

demonstrate the calculations involved when considering more than on factor in the empirical 

model. Both classical and modern regression procedures for selecting the best multiple 

regression equation are described. Next, polynomial regression is discussed as the general form 

for stating empirical models in human factors research. 

2.2.23 Topic 23. Central-Composite Designs (CCDs) 

Topic 23 describes experimental designs that can be used to collect data for building secondorder empirical models. Specifically, this topic focuses on CCDs that were developed to explore 

response surfaces using empirical models. The background, specification, analysis, and an 

example of a CCD along with a comparison to alternative second-order experimental designs are 

discussed in this topic. 
9 

2.2.24 Topic 24. Sequential Experimentation 

Topic 24 incorporates the previous topics into an overall strategy for conducting research on 

complex systems often addressed in human factors and ergonomics research. Due to nature of 

complex research, the experimenter can conduct a series of small independent, but interrelated, 

experiments rather than one large complex study. This topic describes general strategies for 

conducting complex experiments and specifically draws upon considerations made in response 

surface methodology for conducting a series of small, interrelated studies. A general paradigm 

for conducting sequential research is presented along with a detailed example of using this 

paradigm in human factors research that resulted in a set of guidelines for sequential 

experimentation. The topic concludes with a description of combining sequential experiments 

into a common database that incorporates the results of several experiments. This database can 

then be interrogated to generate integrated empirical models across experiments. 

2.2.25 Topic 25. Summary of Empirical Models 

Finally, topic 25 summarizes the empirical model building techniques. The purpose and 

characteristics of empirical model building experiments are reviewed, the major components of 

empirical models are listed, and a sequential experimentation process for human factors research 

is provided. 

2.3 CADRE Bookmarks 

The PDF file in CADRE is heavily bookmarked around the nine expandable major bookmark 

tabs shown in table 2, which include over 500 bookmarks when fully expanded. The bookmarks 

were designed to provide the CADRE user with alternative paths for rapid location of 

information within the tool. 

Table 2. Major bookmark tabs in CADRE. 

Bookmark Tab 1. Forward 

Bookmark Tab 2. List of Topics 

Bookmark Tab 3. Notes on Computational Examples 

Bookmark Tab 4. Experimental Design Procedures 

Bookmark Tab 5. Statistical Analysis Procedures 

Bookmark Tab 6. Appendix on SAS Examples 

Bookmark Tab 7. Acknowledgements 

Bookmark Tab 8. References/Supplemental Readings 

Bookmark Tab 9. Subject Index 

The first two major bookmark tabs shown in table 2 deal with selecting specific content in 

CADRE. Bookmark 1 is a single bookmark to an overview for using the CADRE tool. 

Bookmark tab 2 expands to provide immediate access to various subsections of the reference 

material for each topic in the table of contents listed in table 1. 
10 

Two major tab sections shown in table 2 expand to bookmarks that provide direct access to 

various experimental design and statistical analysis procedures described in the reference 

material. Experimental design procedures in bookmark tab 4 deal with issues such as research 

stages, critical research methods, basic and advanced ANOVA design selection, paradigms and 

guidelines for sequential experimentation, supplemental data collection, and components of 

research reports. Bookmark tab 5 deals with various statistical analysis procedures including 

basic statistics; various basic and advanced ANOVA computations; simple, multiple linear, and 

polynomial regression analysis; and empirical model building procedures. 

Bookmark tabs 3 and 6 expand to the 40 computational examples provided throughout CADRE 

(see section 2.4). Bookmark tab 3 expands to a discussion of statistical computations for each 

example in the reference material. Alternatively, bookmark tab 6 provides immediate access to 

the SAS program appendix (Slater and Williges, 2006) that describes all the SAS (SAS, 2004) 

statistical analysis procedures used in CADRE. 

The final three major bookmark tabs shown in table 2 deal with additional reference material. 

Bookmark tab 7 is a single bookmark to the tool acknowledgements. Bookmark tab 8 includes 

an alphabetical listing of all references used in the CADRE tool, and can be expanded to provide 

bookmarks to supplemental reading for book chapters and articles in the scientific literature to 

elaborate upon the reference material presented for each topic listed in table 1. Finally, 

Bookmark tab 9 expands to an extensive subject index arranged alphabetically for easy access to 

various topics in the reference material. 

2.4 CADRE Links and Hyperlinks 

The PDF file of CADRE also supports links between the computational descriptions of the 

examples in the reference material with the 40 examples in the Slater and Williges (2006) 

appendix, as shown in table 3. The reference material describes the various steps in the 

statistical calculation; whereas, the use of the SAS (2004) statistical package for conducting the 

analysis is described in the appendix. Each example in the appendix provides a problem 

description, a SAS input file, a SAS output file, and output explanation. Notes are provided 

throughout the appendix to describe tailored modifications when using various SAS procedures. 
11 

Table 3. Hyperlinked SAS examples in CADRE. 

Example 1. Interval Estimation 

Example 2. Single-Sample t-Test 

Example 3. Between-Subjects t-Test 

Example 4. Within-Subjects t-Test 

Example 5. Chi-Square Goodness of Fit Test 

Example 6. Chi-Square Test of Independence (2x2 Contingency Table) 

Example 7. Chi-Square Test of Independence (RxC Contingency Table) 

Example 8. Chi-Square Test of Independence (Two Additive 2x2 Partitions) 

Example 9. McNemar Change Test 

Example 10. Cochran Q Test 

Example 11. Kolmogorov-Smirnov Tests 

Example 12. Kruskal-Wallis One-Way ANOVA 

Example 13. Wilcoxon Signed Ranks Test 

Example 14. Friedman Two-Way ANOVA 

Example 15. One-Factor, Between-Subjects ANOVA 

Example 16. Two-Factor, Between-Subjects ANOVA 

Example 17. Planned Comparisons 

Example 18. Unplanned Comparisons 

Example 19. Analysis of Interactions 

Example 20. One-Factor, Within-Subjects ANOVA 

Example 21. Two-Factor, Within-Subjects ANOVA 

Example 22. Geisser-Greenhouse and Huynh-Feldt Corrections 

Example 23. Testing Order Effects in Balanced Latin Squares 

Example 24. Within-Subjects and Between-Subjects Design Comparison 

Example 25. Two-Way, Mixed-Factors ANOVA 

Example 26. Complete Hierarchical Between-Subjects Design 

Example 27. Partial Hierarchical Between-Subjects Design 

Example 28. Simple Blocking of 2k Within-Subjects Design 

Example 29. Complex Blocking of 2k Within-Subjects Design 

Example 30. One-Half Replicate of 24 Between-Subjects Design 

Example 31. 4x4 Latin Square Designs 

Example 32. Linear Correlation Coefficient 

Example 33. Alternative Linear Correlations 

Example 34. Simple Linear Regression 

Example 35. Analysis of Covariance (ANCOVA) One-Way, Between-Subjects Design 

Example 36. Multiple Linear Regression 

Example 37. Best Regression Equation 

Example 38. Polynomial Regression 

Example 39. Orthogonal, Between-Subjects, Central-Composite Design 

Example 40. Blocked, Within-Subjects, Central-Composite Design

Each of the 40 statistical examples in the Slater and Williges (2006) appendix is also hyperlinked 

to version 9.1.3 of the SAS (2004) statistical analysis package. If the SAS interactive statistical 

package is active on a user’s computer, the CADRE tool can be directly linked to the SAS 
12 

program file for any example covered in the appendix. These programs also can be used as a 

guide for analyzing an experimenter’s specific data set by simply modifying the names and 

numbers of independent variables and the specific data set of the dependent variables. 

3. The Focus of CADRE 

The major focus of CADRE is on understanding the various techniques and processes considered 

in applied experimental design by researchers who have a basic knowledge in statistics and 

experimental design. In order to conduct complex system research experiments, researchers 

must consider a variety of experimental design and analysis techniques. The CADRE tool 

features a review of the design construction, assumptions, and basic computations in a variety of 

experimental design techniques with an emphasis on ANOVA procedures and related designs. 

Rather than mathematical derivations, algorithms and guidelines are used to promote 

understanding. Saying that, references to the scientific literature are provided throughout 

CADRE for details on the various techniques and their use in complex applied research 

problems. Most researchers use statistical packages for analysis, so the focus is on the use and 

interpretation of SAS rather than hand calculations. Examples of hand calculations are provided 

in CADRE to promote a better understanding of the various experimental design techniques. 

The process of choosing various techniques in applied experimental design is also a focus of 

CADRE. This includes considering the various stages in the research process; the supplemental 

data collection and analysis process; the process of empirical model building; and a process for 

conducting sequential experimentation. 

CADRE represents one example of a computer-based desktop interactive tool that can assist 

researchers in conducting complex systems research. This tool can be expanded to incorporate 

additional experimental design procedures and additional statistical analyses packages that can 

hyperlinked to the reference material in CADRE. This tool needs to be widely available to the 

research community so that improvements based on usage can be made to both the content and 

interactive nature of CADRE. 

In addition, the approach used in developing the CADRE tool can be extended to other desktop 

tools to assist the human factors researcher. CADRE can be considered the first prototype in a 

series of tools. Other computer-based tools addressing a variety of research methods (e.g., 

multivariate analysis, survey procedures, and cognitive task analysis), system design methods 

(e.g., focus groups, design guidelines, specifications, and requirements), and evaluation methods 

(e.g., usability assessment, operator workload evaluation, situation awareness, and crew resource 

management) can be added to this family of tools to be used by human factors professionals. 
13 

4. Conclusions 

Research on Army C2 systems requires knowledge and use of complex experimental design and 

analysis. The CADRE tool can assist in designing effective experiments that allow data 

collection requiring minimum effort and maximum effectiveness of results. Sections 4 and 5 of 

CADRE deal specifically with fractional-factorial designs, polynomial regression, empirical 

model building, and sequential experimentation, which are particularly appropriate for 

investigating emerging C2 systems. 

The studies that provided impetus to spawn the development of the CADRE tool were part of 

ongoing efforts to develop better understandings of how to structure socio-technical work 

environments. These conditions are characterized where teams function in a high stress, time 

dependent, decision-oriented work setting. In these situations, the goal is to optimize human 

cognitive performance in this type of work domain, such as can be found in military C2 systems. 

This work is especially relevant for studies of systems not yet in existence, such as the Future 

Combat System (FCS) and Objective Force Warrior (OFW). 
14 

5. References 

Keppel, G.; and Wickens, T. D. Design and Analysis: A Researcher’s Handbook, 4 ed.; Prentice 

Hall: New Jersey, 2004. 

Middlebrooks, S. E.; Knapp, B. G.; Barnette, B. D.; Bird, C. A.; Johnson, J. M.; Kilduff, P. W.; 

Schipani, S. P.; Swoboda, J. C.; Wojciechowski, J. Q.; Tillman, B. W.; Ensing, A. R., 

Archer, S. G.; Archer, R. D.; Plott, B. M. CoHOST - Computer Simulation Models to 

Investigate Human Performance Task and Workload Conditions in a U.S. Army Heavy 

Maneuver Battalion Tactical Operations Center. In Proceedings of the HFES 43rd Annual 

Meeting, Human Factors and Ergonomics Society: Houston, TX, 1999, pp 242–246. 

Mills, R. G.; Meyer, E. G.; Rickels, W.; Hoyland, C.. Introduction to Experimental Design and 

the User Assisted Automated Experimental (Test) Design Program (Aed): Version II. 

Volume I; Technical Report TM-HU-567/001/00; Aerospace Medical Research Laboratory, 

Aerospace Medical Division, Air Force Systems Command, 1982. 

Montgomery, D. C. Design and Analysis of Experiments, 7 ed.; John Wiley and Sons, Inc.: New 

York, NY, 2008. 

Myers, R. H.; Montgomery, D. C. Response Surface Methodology, 2 ed.; John Wiley and Sons, 

Inc.: New York, NY, 2002. 

SAS System Version 9.1.3 for Windows and Online Documentation. The SAS Institute, Inc.: 

Cary, NC, 2004. 

Shadish, W. R.; Cook, T. D.; Campbell, D. T. Experimental and Quasi-Experimental Designs for 

General Causal Inference, Houghton Mifflin: New York, NY, 2002. 

Siegel, S.; Castellan, N. J., Jr. (1988). Nonparametric Statistics for the Behavioral Sciences, 2 

ed.; McGraw-Hill: New York, NY, 1998. 

Slater, C. R.; Williges, R. C. Appendix: SAS Examples for Human Factors Experimental Design 

and Analysis Reference; Technical Report HFEEC-06-02; Virginia Polytechnic Institute and 

State University: Blacksburg, VA., 2006. 

Walpole, R. E.; Myers, R. H.; Myers, S. L.; Ye, K. Probability and Statistics for Engineers and 

Scientists, 7 ed.; Prentice Hall: New Jersey, 2002. 

Williges, R. C. Human Factors Experimental Design and Analysis Reference; Technical Report 

HFEEC-06-01; Virginia Polytechnic Institute and State University: Blacksburg, VA, 2006. 
15 

Williges, R. C. Computer-Aided Design Reference for Experiments (Cadre): A Tool for Human 

Factors and Ergonomics Research; Technical Report CD-ROM-07-01; Virginia Polytechnic 

Institute and State University: Blacksburg, VA, in press. 

Winer, B. J.; Brown, D. R.; Michels, K. M. Statistical Principles in Experimental Design, 3 ed.; : 

McGraw-Hill, Inc.: New York, NY, 1991. 
16 

Acronyms and Abbreviations 

ADA Advanced Decision Architectures 

ANCOVA Analysis of Covariance 

ANOVA analysis of variance 

ARL U.S. Army Research Laboratory 

C2 command and control 

CADRE Computer-Aided Design Reference for Experiments 

CCD Central-Composite Designs 

FCS Future Combat System 

HRED Human Research and Engineering Directorate 

OFW Objective Force Warrior 

PDF Postscript Definition File 

Virginia Tech Virginia Polytechnic Institute and State University 
17 

No. of

Copies Organization

1 ADMNSTR 

ELECT DEFNS TECHL INFO CTR 

 ATTN DTIC OCP (ELECTRONIC COPY) 

 8725 JOHN J KINGMAN RD STE 0944 

 FT BELVOIR VA 22060-6218 

10 ARMY RSRCH LABORATORY-HRED 

 ATTN AMSRD ARL HR MV 

 HQ USAOTC S MIDDLEBROOKS 

 91012 STATION AVE RM 348 

 FT HOOD TX 76544-5073 

1 US ARMY RSRCH LAB 

ATTN AMSRD ARL CI OK TP 

TECHL LIB T LANDFRIED 

 BLDG 4600 

 APG MD 21005-5066 

1 US ARMY RSRCH LAB 

 ATTN AMSRD ARL CI OK PE 

 TECHL PUB 

 ATTN AMSRD ARL CI OK TL 

 TECHL LIB 

ATTN IMNE ALC IMS MAIL & 

RECORDS MGMT 

 ADELPHI MD 20783-1197 

1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR M 

 DR M STRUB 

 6359 WALKER LANE SUITE 100 

 ALEXANDRIA VA 22310 

1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR ML J MARTIN 

 MYER CENTER RM 2D311 

 FT MONMOUTH NJ 07703-5601 

1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MZ 

 A DAVISON 

 199 E 4TH ST STE C TECH PARK BLG 2 

 FT LEONARD WOOD MO 65473-1949 

1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MD T COOK 

 BLDG 5400 RM C242 

 REDSTONE ARSENAL AL 35898-7290 

No. of

Copies Organization

1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MJK 

 J HANSBERGER 

 JFCOM JOINT EXPERIMENTATION J9 

 JOINT FUTURES LAB 

 115 LAKEVIEW PARKWAY SUITE B 

 SUFFOLK VA 23435 

1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MQ M 

 R FLETCHER 

 US ARMY SBCCOM NATICK 

 SOLDIER CTR 

 AMSRD NSC SS E BLDG 3 RM 341 

 NATICK MA 01760-5020 

1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MY 

 DR J CHEN 

 12423 RESEARCH PARKWAY 

 ORLANDO FL 32826 

1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MS 

 C MANASCO 

 SIGNAL TOWERS 118 MORAN HALL 

 FORT GORDON GA 30905-5233 

1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MU 

 M SINGAPORE 

 6501 E 11 MILE RD MAIL STOP 284 

 BLDG 200A 2ND FL RM 2104 

 WARREN MI 48397-5000 

1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MF 

 C HERNANDEZ 

 BLDG 3040 RM 220 

 FORT SILL OK 73503-5600 
18 

No. of

Copies Organization

1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MW E REDDEN 

 BLDG 4 ROOM 332 

 FT BENNING GA 31905-5400 

1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MN 

 R SPENCER 

 DCSFDI HF 

 HQ USASOC BLDG E2929 

 FORT BRAGG NC 28310-5000 

1 ARMY G1 

 ATTN DAPE MR B KNAPP 

 300 ARMY PENTAGON ROOM 2C489 

 WASHINGTON DC 20310-0300 

1 DIRECTOR 

 US ARMY RSCH LABORATORY 

 ATTN AMSRD ARL CI OK TP 

 S FOPPIANO 

 BLDG 459 

 APG MD 21005-5066 

1 US ARMY OPERATIONAL TEST COM 

 METHODOLOGY AND ANALYSIS DIR 

 ATTN TEOT MA 

 WAYLAND SMITH 

 91012 STATION AVENUE 

 FORT HOOD TX 76544-5068 

1 US ARMY OPERATIONAL TEST COM 

 TRANSFORMATION TECH DIR 

 ATTN TEOT-TT 

 GAYLE SHULL 

 91012 STATION AVENUE 

 FORT HOOD TX 76544-5068 

TOTAL 38 (1 Elec, 37 HCs)

.