Web Based Factorial Survey and Statistical Analysis Software 

Sam Addala, Principal Investigator, e4Xchange Corporation (sam@e4xchange.com) 

Matthew Hogben PhD, Project Officer, Centers for Disease Control and Prevention 

Vin Addala, Software Executive, e4Xchange corporation 

Abstract: 

Virtually anyone seeking information from a given group of people faces this dilemma: 

as one asks for more information, potential participants become more reluctant to provide 

all the information. Such reluctance is as likely to be on the basis of time and trouble as 

on grounds of sensitivity. Factorial surveys, however, permit researchers to sample 

different combinations of information from different respondents, thus allowing different 

respondents to contribute data to a subset of the full survey. To date, existing factorial 

survey software does not allow a researcher naïve to programming to design a survey and 

sampling protocol, and gather data (via internet) under the same umbrella. 

e4Xchange Corporation, a software development organization and a NASA 

Industry Partner, has developed an intuitive and user friendly software solution to address 

the end–to–end needs of researchers conducting factorial and non-factorial surveys for 

the Centers for Disease Control and Prevention. e4Xchange has completed the Phase I 

proof of concept and has moved on to Phase II, full scale development. Proof of concept 

entailed developing factorial algorithms such that the survey designer could specify 

dimensions and levels of dimensions, which would then be sorted into all possible 

combinations and randomly presented. 

Phase II is being built around four functions: design, web publishing, response, 

and analysis. Design is the cardinal function, with the user able to develop “static” 

survey questions with varied response options, as well as factorial questions with varying 

numbers of factors and levels of factors. The designer simply writes a story (vignette) in 

a text box and names factors directly within that vignette. The designer then defines and 

names levels of factors, earmarks invalid combinations for partial factorial designs, and 

lets the program do the rest. Eligibility criteria and block randomization, if needed, can 

be built into the design, and the designer can specify or randomize the order of questions 

(or even just some of them), as well as denote skip patterns. Notably, the software is 

unique in that the user need not have to be a computer programmer to build complex 

factorial surveys. 

Respondents log on, anonymously or confidentially, and either as a convenience 

sample or according to user-designed sampling protocols. The output is readable in a 

spreadsheet format, and therefore via common statistical programs (e.g., SPSS or SAS). 

Potential statistical analysis of output ranges from descriptive statistics to experimental 

frameworks from the General Linear Model (e.g., analysis of variance/regression, curve 

estimation). The program has web publication functions: when to publish surveys on the 

web, for how long, and for how many respondents. Survey access can be web-based, 

email, or hard copy. 

In sum, the software is designed to accommodate the needs of data-gathering and 

statistical analysis requirements of government, educational, and private industry. 
Introduction 

Virtually anyone seeking information from a given group of people faces this 

dilemma: as one asks for more information, potential participants become more reluctant 

to provide all the information. Sensitivity is certainly one issue, but time and trouble is 

another. When the test taker is remote from the administrator, the issue is exacerbated – 

anyone who has administered a web-based survey will have had people drop out simply 

by shutting down the browser. To persevere with a lengthy survey stuffed with detailed 

questions costs more money and risks enrollment and retention rates, not to mention 

sample bias. Thus, one often has to reduce the number of dimensions assessed (e.g., 

topic areas or domains of content) and to sample them more crudely. 

Public health contains many examples of such reductions in sampling and 

sampling complexity. One example lies in health care-seeking: a questionnaire asking 

about clinical assessment of a sexual history will typically have a basic yes/no question: 

Do you take a sexual history from your patients? Ongoing research, however, suggests 

that many who answer yes to this question answer no when they are asked if they assess 

particular elements of a sexual history. (1) 

Factorial surveys, however, permit researchers to sample numerous different 

levels of numerous dimensions while maintaining their ability to examine interactions 

between the different dimensions. In a factorial survey approach, each respondent 

receives at least one stimulus question (typically a vignette) that contains one 

combination of systematic variations in the levels of each dimension included in the 

survey. 

2

The Details of Factorial Survey Sampling 

 Let us observe the process in more detail. The sample material that follows is 

drawn from one of the author’s previous work on clinical assessment of patients with a 

possible sexually transmitted infection. (2) First, we show the requirements for a 

standard survey in this area that covers the necessary dimensions and sampling level of 

detail. The dimensions included symptoms, patient characteristics, physician 

characteristics, sex partners and partner management, with the intention of assessing the 

individual effects (i.e., main effects controlling for the effects of the other dimensions) 

and the interactions of these factors. A non-factorial survey maintaining the necessary 

detail would rapidly become a very complex instrument. The problem is exacerbated 

further if each dimension has several levels (e.g., several different types of patient 

reported symptoms). If a survey was confined to three symptoms, two levels of every 

other dimension mentioned above, and asked about only one potential physician response 

(e.g., whether to order an STD test), then respondents would have to answer 24 questions 

(3 x 2 x 2 x 2 x 1) where each question would differ in only a small degree from the one 

before. Each question would essentially constitute a small case study. Asking about two 

behaviors would require 48 questions and asking about four behaviors would increase the 

number of questions to 96, quickly exhausting respondents’ patience and probably 

reducing survey participation. In the research from which this example is drawn, the 

authors were interested in 13 possible clinical actions! One solution to this problem of 

exponentially increasing numbers of questions is to ask about each dimension separately 

(e.g., would you be more likely to order an STD test if the patient was male?), but this 

eliminates examination of the interactions between the identified dimensions. When, as 
in this example, the interactions are relevant to clinical practice, to lose the ability to 

measure interactions reduces the applicability of the research. 

 Factorial surveys permit researchers to sample different levels freely while 

maintaining their ability to examine interactions between the different dimensions. In a 

factorial survey approach, each respondent receives a vignette that systematically varies 

the levels of each factor included in the study. The survey’s creators ensure that they 

have covered every possible hypothetical combination and that they have enough 

respondents to yield adequate statistical power. Taking the 96-combination version from 

the previous paragraph as an example, a researcher who wanted 10 responses per 

combination, and who was willing to give two combinations (i.e., case vignettes) to each 

respondent, would need only 480 respondents in total. What one gives up with the 

factorial approach is the advantage of collecting complete data for each respondent on all 

possible variations of the vignette, so the factorial approach is most useful when 

variables, rather than individuals, are the primary item of interest. 

 In the study of clinical responses to patients presenting with possible sexually 

transmitted infections, (2) the authors chose to include patient symptoms, as they are 

theoretically the primary determinant of the clinical encounter. They also included the 

health status of the patient’s sex partner(s) (if known) and the patient’s risk behavior(s), 

which are indirect, but important, diagnostic tools that identify cues that a given patient 

may have an inflated STD risk. Finally, when discussion of sex and sexuality is 

concerned, current literature suggested the sex of the physician and the patient may affect 

the course and comfort of discussions about sensitive behaviors during a patient visit. 

Consequently, we chose patient sex as one dimension for the factorial survey and 

physician sex as the final dimension. In total, five dimensions were assessed. Four could 

be embedded in the vignettes, and the final dimension was the sex of the responding 

physician. By comparing physicians’ reports of what clinical actions they would take in 

response to a given vignette, it becomes possible to construct a portrait of physicians’ 

reactions to different hypothetical patient presentations. 

3

Practical Issues Pertaining to Devising a Factorial Survey 

Based on the description above, one might expect that factorial surveys pervade 

the literature, but they do not. The survey noted above required substantial contract 

funds, and over a year to construct. There are computer-based algorithms for generating 

combinations, but these are generally either ad hoc or require programming language 

skills. What is needed is a user-friendly construction tool, with user-friendly entailing no 

need for programming language skills and an output that can be analyzed with the same 

statisitcal tools as are used for other surveys. 

In San Jose, CA, e4xchange corporation has completed a Phase I proof of concept 

and has moved on to Phase II, full scale development. Proof of concept entailed 

developing factorial algorithms such that the survey designer could specify dimensions 

and levels of dimensions, which would then be sorted into all possible combinations and 

randomly presented. The design component was accessible via a web server, respondents 

could log on, and the data output was readable in a spreadsheet format, and therefore via 

common statitical programs (e.g., SPSS or SAS). 

 Phase II is being built around four functions: design, administration, response, and 

analysis. Design is the cardinal function, with the Phase II software including the 
addition of static questions (e.g., “Table 1” items) and the ability to simply write a 

vignette in a text box and name the dimensions directly within that vignette. The 

designer then defines and names levels of dimensions, earmarks invalid combinations for 

partial factorial designs, and lets the program do the rest. Eligibility criteria and block 

randomization, if needed, can be built into the design, and the designer can specify or 

randomize the order of questions (or even just some of them), as well as denote skip 

patterns. Response categories can be categorical, ordinal, interval, or ratio, captured via 

numbers or text. The designer can create a qualitative measure as well as a quantitative 

measure, or mix and match. 

The program will also have full administrative functions: when to publish surveys 

on the web, for how long, and for how many respondents. The administrator can permit 

people to fill out the survey once or multiple times, choose whether to allow the 

respondent to exit and return to the survey, and whether to allow non-responses (IRBs 

will naturally insist on the latter for research). Survey access can be web-based, email, or 

hard copy. Response functions are as above, as constrained by the administrator. The 

most efficient use of analysis seems to be to produce data that is readable in CDC’s most 

common analysis programs, such as SPSS, SAS, and Epi Info. A generic spreadsheet is 

readable in these formats, but Phase III versions may permit one to save directly into one 

of those program formats. 

4