The World’s Simplest Survey Microsimulator (WSSM)

Alan F. Karr, Satkartar K. Kinney and Lawrence H. Cox

National Institute of Statistical Sciences

Research Triangle Park, NC 27709

1 The Need

More sharply than in the past, the future of official statistics surveys is framed by data quality–cost

tradeoffs. In fact, in a larger sense, the issue may be decision quality–cost tradeoffs (Karr, 2012). 

Current and prospective budget pressures dictate that these tradeoffs by informed by scientific 

knowledge. We believe that there is a fundamental gap in survey science—there is no simulation

laboratory for surveys. The World’s Simplest Survey Microsimulator (WSSM) is a step toward 

filling this gap. 

Despite impressive advances in survey theory and methodology, many questions remain to 

which extant knowledge only suggests the answers. One examples is the effects on uncertainties 

of editing, imputation, record linkage, and statistical disclosure limitation (SDL) (Cox et al., 2011). 

Operational decisions reflect the accumulated experience of field personnel, but currently and may 

always lack a theoretical basis. 

Making survey science in part a laboratory science would have dramatic effect. However, of 

course, many “real-world” experiments are simply not possible. Simulation is a feasible, powerful 

alternative. For instance, in Karr (2011), we used simulation to study the differences among several 

configurations of the K–12 longitudinal studies conducted by the National Center for Education 

Statistics (NCES). Among conclusions that arose is that continuation of even small numbers of students from one study to the next is of limited statistical value. Using the real world as a laboratory 

in this case was infeasible. 

Simulation is used in settings ranging from social networks to healthcare. It can make a difference for surveys, because the future can only be more challenging than the past as problems such 

as use of administrative data and disappearance of land-line telephones become more intense. 

2 What is a Survey Microsimulator?

As suggested in §1, a survey microsimulator is an in silico simulation laboratory for surveys—a 

modular, extensible computer model (set of programs) that is: 

Agent-Based, with explicit representation of the survey process, including 

Entities: Subjects, people, interviewers, . . . and their characteristics, including survey variables

1 
Interactions: Interviews, nonresponse, callbacks, . . .

Costs, both fixed and variable 

Operational decisions.

Powerful enough to handle realistic scales 

Simple enough to conduct serious experiments 

Credible enough to be used. 

Responding to these criteria, the WSSM is deliberately oversimplified. Its short-term purpose 

is to support sensitivity analyses that 

• Demonstrate that even simple models can reflect methodological, policy and operational 

considerations; 

• Inform the course of more elaborate modeling efforts in the future. 

In particular, a key goal is to provoke suggestions for enrichments. 

Version 1 of WSSM has three essential characteristics: 

1. The entire underlying population and the behavior on which the survey is focused are both 

simulated, and serve as “ground truth” for calculating measures of data quality. 

2. The survey responses themselves are simulated. 

3. The measures of data quality quantify the fidelity of inferences drawn from the survey responses compared to the same inferences based on the entire population. 

3 Generalities

The focus of WSSM is household surveys involving interviews: sample units are households, and 

the survey responses are amounts spent on various categories of goods and services. 

We employ the following notation: 

• P = population 

• i = data subject/survey unit (e.g., a household) 

• Xi = population attributes for unit i

• Yi = behavioral attributes for unit i, about which the survey questions are meant to elicit 

information 

• S ⊆ P = sample for the survey 

• Wi = sample weight of i ∈ S

2 
• ICj = characteristics of interviewer j

• IAi = interviewer assigned to unit i

• R ⊆ S = set of (unit) respondents 

• Wi

∗ = nonresponse adjusted weight of i ∈ R

• Di = responses to survey questions from unit i ∈ R

• Ci = cost associated with (attempted, if i 6∈ R) data collection from unit i. 

4 Global Assumptions and Structure

Essentially everywhere there could be a conditional independence assumption, there is one. For 

instance, in §6, the {Yi

: i ∈ P } are conditionally independent given {Xi: i ∈ P }. 

The WSSM is extensible, and therefore modular. In the next sections, we describe its principal 

modules. Figure 1 contains a flowchart showing the relationships among them. 

5 Population Module

The Population Module generates the target population P for the survey. For simplicity, the frame 

is assumed to be the target population. 

Generalities The Population Module creates (a subset of) the frame variables Xi for each member 

i of the target population. For experimental purposes, these can be taken from a public use 

dataset from one of the FedStats surveys, such as the American Community Survey (ACS). 

Version 1 The “population” is an ACS microdata sample. There are four categorical attributes: 

number of adults in the household, number of children in the household, geography (at a 

level TBD) and the age of the head of household: 

Xi = (NAi, NCi, Gi, HHAi). (1) 

Extensions are to model undercoverage and add more attributes. 

6 Behavior Module

A distinguishing characteristic of WSSM is that it for all elements of the population it includes

synthesized values of the behavioral variables Yi about which the survey is seeking to gather 

information. The Behavior Module generates these attributes for the population generated by 

the Population Module. 

3 
Generalities Conceptually, these are attributes Yi will be synthesized from a previous, related 

survey, which is used to produce a conditional distribution P(Y |X). Then, Yi

is generated by 

Monte Carlo simulation from P(·|X). If desired, multiple copies of the Yi can be generated 

in order to assess replicate variability. 

Version 1.0 Y consists of four continuous attributes—spending on housing, food, transportation 

and medical care: 

Yi = (SHi, SFi, STi, SMi) = (Yi,1, Yi,2, Yi,3, Yi,4). (2) 

Extensions include categorical survey variables. 

7 Sample Module

The Sample Module determines which units in the population are in the sample S, and calculates 

their design weights {Wi

: i ∈ S}. 

Generalities WSSM can accommodate almost any design that can be simulated. It is not necessary, but may be useful, to assume that every unit in the population has positive probability 

of being selected. 

Version 1 Stratified sample based on the four X attributes, or simple random sampling. 

Extensions include more complex designs. 

8 Interviewer Characteristics Module

The Interviewer Characteristics module generates the population of interviewers for the survey. 

Generalities The population of interviewer may be either finite or infinite. Each interviewer j is 

described by a vector IC j of categorical or numerical characteristics. 

Version 1 There is one (latent) binary characteristic of interviewers, for concreteness called skill. 

The population of interviewers is infinite. Those with Skill = 1 are more effective than those 

with Skill = 0 . 

Extensions include finite populations of interviewers, of whom a fixed number have skill equal to 

1, as well as multi-dimensional characteristics. 

9 Interviewer Assignment Module

The Interviewer Assignment Module models the process by which interviewers are assigned to 

sample units. Let IAi denote the interviewer assigned to i ∈ S. 

4 
Generalities When the population of interviewers is finite, construction of the assignment is a 

constrained optimization problem. When the population is infinite, the assignment is a (possibly randomized) function from possible values of X to possible values of the interviewer 

characteristics. 

Version 1 Determined by a function P(Skill = 1|X) that depends only on the age of the head of 

household. 

Extensions are legion, including dynamic re-assignment of interviewers. 

10 Unit Nonresponse Module

The Unit Nonresponse Module simulates which sample units i ∈ S are respondents. 

Generalities Let URi = 1 if i is a respondent and URi = 0 otherwise. Then conceptually, 

generation of the URi requires a probability distribution 

P(URi = 1|Xi, Yi, Skill(IAi)), (3) 

Let R = {i ∈ S : URi = 1} denote the set of respondents. 

This module also will adjust the sample weights {Wi

: i ∈ S} for unit nonresponse, yielding 

adjusted weights {W∗

: i ∈ R}.

i

Version 1 The probability of unit nonresponse depends only on the skill of the interviewer, and is 

lower for skilled interviewers: 



p0 if Skill(IAi) = 0

P(URi = 1|Xi, Yi, Skill(IAi)) = (4)

p1 if Skill(IAi) = 1 ,

where p0 > p1, so that skilled interviewers are less likely to produce unit nonresponse. 

The W∗ will be calculated using weighting class adjustments based on the sample strata. I

Extensions are multiple—for instance, allowing dependence on Yiin (3) removes—or at least 

reduces—nonresponse bias. 

11 Responses Module

The Responses Module simulates the data—respondents’ answers D to the survey questions from 

their underlying behavior. The responses need not be “correct.” 

Generalities In general, these would given by conditional probabilities of the form 

P(Di

|Xi, Yi, Skill(IAi), URi = 1). (5) 

5 
	

	

Version 1 Recall that the behavior of interest Y consists of four numerical variables (§6). Assume 

that the four questions are of the form “How much did your household spend on [. . .] during 

the period [. . .]?” Responses are then 

Di = Yi + εi, (6) 

where εi ∼ N(0, σ2 ), with σ0

2 > σ12

. That is, the variance is higher for less skilled Skill(IAi)

interviewers. 

Extensions include item nonresponse, dependence of the variance on Y , dependence among responses, and more detailed modeling, e.g., of cognitive characteristics of questions. 



Let D = (Di,1, Di,2, Di,3, Di,4) : i ∈ R denote the set of responses. 

12 Data Quality Module

The Data Quality Module compares estimates of population characteristics derived from the data 

D with the known—because the entire population’s behavior was simulated—corresponding characteristics of the population. 

Generalities As argued in Karr et al. (2006), Cox et al. (2011) and elsewhere, the ultimate measures of data quality are measures of the quality of decisions taken on the basis of D. This 

perspective is currently too remote to be useful, and instead, measures based on the quality 



of inferences will be employed. Since (Yi,1, Yi,2, Yi,3, Yi,4) : i ∈ P is known, the comparison is then between inferences from the latter and corresponding inferences from D. 

Version 1 Two measures will be employed initially: 

• Population estimates. Comparison of the (finite population) mean μY and covariance 

matrix 6Y

2

, which can be calculated exactly, to estimates μcY and 6cY

2 derived from D

and the weights Wi

∗

. 

• Fitted tables. Consider the four “mean” contingency tables in which cells are defined 

by possible values of the population attributes X and entries are averages of the four 

behavioral attributes over the corresponding subsets of P . Each of these can be modeled using a variety of methods, including log-linear models (Bishop et al., 1975). The 

same models will be fit to these four tables and the corresponding tables derived from 

D, and the results compared as in Dobra et al. (2002). 

Extensions are many, based on ideas in various papers from NISS and others. 

6 
13 Cost Module

The Cost Module calculates the costs associated with the survey. 

Generalities Survey costs comprise both fixed and variable components, borne by multiple organizations. Some costs, especially fixed agency costs and costs considered by contractors to 

be proprietary may be difficult to elucidate (Karr and Last, 2006). 

Version 1 There will be solely a variable cost Ci for each unit i, which depends only on the skill 

level of the interviewer IAi and whether i is a unit nonrespondent: 

Ci =

⎧

⎪⎪⎨

⎪⎪⎩

c00 if Skill(IAi) = 0 and URi = 0 

c10 if Skill(IAi) = 1 and URi = 0 

.

c01 if Skill(IAi) = 0 and URi = 1 

c11 if Skill(IAi) = 0 and URi = 1 

(7) 

Extensions See Groves (2004). Ultimately, costs should also have stochastic elements. 

7 
Figure 1: Flowchart for WSSM. 

8 
References

Bishop, Y. M. M., Fienberg, S. E., and Holland, P. W. (1975). Discrete Multivariate Analysis:

Theory and Practice. MIT Press, Cambridge, MA. 

Cox, L. H., Karr, A. F., and Kinney, S. K. (2011). Risk-utility paradigms for statistical disclosure 

limitation: How to think, but not how to act (with discussion). Int. Statist. Rev., 79(2):160–199. 

Dobra, A., Fienberg, S. E., Karr, A. F., and Sanil, A. P. (2002). Software systems for tabular data 

releases. Int. J. Uncertainty, Fuzziness and Knowledge Based Systems, 10(5):529–544. 

Groves, R. M. (2004). Survey Errors and Survey Costs. Wiley, New York. 

Karr, A. F. (2011). National Institute of Statistical Sciences Configuration and Data Integration 

Technical Panel: Final report. Technical report, National Center for Education Statistics, Washington. NCES publication 2011607, available on-line at 

http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2011607. 

Karr, A. F. (2012). Discussion on statistical use of administrative data: Old and new challenges. 

Statist. Neederlandica, 66(1):80–84. 

Karr, A. F. and Last, M. (2006). Survey costs: Workshop report and white paper. 

Technical Report 161, National Institute of Statistical Sciences. Available on-line at 

http://niss.org/sites/default/files/tr161.pdf. 

Karr, A. F., Sanil, A. P., and Banks, D. L. (2006). Data quality: A statistical perspective. Statistical

Methodology, 3(2):137–173. 

9