Assessing How Well Students in Economics Experiments 

Model Decision-Making for Non-Student Populations

Rosch, Fooks, Hellerstein, Lynch, Messer, Raszap Skorbiansky, Weigel, Zeballos

2018 Fed. Committee on Stat. Methodology Research and Policy Conference

Washington, DC

7 March 2018

Disclaimer: The views expressed are those of the authors and should not be attributed to the Economic Research Service or USDA. 
2

Outline

1. Experiments to inform policy-making

2. Selection of subjects

a. Challenges of experiments with farmers

b. Potential for using model populations

3. Research agenda on students as models for non-student 

populations in economic experiments

4. Measuring “representativeness”
3

Benefits of Using Experimental Data

• Complement survey and administrative data

• Identify causal responses to policy changes

• Isolate behavioral mechanisms, such as: 

– How insurance demand changes with differences in risk exposure

– How auction bidding strategies change with differences in the 

numbers of competing bidders
4

Using Experiments to Inform Policy-Making

• Experimental evidence can be useful in answering many policyrelevant questions: 

– Test new auction mechanisms for the Conservation Reserve Program

– Measure voter outreach methods for county commissioner elections

– Measure the impact of subsidies for the Federal Crop Insurance 

Program

– Assess consumer response to new food labeling information 
5

Three Key Tradeoffs in Experiments

• Construct validity:

Does the experiment adequately replicate the real-world context such that 

behaviors within the experiment reflect behaviors in the real world?

• Internal validity:

Does the experiment adequately control for potential confounding factors?

• External validity:

Do the results generated by the experimental subjects represent other 

populations? 
6

Preferred Experimental Subjects 

• Policy-relevant populations, such as:

– Conservation programs => eligible landowners

– Crop insurance => eligible farmers

– County commissioner elections => rural voters

– Nutrition labels => consumers

• Just as with surveys, collecting experimental data from these 

populations requires costs to the government and imposes 

burdens on participants
7

Example: Challenges of Recruiting Farmers for 

Experiments

• Logistically difficult to recruit farmers

• Limited availability for experiments 

• Require high participation fees

• In fact, relatively few published economic experiments used 

farmer subjects: 

– 73 studies used farmer subjects 

– 9 studies sampled US farmers 

– 6 of the 9 studies with US farmers had less than 70 subjects
8

What About Using a Model Population?

• Examples of model populations: 

– Animals for testing of new medications

– Airplane prototypes used for wind tunnel testing

• College students are used by academic researchers to model nonstudent populations: 

– Businesses responses to new emissions trading platforms (Cason and Plott, 

1996)

– Stock traders responses to new information (Lei et al, 2001)

– Public officials incentives to behave corruptly (Drugov et al, 2014)
9

Advantages to Using Students as Experimental 

Subjects

• Farmers:

– Policy-relevant population

– Understand context

– Difficult to recruit

– Require high participation fees

– Limited time for the 

experiment

• College Students:

– Not necessarily policy-relevant

– Less familiar with context

– Easy to recruit

– Low participation fees

– More time for experiment

But what can experiments with student samples tell 

us about farmer behaviors?
10

Comparing Experiments with Student and Farmer 

Subjects

• 8 published studies directly replicated with students and farmer 

samples 

• Studies sample farmers from 4 different countries, involve many 

different methods, and investigate widely different topics

• Mixed outcomes:

– 2 studies found different results between students and farmers for all tests

– 1 study found no differences between students and farmers for all tests

– 2 studies found some differences between students and farmers

– 3 studies did not compare results between students and farmers
11

No Answers Yet...

• Insufficient data to assess students’ abilities to model farmers’ 

decision-making

– Average behavior of farmers

– Heterogeneity of farmer behaviors
12

... And What about Students’ Abilities to Model 

Other Populations of Interest? ...
13

... More Research is Needed!

• Questions about external validity of model populations can only be 

answered with more experiments

• Use clues from studies comparing students and non-students in general to 

prioritize a research agenda:

– Meta-analysis found 9 out of 13 experiments have “similar” results for student 

and non-student subjects (Frechette, 2011)

– One study found differences between student and non-student subjects vary by 

type of economic game used (Belot et al, 2015)

– Another study found differences in trust attitudes for student and non-student 

subjects correlates with differences in demographic characteristics (Falk et al, 

2013)
14

Next Research Goals

• Measure how representative are students’ and non-students’ 

experimental results 

• Characterize the conditions which influence how well students 

model decision-making for non-student populations

1. Are there any types of experiments where student results are more 

likely to be similar to results from non-student subjects?

2. Are there characteristics of student subjects that improve the 

generalizability of their results to non-student populations? 
15

Several Challenges to Resolve 

1. Inclusion and exclusion criteria for studies

2. Categorizing types of experiments

3. Defining “representativeness”

4. Selection of demographic characteristics
16

Defining Representativeness

1. Strongly representative model populations

– average treatment effect for students has the same sign and a similar 

magnitude to the average treatment effect for non-students

2. Partially representative model populations

– students and non-students have treatment effects that correlate with a 

population characteristic in a similar way

3. Unrepresentative model populations

– average treatment effect for students is not similar to the average 

treatment effect for non-students, and correlations between observable 

demographic characteristics and treatment effects are not similar
17

Defining Similar

• Non-statistical test: 

– Same order of magnitude 

• Tests of statistical significance:

– Paired tests (e.g. paired t-test for independent samples)

– Confidence intervals of bootstrapped differences

– Convolutions approach (Poe et al, 1994)

– Other?
18

Current Progress

• Still in the early stages of preparing first drafts

• Expected outputs:

– Report for Office of Management and Budget

– Academic version for publication

• Feedback on assumptions and approaches is most welcome!
For more information please contact: 

Stephanie Rosch

stephanie.rosch@ers.usda.gov