Research and Evidence Webinar Series:

March 30, 2021

Power Analysis for Program Evaluation I: 

An Introduction

1
We’ll get started in just a 

couple of minutes.

Please be advised that 

there is no dial-in for this 

webinar; all audio is 

provided directly through 

the Adobe Connect 

platform.

Welcome!

Power Analysis for Program Evaluation I: An Introduction

Mar-21 | 2
Welcome!

Power Analysis for Program Evaluation I: An Introduction

3

Dr. Lily Zandniapour

Research and Evaluation Manager

Office of Research and Evaluation, AmeriCorps

Mar-21 |
Introductory Remarks

Power Analysis for Program Evaluation I: An Introduction

4

Dr. Carrie Markovitz

Principal Research Scientist

NORC

Mar-21 |
Speaker

Power Analysis for Program Evaluation I: An Introduction

5

Dr. Eric Hedberg

Senior Data Scientist

NORC

Mar-21 |
Power Analysis for Program Evaluation

Level I: Defining and Understanding Statistical Power
Overview of Courses (Three Levels)

Level III

Applied Power Analyses

Level II

Basic Mechanics of Power Analysis

Level I

Power Analysis for Program Evaluation
4/20/2021 8

•Intended for All Audiences: program staff, funders, 

internal or external evaluators, and third-party 

evaluation/evidence reviewers

•Introduces the concept of power

•Explains why power is important for evaluation planning

• Helps support appropriate use of resources for evaluation by 

providing information required to produce an efficient sample (i.e., a 

sample may be sufficient rather than a census)

Level I Overview
Beyond Level I...

Level II: Basic Mechanics of Power 

Analysis

• Intended for program and evaluation 

staff

• Type I and Type II errors

• How each error threatens a study

• Power is complement of Type II error

Level III: Applied Power Analysis

• Intended for program staff or 

evaluators working with statisticians

• How to conduct power analyses

• Parameter values for a power analysis 

• Examples for common designs

• How to use the literature to inform 

specifications

• How to write-up (or read) a power 

analysis
4/20/2021 10

• EXAMPLE: You want to assess the impact of your program on program 

beneficiaries, but tracking the circumstances of all of your beneficiaries is 

cost prohibitive. If instead you tracked outcomes for some, but not all of your 

program participants, how many participants will you need to conduct a 

credible study? 

• In some cases, you will need to conduct a CENSUS (population) and include 

every program participant in your study. But sometimes it is possible to use a 

SAMPLE of program participants and still capture evidence of impacts that 

can be GENERALIZED to the wider population (all program beneficiaries).

Power helps determine how large a sample size you need in 

order to obtain reliable evidence of program impacts.

Why is power important?
Statistics for Impact Studies

• Impact studies want to infer the 

existence and size of the impact for 

the population based on statistics

produced from sample data

• We will not know for sure whether a 

program will be successful in the 

future or for non-study participants 

• We make an informed decision 

based on our sample from which we 

infer to the population
Statistics for Impact Studies

• Impact studies want to infer the 

existence and size of the impact for 

the population based on statistics

produced from sample data

• We will not know for sure whether a 

program will be successful in the 

future or for non-study participants 

• We make an informed decision 

based on our sample from which we 

infer to the population
Studies have few of many samples

Your study is only one of many 

possible samples

• Your margin of error is a range of 

possible outcomes from 95 percent 

of many other samples

• Given a true impact, will your sample 

happen to produce a result that is 

statistically significant?

You only get one jelly bean...will it be 

grape?
4/20/2021 14

•H0 Null Hypothesis– No difference between study groups

•H1 Alternative Hypothesis– Difference between study 

groups (program and control/comparison)

In hypothesis testing we hope to be able to reject the null 

hypothesis of no differences and accept the alternative 

hypothesis that there is a difference between study groups by 

showing a very low chance that the null hypothesis is correct. 

Hypothesis Testing
4/20/2021 15

The level of power calculated for your study is the 

probability of rejecting the null hypothesis (i.e., claim a program impact)

•Power is about planning

•Power analysis sections are estimates about the future

•Power is about using evaluation resources wisely

•Power should be estimated prior to conducting a study 

(i.e., power analysis)

What is power?
4/20/2021 16

High powered studies have many benefits:

•For grantees: Helps increase the chance of detecting true 

effects (obtaining credible evidence of program impact)

•For funders: Helps ensure you are resourcing an 

evaluation that will produce reliable evidence of impacts

•For researchers: Helps increase reliability and trust in 

study findings 

Why is power important?
4/20/2021 17

Why power is important for program 

evaluations

True 

Impact

Results of a poorly powered study (i.e., Type II Error)

Results of a well powered study 
4/20/2021 18

Why power is important for program 

evaluations

Poorly 

Powered 

Study

True 

Impact

Results of a poorly powered study (i.e., Type II Error)

Results of a well powered study 
4/20/2021 19

Why power is important for program 

evaluations

No 

Impact 

Detected

Poorly 

Powered 

Study

True 

Impact

Results of a poorly powered study (i.e., Type II Error)

Results of a well powered study 
4/20/2021 20

Why power is important for program 

evaluations

No 

Impact 

Detected

Poorly 

Powered 

Study

True 

Impact

Results of a poorly powered study (i.e., Type II Error)

Results of a well powered study 

No 

Program 

Evidence
4/20/2021 21

Why power is important for program 

evaluations

No

Impact 

Detected

Poorly 

Powered 

Study

True 

Impact

Reliable 

Evidence of 

Program Impact

Impact 

Detected

Well 

Powered 

Study

True 

Impact

Results of a poorly powered study (i.e., Type II Error)

Results of a well powered study 

No 

Program 

Evidence
4/20/2021 22

The probability of “statistically significant results” is based 

on

•The design of the study

•The amount/structure of the data 

•Generally speaking: more information=more power

•How the data are analyzed

•The size of the intervention/program’s impact

•The selected significance level

What affects power?
What is a power analysis?

• A calculation that helps 

determine if a study has an 

adequate chance to detect a 

statistically significant effect (if 

one truly exists)

• Power analysis is based on the 

relationship between power, 

sample size, and effect size 

(assume two of the elements 

and calculate the third)

 
Power and impact size

Power increases with impact size

Holding sample size/design 

constant:

• Bigger impacts have more 

power

• Smaller impacts have less 

power
Power and sample size

Power increases with sample 

size

Holding impact constant:

• Bigger samples, depending 

on design, have more power

• Smaller samples, depending 

on design, have less power
Power, impact size and sample size

• Data are like a microphone 

trying to detect a sound

• Small samples (microphone) 

can only detect large impacts 

(loud sounds)

• Big samples (microphone) can 

detect impacts both large and 

small (loud sounds and pin 

drops)
4/20/2021 27

• Power analysis gives researchers a chance to determine if their sample 

design is adequate to detect the expected impact before the study

• Power analysis encourages research teams to think critically about and 

explore

• Expected Impacts of their intervention

• The design of their study sample and how it impacts the analysis plan

• Whether the analysis plan is feasible (Is there enough data?)

• Evaluation budget (ensure using resources wisely)

• A power analysis can help programs effectively use and target 

resources

Why conduct a power analysis for 

your study?
4/20/2021 28

• KEY QUESTION #1: How many students are needed to detect an effect size of X?

ANSWER: Y students are needed in the treatment group and Y students are 

needed in the control group.

• KEY QUESTION #2: What effect size must the program achieve to detect a 

statistically significant effect, given the known sample sizes?

ANSWER: An effect size of X standard deviations must be produced by 

the intervention.

What does a power analysis tell you?
4/20/2021 29

• Power analyses are only informative and helpful prior to data collection

• When there is no power analysis and the results are not statistically 

significant: 

Perhaps there is no impact of the intervention 

OR

Perhaps the study was underpowered to detect the actual effect

When to conduct a power analysis

Yuan, K. H., & Maxwell, S. (2005). On the post hoc power in testing mean 

differences. Journal of educational and behavioral statistics, 30(2), 141-167.

BEFORE DATA COLLECTION!
4/20/2021 30

• Power is the chance to find an estimated impact that is “statistically 

significant”

• Poorly powered studies can reduce the chance of detecting the true effect of 

programs

• Power is the result of a process involving study design, analysis, and 

expected impacts

• For every design and analysis, there is a unique power analysis procedure

• Larger samples can detect smaller or larger impacts (are more sensitive) 

compared to smaller samples that can only detect larger impacts (are less 

sensitive)

• Power analysis must occur before data collection in a study

Level I Summary
Resources

• Aberson, C. L. (2011). Applied power analysis for the behavioral sciences. Routledge.

• Cohen, J. (2013). Statistical power analysis for the behavioral sciences. Academic press.

• Hedberg, E. C. (2017). Introduction to Power 

Analysis: Two-group Studies (Vol. 176). Sage Publications.

• Kraemer, H. C., & Blasey, C. (2015). How 

many subjects?: Statistical power analysis in 

research. Sage Publications.

• Liu, X. S. (2013). Statistical power analysis for the social and behavioral sciences: Basic and 

advanced techniques. Routledge.

• Many more papers and blogs exist, likely 

specific to your outcome
Closing Remarks

Power Analysis for Program Evaluation I: An Introduction

32

Dr. Carrie Markovitz

Principal Research Scientist

NORC

Mar-21 |
Questions?

Dr. Andrea Robles

Research and Evaluation Manager, 

Office of Research and Evaluation, AmeriCorps

Q&A

Power Analysis for Program Evaluation I: An Introduction

33

Visit nationalservice.gov/ORE for past webinars, research and evaluation 

resources, reports, and more!

Mar-21 |