Research and Evidence Webinar 

Series:

April, 2021

Power Analysis for Program Evaluation II: 

Basic Mechanics of Power Analysis

1
Welcome!

2

Dr. Lily Zandniapour

Research and Evaluation Manager

Office of Research and Evaluation, AmeriCorps

Power Analysis for Program Evaluation II: Basic Mechanics of Power Analysis
Introductory Remarks

3

Dr. Carrie Markovitz

Principal Research Scientist

NORC

Power Analysis for Program Evaluation II: Basic Mechanics of Power Analysis
Speaker

4

Dr. Eric Hedberg

Senior Data Scientist

NORC

Power Analysis for Program Evaluation II: Basic Mechanics of Power Analysis
Power Analysis for Program Evaluation

Level II: Basic Mechanics of Power Analysis
Overview of Courses (Three Levels)

Level III

Applied Power Analyses

Level II

Basic Mechanics of Power Analysis

Level I

Power Analysis for Program Evaluation
4/20/2021 7

•Intended for program and evaluation staff

•Introduction to power and power analysis

•Additional technical material for program and evaluation 

staff

•Type I and Type II errors

• How each error threatens a study

• Power is complement of Type II error

Level II Overview
Level II is the second part of a 

three-part series

Level I: Defining and 

Understanding Statistical Power

• Intended for All Audiences: program 

staff, funders, Program Officers, 

internal or external evaluators, and 

third-party evaluation/evidence 

reviewers

• Introduces the concept of power

• Explains why power is important for 

evaluation planning

Level III: Applied Power Analysis

• Intended for program staff or 

evaluators working with statisticians

• How to conduct power analyses

• Parameter values for a power analysis 

• Examples for common designs

• How to use the literature to inform 

specifications

• How to write-up (or read) a power 

analysis
4/20/2021 9

Power helps determine how large a sample size you need 

in order to obtain reliable evidence of program impacts.

•Power is about planning

•Power analysis sections are estimates about the future

•Power is about using evaluation resources wisely

•Power should be estimated prior to conducting a study 

(i.e., power analysis)

Review: What is power?
4/20/2021 10

Why power is important for program 

evaluations

No 

Impact 

Detected

Poorly 

Powered 

Study

True 

Impact

Reliable 

Evidence of 

Program Impact

Impact 

Detected

Well 

Powered 

Study

True 

Impact

Results of a poorly powered study (i.e., Type II Error)

Results of a well powered study 

No 

Program 

Evidence
Statistics for Impact Studies

• Impact studies want to infer the 

existence and size of the impact for 

the population based on statistics

produced from sample data

• We will not know for sure whether a 

program will be successful in the 

future or for non-study participants 

• We make an informed decision 

based on our sample from which we 

infer to the population
Type I and Type II errors

We can be right in two ways:

1. There is no impact, and we conclude so

2. There is an impact, and we conclude so

But we can be wrong in two ways:

3. There is no impact, BUT we conclude 

there is an impact (Type I error; αα)

4. There is an impact, BUT we conclude 

there is not an impact (Type II error, ββ)

What is True?

Study Conclusion

Conclude Impact

Exists

Conclude Impact 

Does Not Exist

Impact Exists

Conclusion is 

Correct

Type II Error

ββ

Impact Does Not 

Exist

Type I Error

αα

Conclusion is 

Correct

In deciding whether there is an impact, we are making an inference about 

something we do not ever know. There are four possible outcomes:
4/20/2021 13

• Probability of wrongly detecting an impact (Concluding impact exists 

when it does NOT)

• Often called αα (alpha) 

• Convention is .05

• P-value is the chance of our test-statistic (or a more extreme one) 

occurring (αα)

• If p-value is less than αα, result is considered “statistically significant”

Note: The p-value is not a measure of practical significance, so 

“statistically significant” is a statement about the probabilities of your 

conclusion, nothing more

What is Type I Error?

NOTE: Active discussion underway for lowering αα convention to .005 to combat p-hacking and other human errors (see 

Benjamin, and 71 other authors. (2018). Redefine statistical significance. Nature; Human Behaviour, 2(1), 6-10).
4/20/2021 14

• The data we obtain only represents one possible sample

• αα = .05 means that 1 out of every 20 samples will be significant, EVEN WHEN 

THERE IS NO IMPACT!

• Accepting .05 chance of Type I error means that for a set of studies about a 

program that doesn’t work, 1 in 20 will be “statistically significant” 

• That means any one study has a 1 in 20 chance of being significant no matter 

what:

• 1 in 20 even if there is no impact

• 1 in 20 regardless of sample size (large or small)

• 1 in 20 even if the data are analyzed poorly or well 

What αα means for a set of studies
4/20/2021 15

Statistical tests focus on providing information about the existence of an 

impact through “null-hypothesis significance testing” (NHST)*

1. Assume there is no impact (null hypothesis)

2. Decide on an acceptable chance of being wrong (Type I Error)

3. Analyze/summarize data by calculating a single test statistic

4. Assess how likely it is to observe these results (data): the p-value (αα)

5. Conclusion: If the likelihood of observing these results (data) is less than the 

chance of being wrong, then reject null hypothesis that there is no impact. 

*This approach is not without its issues, and other methods (Baysian) provide answers to different questions about impact. However, the bulk of evaluation research is performed under this paradigm. See, 

Gliner, J. A., Leech, N. L., & Morgan, G. A. (2002). Problems with null hypothesis significance testing (NHST): what do the textbooks say?. The Journal of Experimental Education, 71(1), 83-92.

Role of Type I error in assessing

the existence of impacts
Critical values

• We think a test is “statistically significant” 

if the p-value is < .05

• Each p-value is associated with a “critical 

value” which marks the point on the 

distribution where the probability of error 

beyond this point is considered “small”

• Two things to note:

1. As αα (Type I error) reduces, critical values get larger

2. The critical value for αα on a one- tailed test is the critical value for 

2 × αα for a two-tailed test 

• i.e., one-tailed tests increase 

Type I error

Examples of critical values for a standard-z test

αα

(Type I error)

Critical value 

(one-tail)

Critical value 

(two-tails)

0.1 1.28 1.64

0.05 1.64 1.96

0.01 2.33 2.58

0.001 3.09 3.29
4/20/2021 17

• In addition to Type I error, there is a second possible error, Type II 

error, symbol = ββ

• Type II error is the probability that you fail to conclude that an impact exists 

when it is REAL

• This requires a different probability curve

• This “alternative distribution” is centered on the expected impact

• The area of the alternative distribution before the critical value represents 

possible samples that do not yield significant results

• Power = 11 − ββ

What is Type II error? 
Power and errors

Type I and Type II error work together 

to estimate power based on possible 

outcomes from your one sample draw

• The black curve centered on 0 is the 

null distribution= possible results if 

there is no impact

• The black shade is Type I error

• The blue/purple curve is another distribution based on the expected 

result

• Blue are possible samples that are not 

significant (Type II error)

• Purple are possible samples that are 

significant (Power)
4/20/2021 19

• Power will increase as you increase Type I error (αα) (increase your 

tolerance for wrongly detecting an impact)

• More likely to have a “significant” result if you increase your Type I error

• Finding a balance: 

• Want Type I error rate to be reasonably lower (typically αα = .05)

• Want power to be high enough to minimize Type II error (ββ), so won’t fail to 

detect an effect

Power and Type I error (αα)
4/20/2021 20

Power (purple shade) for different

impacts and αα (but same sample size)

  αα = .05 αα = .0

Small

Large
4/ 21

Power (purple shade) for different

samples and αα (but same impact)

 
4/20/2021 22

• Power analysis gives researchers a chance to determine if their sample 

design is adequate to detect the expected impact before the study 

(Minimize a Type II error)

• Power analysis encourages research teams to think critically about and 

explore

• Expected Impacts of their intervention

• The design of their study sample and how it impacts the analysis plan

• Whether the analysis plan is feasible (Is there enough data?)

• Evaluation budget 

• A power analysis can help programs effectively use and target 

resources

Why conduct a power analysis for 

your study?
4/20/2021 23

• Power analyses are only informative and helpful prior to data collection

• When there is no power analysis and the results are not statistically 

significant: 

Perhaps there is no impact of the intervention 

OR

Perhaps the study was underpowered to detect the actual effect

When to conduct a power analysis

Yuan, K. H., & Maxwell, S. (2005). On the post hoc power in testing mean 

differences. Journal of educational and behavioral statistics, 30(2), 141-167.

BEFORE DATA COLLECTION!
4/20/2021 24

•Plan to collect the appropriate amount of data that 

satisfies the

•Study design

•Statistical analysis

•Expected impact

How do I design a high powered study?
4/20/2021 25

• Groups

• Treatment assignment

• Time

• Sample size

Study Design

• Matching

• Fixed or random effects

• Covariates

• Type I error

Statistical 

Analysis • Effect size

• Difference in means

• Difference in 

proportions

• Difference in events

Expected 

Impact

• Significance

• Type II Error

Power

Design, Analysis, and Impact → Power
What is a power analysis?

• A calculation that helps 

determine if a study has an 

adequate chance to detect a 

statistically significant effect (if 

one truly exists)

• Power analysis is based on the 

relationship between power, 

sample size, and effect size 

(assume two of the elements 

and calculate the third)

 
Power and impact size

Power increases with impact size

Holding sample size/design 

constant:

• Bigger impacts have more 

power

• Smaller impacts have less 

power
Power and sample size

Power increases with sample size

Holding impact constant:

• Bigger samples, depending 

on design, have more power

• Smaller samples, depending 

on design, have less power
4/20/2021 29

• Computations for power analysis are straightforward for basic designs 

and analyses, but covariates, clustering, matching, weighting, and nonresponse adjustments all increase complexity

• There are software that provide power estimates based on your study 

design and analysis plan

• Any major stats package will work with the correct formula (SPSS, Stata, SAS, 

R, Python)

• Power analysis software (PowerUP! and G*Power)

How to perform a power analysis
4/20/2021 30

A mentoring intervention program for high school age youth wants to 

determine if their program is having an impact on students’ school 

attendance. 

• POWER: Assume a conventional power level of .80 (1 − ββ, where ββ is Type II error rate)

• SIGNIFICANCE: Assume a Type I error rate for αα = .05 (two-tailed test)

• EFFECT SIZE: Estimate the sample needed to detect an effect size for the intervention 

study group based on previous studies of mentoring programs (.15)

• SAMPLE SIZE: Estimate the effect size needed for a study where the 

number of students in the program is 250 students with another 

250 students in the control group.

Example #1: Power analysis

Power

Effect Size Sample Size
4/20/2021 31

KEY QUESTION: How many students are needed to detect an effect size of .15? 

• To have an 80% chance of detecting an effect size of .15 with statistical significance at αα =

.05:

• Assume a standard power level of .80 (1 − ββ, where ββ is Type II error rate) and 

• Assume a Type I error rate for αα = .05 (two-tailed test)

• Set effect size at .15

ANSWER: Without controlling for extraneous variables, 350 students are needed 

in the treatment group and 350 students are needed in the control group, for a 

total number of 700 students.

Example #1: Power analysis (Option #1)
4/20/2021 32

Source: www.causalevaluation.org

PowerUp! Findings (Option #1)

Assumptions - Comments

MRES = MDES 0.15 Minimum Relevant Effect Size = Minimum Detectable Effect Size

Alpha Level (α) 0.05 Probability of a Type I error

Two-tailed or One-tailed Test? 2

Power (1-β) 0.80 Statistical power (1-probability of a Type II error)

P 0.50 Proportion of the sample randomized to treatment: nT / (nT + nC)

R2 0.50 Percent of variance in outcome explained by covariates

k* 1 Number of covariates used

M (Multiplier) 2.81 Automatically computed

N (Sample Size) 700 The number of individuals needed for the given MDES.
4/20/2021 33

KEY QUESTION: What effect size must the program achieve to detect a 

statistically significant effect, given the known sample sizes? 

• To have an 80% chance of a statistically significant study result at αα = .05:

• Assume a standard power level of .80 (1 − ββ, where ββ is Type II error rate) and 

• Assume a Type I error rate for αα = .05 (two-tailed test)

• Set sample sizes at 250 for program/treatment group and 250 for control/comparison 

group

ANSWER: Without controlling for extraneous variables, an 

effect size of .18 standard deviations must be produced by 

the intervention

Example #1: Power analysis (Option #2)

Power

Effect Size Sample Size
4/20/2021 34

Source: www.causalevaluation.org

PowerUp! Findings (Option #2)

Assumptions - Comments

Alpha Level (α) 0.05 Probability of a Type I error

Two-tailed or One-tailed 

Test? 2

Power (1-β) 0.80 Statistical power (1-probability of a Type II error)

P 0.50 Proportion of the sample randomized to treatment: nT / (nT + nC)

R2 0.50 Percent of variance in outcome explained by covariates

k* 1 Number of covariates used

n (Total Sample Size) 500

M (Multiplier) 2.81 Computed from T1 and T2

T1 (Precision) 1.96 Determined from alpha level, given two-tailed or one-tailed test

T2 (Power) 0.84 Determined from given power level

MDES 0.178 Minimum Detectable Effect Size
4/20/2021 35

Sample needed by effect size (no pretests)

Plot produced with GPOWER; Erdfelder, E., Faul, F., & Buchner, A. (1996). GPOWER: A general power 

analysis program. Behavior research methods, instruments, & computers, 28(1), 1-11.
4/20/2021 36

A job training intervention program for veterans wants to determine if their 

program is increasing the average wage growth for program participants. 

• POWER: Assume a conventional power level of .80 (1 − ββ, where ββ is Type II error rate)

• SIGNIFICANCE: Assume a Type I error rate for αα = .05 (two-tailed test)

• SAMPLE SIZE: Estimate the effect size needed for a study where the 

number of trainees served in a program year is 80 veterans.

Example #2: Power analysis

Power

Effect Size Sample Size

ANSWER: The effect size needed for a study group of 80 

veterans is 0.32.
4/20/2021 37

GPower

Source: 

https://www.gpower.hhu.de
4/20/2021 38

A health education program for low-income residents wants to determine 

if their program is having an impact on basic knowledge levels in health 

screenings. 

• POWER: Assume a conventional power level of .80 (1 − ββ, where ββ is Type II error rate)

• SIGNIFICANCE: Assume a Type I error rate for αα = .05 (two-tailed test)

• EFFECT SIZE: Previous studies of health education programs among low-income 

populations show an effect size of 0.12.

• SAMPLE SIZE: The number of attendees served by the program is 

500 residents. The study would identify 500 other residents who do not 

participate in the program to serve as a comparison group. 

Example #3: Power analysis

Power

Effect Size Sample Size
4/20/2021 39

KEY QUESTION: What effect size must the program achieve to detect a 

statistically significant effect, given the known sample sizes? 

• To have an 80% chance of a statistically significant study result at αα = .05:

• Assume a standard power level of .80 (1 − ββ, where ββ is Type II error rate) and 

• Assume a Type I error rate for αα = .05 (two-tailed test)

• Set sample sizes at 500 for program/treatment group and 500 for control/comparison 

group

ANSWER: Without controlling for extraneous variables, an effect size of .13 

standard deviations would need to be produced by the intervention. Therefore, 

the program would have to demonstrate a higher level of effectiveness than 

most previous studies of health education programs working with low-income 

populations in order to show a statistically significant effect.

Example #3: Power analysis
4/20/2021 40

Source: www.causalevaluation.org

PowerUp! Findings (Example #3)

Assumptions - Comments

Alpha Level (α) 0.05 Probability of a Type I error

Two-tailed or One-tailed Test? 2

Power (1-β) 0.80 Statistical power (1-probability of a Type II error)

P 0.50 Proportion of the sample randomized to treatment: nT / (nT + nC)

R2 0.50 Percent of variance in outcome explained by covariates

k* 1 Number of covariates used

n (Total Sample Size) 1000

M (Multiplier) 2.80 Computed from T1 and T2

T1 (Precision) 1.97 Determined from alpha level, given two-tailed or one-tailed test

T2 (Power) 0.84 Determined from given power level

MDES 0.125 Minimum Detectable Effect Size
Power, impact size and sample size

• Data are like a microphone 

trying to detect a sound

• Small samples (microphone) 

can only detect large impacts 

(loud sounds)

• Big samples (microphone) can 

detect impacts both large and 

small (loud sounds and pin 

drops)
4/20/2021 42

• Power is the chance to find an estimated impact “Statistically Significant”

• Power is the result of Type II Error relative to Type I Error

• Well powered studies can increase confidence in evaluation findings

• Power is the result of a process involving study design, analysis, and impacts

• Larger samples can detect smaller or larger impacts (are more sensitive) 

compared to smaller samples that can only detect larger impacts (are less 

sensitive)

• Power analysis helps you minimize Type II error

• Power analysis must occur before the study is conducted

Level II Summary
4/20/2021 43

• PowerUP! (with clustering available)

• https://www.causalevaluation.org/

• Excel and R versions

• Impacts, moderation (subgroup), and mediation analyses

• Other designs and analyses (without clustering): G*Power

• https://www.gpower.hhu.de

• Regression/Correlation, ANOVA, Probability Models

Free power analysis software
Resources

• Aberson, C. L. (2011). Applied power analysis for the behavioral sciences. Routledge.

• Cohen, J. (2013). Statistical power analysis for the behavioral sciences. Academic press.

• Hedberg, E. C. (2017). Introduction to Power 

Analysis: Two-group Studies (Vol. 176). Sage Publications.

• Kraemer, H. C., & Blasey, C. (2015). How 

many subjects?: Statistical power analysis in 

research. Sage Publications.

• Liu, X. S. (2013). Statistical power analysis for the social and behavioral sciences: Basic and 

advanced techniques. Routledge.

• Many more papers and blogs exist, likely 

specific to your outcome
Closing Remarks

45

Dr. Carrie Markovitz

Principal Research Scientist

NORC

Power Analysis for Program Evaluation II: Basic Mechanics of Power Analysis