# Presentation: 226781

## Guidelines for Multiple Testing in Impact EvaluationsPeter Z. Schochet

**June 2008**

## What Is the Problem?

**What Is the Problem?**

**Multiple hypothesis tests are often conducted in impact studies**
**Outcomes**
**Subgroups **
**Treatment groups**** **
**Standard testing methods could yield:**
*** *****Spurious**** significant impacts **
** ****Incorrect**** policy conclusions  **

## Overview of Presentation 

**Overview of Presentation **

**Background  **
**Suggested testing guidelines**

## Background

## Assume a Classical Hypothesis Testing Framework

**Assume a **_**Classical**_** Hypothesis Testing Framework**

**True impacts are fixed for the study population**
**Test H****0j****: *****Impact******j****** = 0*****   **
**Reject H****0j ****if *****p*****-value of *****t*****-test ****< ********=.05**
**Chance of finding a spurious impact is 5 percent for ****each**** test alone**

## But If Tests Are Considered Together and No True Impacts... 

**But If Tests Are Considered **_**Together**_** and **_**No**_** True Impacts...**** **

**                                                       ****Probability ********1 *****t*****-test**_** **_
_**Number of Tests**_**a****              **_**Is Statistically Significant**__**  **_
**           ****1                                                 .05**
**           ****5                                                 .23**
**         ****10                                                 .40 **
**         ****20                                                 .64**
**         ****50                                                 .92**
**a****Assumes independent tests**

## Impact Findings Can Be Misrepresented

**Impact Findings Can Be Misrepresented**

**Publishing bias**
**A focus on “stars”****   **

## Adjustment Procedures Lower          Levels for Individual Tests

**Adjustment Procedures Lower         *********** *****Levels**** for Individual Tests**

**Methods control the  ****“**_**combined**_**”**** error rate **
**Many available methods:**
**Bonferroni: Compare *****p*****-values to (.05 / # of tests) **
**Fisher’s LSD, Holm (1979), Sidak (1967),           Scheffe (1959), Hochberg (1988), Rom (1990), Tukey (1953)**
**Resampling methods (Westfall and Young 1993)**
**Benjamini-Hochberg (1995)**

## These Methods Reduce Statistical Power: The Chances of Finding Real Effects 

**These Methods Reduce Statistical Power: The Chances of Finding **_**Real**_** ****Effects**** **

**                                    **_**Simulated Statistical Power**__**a**_
_**                                      **_
_**Number of Tests **_**          **_**Unadjusted**_**    **_**Bonferroni**__**  **_
**           **
**           ****5                                  .80                  .59**
**         ****10                                  .80                  .50 **
**         ****20                                  .80                  .41**
**         ****50                                  .80                  .31**
**a**** Assumes 1,000 treatments and 1,000 controls, 20 percent of all null**
**  ****hypotheses are true, and independent tests **

## Big Debate on Whether To Use Adjustment Procedures

**Big Debate on Whether To Use Adjustment Procedures**

**What is the proper balance between Type I and Type II    errors?**

##    To Adjust  or Not To Adjust?

## February, July, December 2007 Advisory Panel Meetings Held at IES 

**February, July, December 2007 Advisory Panel Meetings Held at IES**** **

**    **_**Chairs:**_
** **
**Phoebe Cottingham, IES**
**Rob Hollister, Swarthmore**
**Rebecca Maynard, U. of PA   **

**    **_**Participants**_**:**** **
**Steve Bell, Abt**
**Howard Bloom, MDRC **
**John Burghardt, MPR**
**Mark Dynarski, MPR**
**Andrew Gelman, Columbia**
**David Judkins, Westat**
**Jeff Kling, Brookings**
**David Myers, AIR**
**Larry Orr, Abt**
**Peter Schochet, MPR**

## Views Expressed Here May Not Represent Those of allPanel Members 

## Basic Testing Principles 

## The Problem Should Not               Be Ignored

**The Problem Should Not               Be Ignored**

**Erroneous conclusions can result otherwise**
**But need to balance Type I and II errors**

## Limiting the Number of Outcomes and Subgroups Can Help 

**Limiting the Number of Outcomes and Subgroups Can Help**** **

**But not always possible or desirable**
**Need flexible strategy for confirmatory **_***and***_** exploratory analyses**

## Problem Should Be Addressed by First Structuring the Data

**Problem Should Be Addressed by First ****Structuring**** the Data**

**Structure will depend on the research questions **
**Adjustments should **_***not***_** be conducted blindly across all contrasts**

## Suggested Testing Guidelines 

## The Plan Must Be Specified  Up Front

**The Plan Must Be Specified  Up Front**

**Study protocols should specify:**
**Data structure**
**Confirmatory analyses**
**Testing strategy**

## Delineate Separate              Outcome Domains 

**Delineate Separate              Outcome Domains **

**Based on a conceptual framework**
**Represent key clusters of constructs**
**Domain ****“items”**** are likely to measure the same underlying trait (have high correlations)**
**Test scores**
**Teacher practices**
**School attendance**

## Testing Strategy: Both Confirmatory and Exploratory Components

**Testing Strategy: **_**Both**_** Confirmatory and Exploratory Components**

_**Confirmatory component**_
**Addresses central study hypotheses**
_**Must**_** adjust for multiple comparisons**
_**Exploratory component**_
**Identify impacts or relationships for future study**
**Findings should be regarded as preliminary   **

## Confirmatory Analysis Has Two Potential Parts

**Confirmatory Analysis Has Two Potential Parts**

**Domain-specific analysis**
**Between-domain analysis **

## Domain-Specific Analysis 

## Test Impacts for Outcomes as a Group

**Test Impacts for Outcomes ****as a **_**Group**_

**Create a **_**composite**_** domain outcome**
**Weighted average of standardized outcomes**
**Equal weights**
**Expert judgment**
**Predictive validity weights**
**Factor analysis weights**
**MANOVA not recommended**
**Conduct a *****t*****-test on the composite**

## What About Tests for Individual Domain Outcomes?

**What About Tests for ****Individual ****Domain Outcomes?**

**If impact on composite is significant**
**Test impacts for individual domain outcomes ****without**** multiplicity corrections **
**Use only for interpretation**
**If impact on composite is not significant**
**Further tests are not warranted**

## Between-Domain Analysis 

## Applicable If Studies Require Summative Evidence of Impacts

**Applicable If Studies Require Summative Evidence of Impacts**

**Constructing “unified” composites may not make sense**
**Domains are likely to measure different latent traits**
**Test domain composites ****individually**** using adjustment procedures**

## Testing Strategy Will Depend on the Research Questions

**Testing Strategy Will Depend on the Research Questions**

**Are impacts significant in ****all**** domains? **
**No adjustments are needed**
**Are impacts significant in ****any***** *****domain? **
**Adjustments are needed **

## Other Situations That Require Multiplicity Corrections

## Designs With Multiple           Treatment Groups

**Designs With Multiple           Treatment Groups**

**Need stringent evidence to conclude that some treatments are preferred over others**
**Apply Tukey-Kramer, Dunnett, Orthogonal Contrasts or resampling methods to domain composites**

## Subgroup Analyses That Are Part of the Confirmatory Analysis    

**Subgroup Analyses That Are Part of the Confirmatory Analysis    **

**Limit to a few educationally meaningful subgroups**
**Justify subgroups**** **
**Stratify by subgroup in sampling**
**Conduct *****F*****-tests for***** *****differences**** across subgroup impacts**

## Exploratory Analyses? 

**Exploratory Analyses? **

**Two schools of thought**
**The use of corrections does **_***not***_** make exploratory findings confirmatory **

## Statistical Power  

**Statistical Power  **

**Studies must be designed to have sufficient statistical power for ****all ****confirmatory analyses**
**Includes subgroup analyses**

## Reporting  

**Reporting  **

_**Qualify**_** confirmatory and exploratory analysis findings in reports**
**No one way to present adjusted and unadjusted *****p*****-values**
**Confidence intervals could be helpful**
**Confirmatory analysis results should be ****emphasized**** in the executive summary  ****   **

## Testing Approach Summary

**Testing Approach Summary**

**Specify plan in study protocols**
**Structure the data**
**Delineate outcome domains**
**Confirmatory analysis**
**Within and between domains**
**Exploratory analysis**
**Qualify findings appropriately**