# Presentation: 785714

## E. Michael Foster
School of Public Health
UNC-CH

- E. Michael Foster
- School of Public Health
- UNC-CH

- These comments are my own, but I would like to thank Len Bickman.

## We have relatively little meaningful evidence about intervention effectiveness (and cost-effectiveness).

- Much (?) of the evidence is of poor quality, largely because of poor methodology, and evidence on failures is buried.

- Foster - Intervening Early

## These problems are not going to get a bit better as we move to effectiveness trials

**These problems are not going to get a bit better as we move to effectiveness trials**
**The problems will get worse**
- Examples:
- Dosage
- Condition switching (WLC)
- Fidelity not randomly assigned
- Differential impact
- Craziness with administrative data

- Foster - Intervening Early

## The FDA would not consider much of our research as credible evidence of efficacy

- Outsiders often judge published evidence as poor (case study: MST)
- The published literature is not representative of overall findings
- Very few of these findings would replicate
- The solution: A Culture of Open Inquiry

- Foster - Intervening Early

## “We spent so much money, 

- Foster - Intervening Early

- Someone has their darned foot on the scale!

- “We spent so much money,
- there must be effects

## “We get effects if we include an interaction with time-squared and we leave out the main effect of that term.”

- “We get effects if we include an interaction with time-squared and we leave out the main effect of that term.”
- When I talk to other methodologists, they have burst out laughing, saving “I’ve been told that, too!”

## FDA guidelines indicate that

- “For each clinical trial contributing to a marketing application, all important details of its design and conduct and the principal features of its proposed statistical analysis should be clearly specified in a protocol written **before the trial begins**.”

- Foster - Intervening Early

- ”The extent to which the procedures in the protocol are followed and the primary analysis is planned a priori will contribute to the degree of confidence in the final results and conclusions of the trial.”

- Foster - Intervening Early

## “However, in contrast to confirmatory trials, [the] objectives [of exploratory trials] may not always lead to simple tests of predefined hypotheses. In addition, exploratory trials may sometimes require a more flexible approach to design so that changes can be made in response to accumulating results. Such trials cannot be the basis of the formal proof of efficacy, although they may contribute to the total body of relevant evidence. “

- “However, in contrast to confirmatory trials, [the] objectives [of exploratory trials] may not always lead to simple tests of predefined hypotheses. In addition, exploratory trials may sometimes require a more flexible approach to design so that changes can be made in response to accumulating results. **Such trials cannot be the basis of the formal proof of efficacy**, although they may contribute to the total body of relevant evidence. “

- Foster - Intervening Early

- “Only results from analyses envisaged in the protocol (including amendments) can be regarded as confirmatory.”

- Foster - Intervening Early

## Methodological quality

*Methodological quality*
- Sloppy or least a lack of detail
- Technical problems (e.g., hazard modeling and length of followup period)
- Nearly all studies by developers

- Foster - Intervening Early

## “There is inconclusive evidence of the effectiveness of MST compared with other interventions with youth. There is no evidence that MST has harmful effects.”

- “There is inconclusive evidence of the effectiveness of MST compared with other interventions with youth. There is no evidence that MST has harmful effects.”
- Littell JH, Popa M, Forsythe B. Multisystemic Therapy for social, emotional, and behavioral problems in youth aged 10-17. *Cochrane Database of Systematic Reviews 2005, Issue 4. Art. No.: CD004797. DOI: 10.1002/14651858.CD004797.pub4.*

- Foster - Intervening Early

## The smaller the studies conducted in a scientific field, the less likely the research findings are to be true.

- The smaller the effect sizes in a scientific field, the less likely the research findings are to be true.
- The greater the number and the lesser the selection of tested relationships in a scientific field, the less likely the research findings are to be true.

- Foster - Intervening Early

## The greater the flexibility in designs, definitions, outcomes, and analytical modes in a scientific field, the less likely the research findings are to be true.

- The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true.

- Foster - Intervening Early

**Notes:**
First point: really ties into my corrollary—more nuanced and subtle, the less likely the findings are to be true.

 All different measures – some of this is endemic to mental health.  We don’t have physical outcomes as first-line outcomes.

## Much of prevention research meets all of these criteria.  These criteria raise the possibility that my experiences reflect widespread problems.

- The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true.
- Ioannidis JPA (2005) Why most published research findings are false. PLoS Med 2(8): e124.

- Foster - Intervening Early

*Much of prevention research meets all of these criteria.  These criteria raise the possibility that my experiences reflect widespread problems.*

## In general, what we need is an open process, informed by relevant technical expertise.

- Foster - Intervening Early

## Provides essential detail

- Provides opportunity to examine robustness and representativeness
- Establishes accountability

- Foster - Intervening Early

## Protect confidentiality

- Widespread availability (not three collaborators)
- Three levels
- Data used in published analyses
- Data used to execute analysis plan in grant
- All of the taxpayer’s data

- Foster - Intervening Early

## Match the design of the study 

- Match the design of the study
- (as implemented, not hoped for)
- Adjusts for multiple testing
- Includes cross-validation
- When model specification is data driven
- When unexpected findings appear
- Deals seriously with the possibility of failure at the start of the study
*Good data quality *(often “amateurish”)

- Foster - Intervening Early

## The basic analysis in the analysis plan should be reported fairly quickly.

- Determined the power
- Clearly set before one examines the data
- If you tried it both ways, report both ways.
- May include multiple outcomes but all are reported

- Foster - Intervening Early

## The good news is that there are solutions available for dealing with many challenging problems, but

- Quantitative training in psychology is inferior to that in other social sciences* (I can document this)
- More biostatisticians and social scientists from other fields are needed

- Foster - Intervening Early

## Be very skeptical if you are reviewing a paper from a project that reports

- results from five years earlier
- transformed variables but not the ones in the original scale.
- terribly nuanced  findings
*Ask yourself: Is nuance informative?*

- Foster - Intervening Early

## Skeptics needed; True believers, not so much.

- Researchers from the same background share the same weaknesses
*The skills that make one good at developing interventions are very likely not the ones that are good for evaluating them.*
- Senior methodologist collaborators

- Foster - Intervening Early

## Data sharing

*Data sharing*
- Incredible Years
- Abcederian
- New Hope
*Mix of Investigators*
*MTA*

- Foster - Intervening Early

## Accountability by NIH—real reports on what’s in the original analysis plan or the $$$ stops

- Clinical trials data center
- More money for analyses and analysts
- Journals can really help by
- Publishing null findings
- Requiring data sharing

- Foster - Intervening Early

## Foster - Intervening Early

## Self interest—hoarding data is indeed good for one’s career

- Self interest—testing new interventions  is more rewarded than evaluating existing ones
- Technical expertise (e.g., data quality and analysis)
- Scientific interest
- (ex: sample representativeness)

- Foster - Intervening Early

## NIH would need more resources for monitoring findings and progress

- Data privacy (HIPAA)

- Foster - Intervening Early

## “I put my heart and soul into collecting data”

- “I put my heart and soul into collecting data”
- “If I start sharing my data, I’m going to have to document the data”
- “What?  Should I just let other people write half-ass papers with my data?”
- “Other people don’t understand my product well enough to analyze the data”
- “I’m doing what I’m supposed to do”

- Foster - Intervening Early

## “You can have my data but I ain’t giving you any money” or “I’d love to have more biostat input—but have you priced one lately!!!”

- “You can have my data but I ain’t giving you any money” or “I’d love to have more biostat input—but have you priced one lately!!!”
- “How in the world am I supposed to capture the complexity of what we’re doing in advance in an analysis plan”
- “What we do is just more complicated than drug trials.  You just don’t get it.”

- Foster - Intervening Early