# Presentation: 892956

## Evaluation of Non-Randomized Studies

- S. Stanley Young
- National Institute of Statistical Sciences
- Robert L. Obenchain
- Eli Lilly & Company
- FDA 21Sept2006

## UPSIDE: the uniform principle for sharing integral data andmaterials expeditiously

- Committee on Responsibilities of
- Authorship in the Biological Sciences
- National Research Council
- National Academy of Science
- http://www.nap.edu/catalog/10613.html
- 2003

## Community Standards

- More specifically, the act of publishing is a *quid pro quo*
- in which authors receive credit and acknowledgment
- in exchange for disclosure of their scientific findings.
- An author’s obligation is not only to release data and materials
- to enable others to verify or replicate published findings
- (as journals already implicitly or explicitly require) but
- also to provide them in a form on which other scientists
- can build with further research.
- ...whatever is necessary to support the major claims of the
- paper and would enable one skilled in the art to verify or
- replicate the claims.

## Materials and Methods

- Most journals today explicitly or implicitly
- require that authors provide enough detail
- about their materials  and methods to allow
- a qualified reader to verify, replicate, or refute
- the findings reported in a paper.

## Further Comment

- It is unacceptable to require collaboration or
- coauthorship as a condition of providing a
- published material, because that requirement
- can inhibit a scientist from publishing findings
- that are contrary to the provider’s published
- conclusions.

## Mangano et al. Claim

- Claim: “The association between aprotinin and serious
- end-organ damage indicates that continued use is not prudent.”
-                              This is a serious claim.
-            The data structure and analysis is quite complex.
- Mangano et al. have refused multiple requests for the data set.
- Mangano et al. give insufficient analysis details for their
-    analysis results to be replicated.

## External Consideration

- It is error only, and not truth, that shrinks from inquiry.
-                                                    Thomas Paine, 1737-1809

## Analysis of Complex Data Sets

- There are multiple response questions.
- Responses can be combined in numerous ways.
- There are infinitely many analysis options
-        with hundreds of covariates.
- 4.    P-values can be moved many orders of magnitude.
- 5.    In exploratory analysis, p-values and RR have no meaning.

## Patient Allocation

**Notes:**
The control patients are so different from the treated patients that they should be removed from the analysis.

Essentially all the high risk and very high risk patients were assigned by physicians to the aprotinin treatment group. 

As there are no comparators to this group at this risk level, the group should be removed from the analysis.
The three treatments can now be compared using blocking or propensity score analysis, the 2x3 or 3x3 grid.

Any analysis that summarizes effects down the treatment columns without removal of the “Very high risk” aprotinin cell will be gravely misleading.

## External Consideration

- 80% of highly cited non-randomized studies
-  were contradicted
-                        versus
- only 23% of randomized controlled trials
- were contradicted.

- Contradicted and Initially Stronger Effects
-         in Highly Cited Clinical Research
-                                    John Ioannidis, JAMA (2005)

## Summary

- Until the data are made available to others,
-        and a proper analysis computed, these claims
-        should be ignored.
- The statistical analysis is seriously flawed.
-       Where were the NEJM editors and referees?