# Presentation: 516814

## Chaos in the brickyard:Translational research in 2007

**Chaos in the brickyard:****Translational research in 2007**

- David F. Ransohoff MDDepartments of Medicine and EpidemiologyUniversity of North Carolina at Chapel Hill
- ‘Medicine: Mind the Gap’NIH seminar series - November 5, 2007

## Slide 2

## Chaos in the brickyard                     Forscher BK. Science 1963;142:339

- Once upon a time, among the activities and occupations of man there was an activity called scientific research and the performers of this activity were called scientists. In reality, however, these men were builders who constructed edifices, called explanations or laws, by assembling bricks, called facts.

## When the bricks were sound and were assembled properly, the edifice was useful and durable and brought pleasure, and sometimes reward, to the builder. 

## If the bricks were faulty or if they were assembled badly, the edifice would crumble, and this kind of disaster could be very dangerous to innocent users of the edifice as well as to the builder who sometimes was destroyed by the collapse.

## And then it came to pass that a misunderstanding spread among the brickmakers.... The brickmakers became obsessed with the making of bricks. When reminded that the ultimate goal was edifices, not bricks, they replied that, if enough bricks were available, the builders would be able to select what was necessary and still continue to construct edifices.

- It became difficult to complete a useful edifice because, as soon as the foundations were discernible, they were buried under an avalanche of random bricks.

## And, saddest of all, sometimes no effort was made even to maintain the distinction between a pile of bricks and a true edifice.

## Topics discussed by Forscher

- 	1. bricks2. edifices3. builders vs brickmakers 4. training

## Topics discussed by Forscher

- 	1. **bricks**2. edifices3. builders vs brickmakers 4. training
- “If the bricks [facts] were faulty...the edifice would crumble....”

## Translational Research

**Type 1: Apply lab discoveries to studies in humans (bench to bedside)**
- Type 2: Adopt practices in community
-                    				     from RFA-RM-06-002:
- Institutional Clinical and Translational Science Award (CTSA)

## How do we know: Is brick faulty? 

- Challenge: Translational research involves different disciplines; ‘rules of evidence’ (to decide ‘when is brick strong’) may vary in different fields.
- 		Nat Rev Cancer 2004; 4:309-14		Nat Rev Cancer 2005; 5;142-9		J Clin Epi 2007 doi:10.1016/j.jclinepi.2007.04.020

## How do we know: Is brick faulty?

- Use ***fundamental principles of science.***

## Fundamental principles

- “Details that could throw doubt on your interpretation must be given, if you know them.... [I]f you know anything at all wrong, or possibly wrong--to explain it.”						Feynman 1974
- Ask:* “****What could be wrong****.”*

## Fundamental principles

- Ask:* “****What could be wrong****.”*
- “... in analyzing theories of antibody formation, Joshua Lederberg* (8)* gives a list of nine propositions “subject to denial,” discussing which ones would be “most vulnerable to experimental test.”
- 		      Platt JR. Strong inference. Science 1964;146:347-353.

## Chaos in the brickyard:Translational research in 2007

- to illustrate problems: *‘markers for cancer’*

## Slide 18

## Slide 19

## Slide 20

## Markers for cancerBricks can be faulty

- Past• History• ‘Threats to validity’ of clinical research
- Present• RNA expression genomics for cancer prognosis.• Serum proteomics for cancer diagnosis.
- Future• Lessons: making research reliable, efficient

## History: Validation of cancer markersis ‘disappointing’ (not reproducible)

- Non-invasive markers: holy grail of cancer diagnosis  • carcinoembryonic antigen (CEA)   • CA-125• magnetic resonance spectroscopy of plasma

## Slide 23

## History: Validation of cancer markersis ‘disappointing’ (not reproducible)

- Non-invasive markers: holy grail of cancer diagnosis  • carcinoembryonic antigen (CEA)   • CA-125• magnetic resonance spectroscopy of plasma
- 2. 	Lessons from CEA• initial results (PNAS): ~100% sensitivity, specificity for 	colon cancer• high expectations...followed by disappointment

## History: Validation of cancer markersis ‘disappointing’ (not reproducible)

- Non-invasive markers: holy grail of cancer diagnosis  • carcinoembryonic antigen (CEA)   • CA-125• magnetic resonance spectroscopy of plasma
- 2. 	Lessons from CEA• initial results (PNAS): ~100% sensitivity, specificity for 	colon cancer• high expectations...followed by disappointment• experience led to lessons, ‘rules of evidence’ to 	evaluate diagnostic tests    (Ransohoff and Feinstein. NEJM 1978)

## Slide 26

## Now, cancer markers are promising

**Knowledge** of molecular biology provides targets to measure• past:  knew little about what to target • now:  know DNA ‘path’ from normal.. adenoma.. cancer
**Assays** can measure targets• past: assays ‘one dimensional,’ like CEA, fecal occult	blood testing (FOBT); prostate-specific antigen (PSA) • now: assays multi-dimensional; can measure any target 	-DNA - primers and probes, PCR	-protein - mass spectrometry

## Now, cancer markers promising, but..

- Mother Nature guards her secrets closely.
- New reductionist methods mean more data, but not necessarily more knowledge.
***Rules of evidence have not changed.***
- Our job:
- 	• to explore new technologies/fields efficiently• to avoid predictable mistakes, inflated expectations • make effort interdisciplinary, translational:    molecular biology, biochemistry, etc...    ... and clinical epidemiology, biostatistics.
***Culture clash may hinder exploration.***

## Markers for cancerBricks can be faulty

- Past• History• ‘Threats to validity’ of clinical research
- Present• RNA expression genomics for cancer prognosis.• Serum proteomics for cancer diagnosis.
- Future• Lessons: making research reliable, efficient

## “Validity”

*        **Meaning of “validity” is broad (Lat: “strong”)**     and confusing; meaning must be clarified.*
*                *
- 				               Nat Rev Cancer 2004; 4:309-14

## Two critical threats to validity

**Chance****Does chance explain ‘discrimination’?**** **
**Bias**Does bias explain ‘discrimination’?
- 			                 Nat Rev Cancer 2005;5:142-9

## Gene Expression Signature as a Predictor of Survival in Breast Cancer

**Gene Expression Signature as a Predictor of Survival in Breast Cancer**

## Strong discrimination led to interpretation as “definitive”

**for clinical practice**
- “... gene-expression patterns of primary tumours are better than available clinicopathological methods for determining the prognosis of individual patients.6,10,11”
- 	   		Ramaswamy and Perou, Lancet 2003;361:1576-7
**for biological research**
- “... compelling evidence... genetic program of a cancer cell at diagnosis defines its biologic behavior many years later, refuting a competing hypothesis....”		Wooster and Weber, NEJM 2003;348:2339-47

## Can chance explain results?

- Definition: In multivariable predictive models, overfitting (a problem of ‘chance’) occurs when large N of predictor variables is fit to a small N of subjects.  A model may ‘fit’ perfectly by chance, *even if no real relationship*. 								(Simon, JNCI 2003)
- Consequence: Results not reproducible in independent group.
- Method to check for: Assess reproducibility in independent group.

## Can chance explain results?

***to the editor:***
- “In research to validate a prognostic system, the inclusion of 61 patients from the... [training group in the validation group (N=295) means] the validation group is not independent.... [and] the degree of prognostic discrimination may have been inflated....”
- 				                   (NEJM 2003;348:1716)

## Slide 38

## If less discrimination, would interpretation be so strong? 

**for clinical practice**
- “... gene-expression patterns of primary tumours are better than available clinicopathological methods for determining the prognosis of individual patients.6,10,11”
- 	   		Ramaswamy and Perou, Lancet 2003;361:1576-7
**for biological research**
- “... compelling evidence... genetic program of a cancer cell at diagnosis defines its biologic behavior many years later, refuting a competing hypothesis....”		Wooster and Weber, NEJM 2003;348:2339-47

## To check for overfitting, assess reproducibility in independent group                                                              Nat Rev Cancer 2004;4:309. 

## Overfitting is not addressed in many studies of RNA expression                                       Lancet.  Feb 5, 2005 

- Michiels et al.When studies of RNA expression and prognosis of cancer were ‘reanalyzed’, using original data, in 5 of 7,	        ***results were ‘no better than chance.’***** **
- Ioannidis, in editorial ( “*Microarrays and molecular research: noise discovery?*”), suggests: “validation” groups were ***not*** independent.

## Overfitting is not addressed in many studies of RNA expression                                       Lancet.  Feb 5, 2005

- Michiels et al.When studies of RNA expression and prognosis of cancer were ‘reanalyzed’, using original data, in 5 of 7,	        ***results were ‘no better than chance.’***** **
- Ioannidis, in editorial ( “*Microarrays and molecular research: noise discovery?*”), suggests: “validation” groups were ***not*** independent.               *This problem is readily avoidable.*

## Slide 43

##                                    N Engl J Med 2004;351:2817-26.

*   **... because Methods showed ‘independent validation’:*
- “The prospectively defined assay methods and end points were finalized in a protocol signed on August 27, 2003.  RT-PCR analysis was initiated on September 5, 2003, and... data were transferred... for analysis on September 29, 2003.”

- Chance/overfitting *is* addressed in
-          study of RNA expression

## Slide 45

## Two critical threats to validity

**Chance**Does chance explain ‘discrimination’?		  (illustration: genomics)
**Bias****Does bias explain ‘discrimination’?****	****  (illustration: proteomics) **
- 		          		Nat Rev Cancer 2005;5:142-9

## Bias

**Definition**Systematic difference between compared groups, so that comparison is ‘erroneous.’
**Bias is *****Serious***• Biases are common in observational research.• Even one bias can be “fatal.”

## Strong claims that serum proteomics can diagnose cancer

- Claims:
- • for multiple cancers (ovary, prostate, breast) 		• sensitivity: 95-100%	• specificity: 95-100%
- • appeared in Lancet, Clin Chem, WSJ, NBC, PBS, etc.
- • led to plans for commercial test, Ovacheck, in 2003; 	but plans delayed by FDA
- • led researchers to redirect effort, grant proposals.

## Slide 49

## Purposeto diagnose ovarian cancer vs no cancer

Methods• ovarian cancer, controls• serum assessed by mass spectroscopy (SELDI-TOF) • spectra analyzed by ‘genetic algorithm’ (Correlogic)

- Methods• ovarian cancer, controls• serum assessed by mass spectroscopy (SELDI-TOF) • spectra analyzed by ‘genetic algorithm’ (Correlogic)

## Slide 51

## Results:“The discriminatory pattern correctly identified 
all 50 ovarian cancer cases.... 
[for] a sensitivity of 100%... specificity of 95%...”

**Results:****“The discriminatory pattern correctly identified **
**all 50 ovarian cancer cases.... **
**[for] a sensitivity of 100%... specificity of 95%...”**

## Does bias explain some serum proteomics results for ovarian cancer?

-                      (Keith Baggerly’s proposal, as reported in Nature news 2004)
-               Was bias introduced by ‘run order’ of specimens?
-     If cancers and non-cancers are run on different days and if the mass spec ‘drifts’ over time, then ***non-biologic ‘signal,’ associated with Ca vs no-Ca, is hard-wired into results.  Bias (signal) is not detected or removed by ‘splitting’ sample into training and validation groups; signal is actually present.***

## Recent example:Bias may explain ‘discrimination’

* **J Clin Invest 2006;116:271*

## Bias may explain ‘discrimination’

- PromisePeptide pattern in serum discriminates PrCa vs control:  	~100% sensitive, specific
- Interpretation•Exoprotease activities should be focus of “future  	peptide biomarker discovery efforts” (authors)•Low molecular weight “biomarker pipeline is surging	with potential” (editorialists)

## Bias may explain ‘discrimination’

- Compared groups are different:	•Cancer:  mean age 67 y.o.;  100% men

## Bias may explain ‘discrimination’

- Compared groups are different:	•Cancer:  mean age 67 y.o.;  100% men   	•Control:  mean age 35 y.o.;  58%  women

## Bias may explain ‘discrimination’

- Compared groups are different:	•Cancer:  mean age 67 y.o.;  100% men   	•Control:  mean age 35 y.o.;  58%  women
- Are there other differences (biases)?

## Bias is the challenge in observational research

- Bias is not ‘icing on the cake’; it*** is ***the cake.
- Bias is a large topic, difficult:• multiple biases; require different methods to address	(e.g., randomization, blinding, uniform handling, etc)• some methods not available in observational research • some biases may be impossible to identify• even ONE bias may be fatal
***The ‘process’ to deal with bias is routinely ignored by authors, reviewers, editors in ‘omics’ research.***

## Bias is so serious that results are guilty (of bias) until proven innocent. 

- Innocence is proven by**:*****Doing***** process **	• design - to avoid bias		• conduct - to measure if bias occurred	• interpretation - to determine if important***Reporting***** process*****	***Methods, Results, Discussion
- (Bias as a threat to the validity of cancer molecular-marker research.  Ransohoff. Nat Rev Cancer 2005;5:142-9)

## Process to deal with bias of ‘baseline inequality’ in RCT:  Report ‘results of randomization’, in ‘Table 1’   

## Process to deal with bias of ‘baseline inequality’ in typical ‘-omics’ study

## “All samples were stored at -80oC until use.”

- OK... but were specimens *handled equally* in all steps, e.g.,  	- time from blood draw to spin/freeze	- number of thaw-freeze cycles	- duration of storage	- type of blood collection tube (red/purple)	- time from thawing to assay	- etc....
*Any step is possible source of fatal bias **in a proteomics study.*

## Does serum proteomics diagnose cancer?

**Question**: ‘Where is the landmark study in NEJM,       or Science, that shows proof of principle?’	(I.e., convincingly avoids chance, bias)
**Answer**:    Nov  2007:  	None exists.> Nov  2007:  	Who knows?

## Many sources of bias to

- After specimens are received,
- differences occur in handling:time, ’place,’ etc.

** **

- Before specimens are received,
- differences occur in demographics,
- collection methods, etc.

** **

**Many *****sources of bias***** to**
** ****explain ‘discrimination’**

- Cancer

- Control

**Specimens**
**received in lab**

- Discipline:
*study design;**clinical epidemiology*

- Discipline:
- quality control in
- laboratory science

## Design of ‘experiment’:

-  Bias may be avoided by randomization, other methods

## Lessons for ‘translational’ research

- What kinds of translational research do problems of ‘illustrative example’ apply to?•’omics’ research (genomics, proteomics, etc.) •discovery-based research?•research about diagnosis or prognosis in people? •other?

## Lessons for ‘translational’ research

- What kinds of translational research do problems of ‘illustrative example’ apply to?•’omics’ research (genomics, proteomics, etc.) •discovery-based research?•research about diagnosis or prognosis in people? •other?
- Answer:*Whenever study design or method is observational, not experimental.*

## Many sources of bias to

- After specimens are received,
- differences occur in handling:time, ’place,’ etc.

** **

- Before specimens are received,
- differences occur in demographics,
- collection methods, etc.

** **

**Many *****sources of bias***** to**
** ****explain ‘discrimination’**

- Cancer

- Control

**Specimens**
**received in lab**

- Discipline:
*study design;**clinical epidemiology*

- Discipline:
- quality control in
- laboratory science

## Slide 71

## Slide 72

## Topics discussed by Forscher

- 	1. **bricks**2. edifices3. builders vs brickmakers 4. training
- “If the bricks were faulty...the edifice would crumble....”

## Lessons for ‘translational’ research

- In 2007:
- 1.	Faulty bricks exist in many areas of current translational research.
- ‘Rules of evidence’ can help us test strength of bricks and, ultimately, to build stronger edifices.
- Future efforts should emphasize ‘faulty brick prevention.’

## For future...

- 1. Role of journals
- 2. Role of specimens
- 3. ‘Conceptual framework’:  research v development
- 4. Incentives
- 5.	‘Attitude’
- 6.	etc.

## Slide 76

## For future...

- 1. Role of journals
- 2. Role of specimens
- 3. ‘Conceptual framework’:  research v development
- 4. Incentives
- 5.	‘Attitude’
- etc.
*Over time we will learn strategies for doing  ‘translational research’ that are more reliable, efficient.*

## Special thanks

***Support***: BRG/DCP/NCI
- CPTI/NCI
***Collaborators***: UNC-CHEDRN/DCP/NCI
- CPTI/NCI
***Colleagues***:UNC-CH
- NCI

## References

## Slide 80

## Thank You!

**Thank You!**

- Please visit the Medicine: Mind the Gap homepage to view upcoming seminars as they become finalized.
- http://consensus.nih.gov/mindthegap
**Contact Us:**
- NIH Office of Medical Applications of Research Kelli Marciel marcielk@od.nih.gov | 301.496.4819
- 	Presented by the NIH Consensus 	Development Program 	Celebrating 30 Years of Service