# Presentation: 045436

## Biostatistics Research Outside of Academia 

- The experience of one alum at the National Institutes of Health
- Stuart G. Baker, Sc.D.

## Background

- Harvard-Biostatistics (1980-84; post-doc 1985)
- Special thanks to Steve Lagakos, Christine Waternaux, Milt Weinstein, Nan Laird, and Marvin Zelen
- NIH-Cancer Prevention-Biometry (1985-present)
- Special thanks to Peter Greenwald, David Byar, Laurence Freedman, and Philip Prorok

## Unusual co-authors

- My wife  (Karen Lindeman, M.D., Dept of Anesthesiology, Johns Hopkins)
- Members of my wife’s department
- My elementary school best friend who I had not seen in 20 years until the ENAR meeting in Memphis (Paul Pinsky, Ph.D. now at NCI)
- The editor of the Journal of the National Cancer Institute (Barry Kramer, M.D.)
- Three Nordic researcher pen pals

## Research projects

- Grouped by motivation

## Supporting general initiatives

## Figuring out the question

- Need to make sense of all the anticipated data from early detection biomarkers
- Generally agreed that lots of computing will be needed–NASA called in
*Computers are useless. They only give answers*– Pablo Picasso

##  Key question: Which biomarkers, if any, are promising for further study as triggers for early intervention ?

- Baker(1998); Baker,Srivastava,Kramer(2002)
- _Study design: _
-  cohort study with stored specimens
-  test for marker in all cases and some controls
-  sample size
- _Estimation_
-  false and true positive rates (target values)
-  avoid overfitting

## Harried Sorter and the Curse of Dimensionality

| Marker | B=1 | B=2 | B=3 |
| --- | --- | --- | --- |
| A=1 | .00 | .00 | .00 |
| A=2 | .05 | .15 | .20 |
| A=3 | .20 | .30 | .10 |

| Marker | B=1 | B=2 | B=3 |
| --- | --- | --- | --- |
| A=1 | .54 | .01 | .01 |
| A=2 | .10 | .03 | .20 |
| A=3 | .05 | .05 | .01 |

| Marker | B=1 | B=2 | B=3 |
| --- | --- | --- | --- |
| A=1 | 0 | 0 | 0 |
| A=2 | 1/2 | 5 | 4 |
| A=3 | 4 | 6 | 10 |

- Controls: False Positive Rate

- Cancers: True positive rate

- What regions (of A and B) optimize ROC curve?

- Ratio: true/false positive rate

- No need to check 29 regions

- Select regions by ratio of true to false positive rate

## Harried Sorter and the Curse of Dimensionality

| Marker | B=1 | B=2 | B=3 |
| --- | --- | --- | --- |
| A=1 | .00 | .00 | .00 |
| A=2 | .05 | .15 | .20 |
| A=3 | .20 | .30 | .10 |

| Marker | B=1 | B=2 | B=3 |
| --- | --- | --- | --- |
| A=1 | .54 | .01 | .01 |
| A=2 | .10 | .03 | .20 |
| A=3 | .05 | .05 | .01 |

| Marker | B=1 | B=2 | B=3 |
| --- | --- | --- | --- |
| A=1 | 0 | 0 | 0 |
| A=2 | 1/2 | 5 | 4 |
| A=3 | 4 | 6 | 10 |

- Controls: False Positive Rate

- Cancers: True positive rate

-  **Positive **** if A=3, B=3**

- Ratio: true/false positive rate

## Harried Sorter and the Curse of Dimensionality

| Marker | B=1 | B=2 | B=3 |
| --- | --- | --- | --- |
| A=1 | .00 | .00 | .00 |
| A=2 | .05 | .15 | .20 |
| A=3 | .20 | .30 | .10 |

| Marker | B=1 | B=2 | B=3 |
| --- | --- | --- | --- |
| A=1 | .54 | .01 | .01 |
| A=2 | .10 | .03 | .20 |
| A=3 | .05 | .05 | .01 |

| Marker | B=1 | B=2 | B=3 |
| --- | --- | --- | --- |
| A=1 | 0 | 0 | 0 |
| A=2 | 1/2 | 5 | 4 |
| A=3 | 4 | 6 | 10 |

- Controls: False Positive Rate

- Cancers: True positive rate

-  **Positive **** if A=3, B=3**

- Ratio: true/false positive rate

**            ****or if A=3, B=2**

## Harried Sorter and the Curse of Dimensionality

| Marker | B=1 | B=2 | B=3 |
| --- | --- | --- | --- |
| A=1 | .00 | .00 | .00 |
| A=2 | .05 | .15 | .20 |
| A=3 | .20 | .30 | .10 |

| Marker | B=1 | B=2 | B=3 |
| --- | --- | --- | --- |
| A=1 | .54 | .01 | .01 |
| A=2 | .10 | .03 | .20 |
| A=3 | .05 | .05 | .01 |

| Marker | B=1 | B=2 | B=3 |
| --- | --- | --- | --- |
| A=1 | 0 | 0 | 0 |
| A=2 | 1/2 | 5 | 4 |
| A=3 | 4 | 6 | 10 |

- Controls: False Positive Rate

- Cancers: True positive rate

-  **Positive **** if A=3, B=3**

- Ratio: true/false positive rate

**            ****or if A=3, B=2**

-         **or if A=2, B=2**

## Harried Sorter and the Curse of Dimensionality

| Marker | B=1 | B=2 | B=3 |
| --- | --- | --- | --- |
| A=1 | .00 | .00 | .00 |
| A=2 | .05 | .15 | .20 |
| A=3 | .20 | .30 | .10 |

| Marker | B=1 | B=2 | B=3 |
| --- | --- | --- | --- |
| A=1 | .54 | .01 | .01 |
| A=2 | .10 | .03 | .20 |
| A=3 | .05 | .05 | .01 |

| Marker | B=1 | B=2 | B=3 |
| --- | --- | --- | --- |
| A=1 | 0 | 0 | 0 |
| A=2 | 1/2 | 5 | 4 |
| A=3 | 4 | 6 | 10 |

- Controls: False Positive Rate

- Cancers: True positive rate

-  **Positive **** if A=3, B=3**

- Ratio: true/false positive rate

**            ****or if A=3, B=2**

-         **or if A=2, B=2**

-      **  ****or if A=3, B=2 **

-      **  ****or if A=2, B=3 **

## Clarifying the issues

- “Validation” of surrogate endpoints
- Muddled clinical view
*Validation is one of those words ... that is constantly used and seldom defined** - *Alvin Feinstein
- Confusing statistics literature
-  Day & Duffy / Begg & Leung paradoxes
-  Prentice criterion for estimation
-  Baker, Izmirlian, and Kipnis (submitted)

## Again, need the question

- Question:* Is inference about *_*treatment effect*_* likely to be the same when using a “potential” surrogate endpoint as when using a true endpoint?*
*A correlate does not a surrogate make *–Fleming and DeMets (1996)
*A perfect correlate does not a surrogate make* Baker and Kramer (2003)
- Graphical clarification

## Control group

- Surrogate

- True

- Study group

-  control group

- control

-  Study group

- study

- Perfect correlation (lines)

- Yields valid hypothesis testing

- +Prentice Criterion (lines coincide)

## Control group

- Surrogate

- True

- Study group

-  control

- control

- study

- Perfect correlation, No  Prentice Criterion (lines differ)

- study

- Hypothesis testing: Not valid

- Valid estimation (if estimate lines from previous study)

## Cancer screening-not your usual randomized trial

- 0

-  5 year follow-up

- 10 year follow-up

- 15 year follow-up

- adaptive

- 40

- Reduction in breast cancer mortality per 10,000 due to receipt of screening – data from HIP trial

- Simple adjustment for dilution after screening stopped (Baker, et al. 2003)

## Thinking “out of the box”

## Missing, missing everywhere, with finesse to spare:potential outcomes

- Baker and Lindeman (1994)
- paired availability design for combining data from multiple before-and-after studies
- effect of epidural analgesia on probability of Cesarean section
- before period: epidural less available
-  after period:   epidural more available

## Thought experiment

| 
subjecttype | Treatment in 
before period  
epidural is less available | Treatment in 
after period 
epidural is more available |
| --- | --- | --- |
|  |  |  |
|  |  |  |
|  |  |  |

## Thought experiment

| 
subjecttype | Treatment in 
before period  
epidural is less available | Treatment in 
after period 
epidural is more available |
| --- | --- | --- |
| EE | epidural | epidural |
|  |  |  |
|  |  |  |

## Thought experiment

| 
subjecttype | Treatment in 
before period  
epidural is less available | Treatment in 
after period 
epidural is more available |
| --- | --- | --- |
| EE | epidural | epidural |
| NN | no epidural  | no epidural |
|  |  |  |

## Thought experiment

- Estimate effect of receipt of epidural in *NE*
- Similar model later independently proposed by Angrist, Imbens, Rubin (1996)– randomized trials

| 
subjecttype | Treatment in 
before period  
epidural is less available | Treatment in 
after period 
epidural is more available |
| --- | --- | --- |
| EE | epidural | epidural |
| NN | no epidural  | no epidural |
| NE | no epidural  | epidural |

## =

*NN*
*NE*

*NN*

*NE*
*EE*

- no
- epidural

**epidural**

- no
- epidural

**epidural**

*EE*

**pr( epidural | after)**

**pr( epidural | before)**

- -

- pr(*NN*)

- pr(*NE*)** **

- pr(*EE*)** **

- pr(*EE*)** **

- pr(*NE*)

- pr(*NN*)

**before**

- pr(*NE*)** **

- +

- +

## after

*NN*
*NE*

*NN*

*NE*
*EE*

- no
- epidural

**epidural**

- no
- epidural

**epidural**

*EE*

- pr(*NN*)

- pr(*NE*)** **

- pr(*EE*)** **

- pr(*EE*)** **

- pr(*NE*)

- pr(*NN*)

**before**

- +

- +

## -

*NN*
*NE*

*NN*

*NE*
*EE*

- no
- epidural

**epidural**

- no
- epidural

**epidural**

- Pr(CS| *NN*, no epidural )

- pr(CS| *EE*, epidural)

- pr(CS| *NE*, no epidural)

- pr(CS| *NN*, no epidural)** **

- pr(CS| *EE*, epidural)

- pr(CS| *NE*, epidural)

- pr(CS|*NE*,no epid)

** **pr(CS|*NE,*epid)

*EE*

- pr(CS| **after**)

- pr(CS| **before**)

- =

- pr(epid|**after**)

- pr(epid|**before**)

- pr(*NN*)

- pr(*NE*)** **

- pr(*EE*)** **

- pr(*EE*)** **

- pr(*NE*)

- pr(*NN*)

- X

- X

- X

- X

- X

- X

**before**

- pr(Cesarean-section (CS) | **after**)=

- pr( Cesarean section (CS)| **before**)=

- +
- +

- +

- +

## Applications of potential outcomes

- Paired availability design
- results compared with randomized trials, multivariate adjustment for observational data (Baker and Lindeman 2000)
- cancer screening (Baker et al., in press)
- Non-compliance and survival
- screening trial with refusers (Baker 1998)
- Non-compliance and auxiliary variable
- missing by design (no biopsy if low PSA) (Baker 2000)

## Computational Necessity

## Regression models with missing categorical data

- Computations to extend thesis to discrete survival, diagnostic testing, longitudinal data .... case-control with haplotypes
- User specifies matrices and link functions
- Program generates EM, Newton-Raphson
*To paraphrase the maxim that amateurs discuss strategy and professional discuss logistics, ... users discuss models and developers discuss computation **–* from Baker (1994)

## Generality leads to simplicity

- How to maximize a multinomial likelihood with parameters that are ratios of functions to summations of functions?
- Poisson likelihood with an extra parameters for each multinomial constraint
- MP (Multinomial-Poisson) transformation
*Don’t prove anything unless you know it is true* - Steve Piantadosi, Johns Hopkins

## Perfect-fit paradigm

- Models for missing, survival, non-compliance: multinomial data
- Goal: Minimum assumptions
- Perfect-fit closed-form solutions
- Simple asymptotic variance
- Delta method; M-P transformation
-  symbolically computed derivatives

## Computational Serendipity

## Models to evaluate cancer screening

- Trying to develop a model with as few underlying parameters as possible
*It makes no sense to convey a beguiling sense of “reality” with irrelevant detail, when other equally important factors can only be guessed at *– Robert May
- Weak links of cancer screening models
-  effect of screening on cancer mortality
-  self-selection bias

## Simulation Surprise

- Rate of cancer detection in absence of screening is identifiable using *only data from subjects screened*
- Reduces self-selection bias (no birth cohort effect, progressive detection)
- Baker and Chu (1990) and simplified in  Baker(1998); Baker,et al.(2003)
*Make everything as simple as possible and not simpler* -Albert Einstein

- Estimating detection rate if no screening using only data from subjects who were screened

- F50

- F51

- N50

- I50

- S50

- 50

- F50= pr(detected first screen age 50)

- I50 = pr(detected in interval age 50)

- S50 = pr(detected second screen after age 50)

- If first screen at age 50

- 51

- If first screen at age 51

- F51 = pr(detected first screen age 51)

- N50 =pr(detected no screen age 50)

- = pre-clinical              phase

**KEY EQUATION**: **N****50****= ****F****50**** + I****50**** + S****50**** – ****F****51**

- age

- age

## Analyzing data

## Observer agreement with replicates

- Baker, Freedman, Parmar (1993)
- Goal: estimate within- and between- observer variation in pathology classification
- Data: repeated classifications by pathologists
- Method: novel latent class model

## Double sampling survival data

- Baker, Wax, and Patterson (1993)
- Goal: Estimate the effect of a drain on the probability of a wound infection
- Data
- Full follow-up: after hospital discharge
- Partial follow-up: censored at discharge
- Model: informative censoring

## Non-ignorable missing survey data

- Baker, Ko, and Graubard (2002)
- Data: large health survey
- Goal: Estimate the effect of balance on the probability of depression
- Method
- Missing in depression depends on depression and covariates
- Matrix variance formula with complex survey design

## Improving understanding

## Seeing is understanding

- Simpson’s Paradox: *Good for men, good for women, bad for people**  *—Tom Louis
- BK-Plot (Baker, Kramer 2002) but independently developed earlier
*Stigler’s Law of Eponymy... no invention or discovery is ever named after the right person  *- Howard Wainer (Chance magazine), who coined the name BK-Plot

| girls | Grow little | Grow alot | Frac grow alot |
| --- | --- | --- | --- |
| No milk |          4   |          16 |      8/10 |
| milk |          1  |           9 |      9/10 |

| boys | Grow little | Grow alot | Frac grow alot |
| --- | --- | --- | --- |
| No milk |          8   |          2 |      2/10 |
| milk |         14  |          6 |      3/10 |

| Both | Grow little | Grow alot | Frac grow alot |
| --- | --- | --- | --- |
| No milk |          12 |            18 |      6/10 |
| milk |          15 |            15 |      5/10 |

- Good for Boys, Good for Girls, Bad for Children   (for 3rd grade math class)

## BK-Plot

- 2/10

- 3/10

- Fraction that are girls

- 9/10  if milk

- 8/10 if no milk

- 2/3

- 1/3

- 5/10

- 1

- 0

- 6/10

- Fraction that grow a lot

**girls**

**boys**

- If milk

- If no milk

**both**

**both**

- If milk

- If no milk

- milk

- No milk

## A casual remark on “causality”

-  **A** bests **B** in one randomized trial
-  **B** bests **C** in another randomized trial
-  Does **A** best **C**?
- Apply B-K plot (Baker and Kramer, 2003)
*It's odd that logical acuity... often reveals hidden ambiguities*" John Allen Paulos, mathematician

## A bests B

**B**

**C**

**A**

- Percent with confounder

- Benefit

**B**

- Benefit

**A**** bests ****B**

**C**** bests ****A**

**B**

**A**

**C**

**B**** bests ****C**

- Two Randomized Trials

- One Randomized Trial

- 30%

- 10%

- Percent with confounder

- 10%

## Meta-analysis - binary outcomes

- Compare
-  risk difference
-  relative risk
-  odds ratio
- When the distribution of an unobserved binary variable (with no treatment interaction) differs across studies

## Risk Difference

- risk difference

- Fraction with unobserved binary variable

- probability

- Fraction with unobserved binary variable

- 0

- 0

- 1

- 1

- Constant

## Relative Risk

- relative risk

- Fraction with unobserved binary variable

- probability

- Fraction with unobserved binary variable

- 0

- 0

- 1

- 1

- Constant

## Odds Ratio

- Odds ratio

- Fraction with unobserved binary variable

- probability

- Fraction with unobserved binary variable

- 0

- 0

- 1

- 1

- NOT Constant

## Missing binary outcomes in a randomized trial

- Baker and Freedman (2003)
- Missing due to an unobserved binary covariate with no interaction with treatment
- Sensitivity analysis
-  one parameter instead of the usual two
-  explicit use of randomization

## Helping out

## The FDA comes to NIH

- Design a study to compare performance of digital versus analog mammography
- Baker et al. (1998); Baker & Pinsky (2001)
- Modified paired design:
- if analog negative, only a random sample get digital (reduces costs)
- partial area under ROC curves

## Inspiration from an unlikely source: protocol review

- Estimate effect of tamoxifen on breast cancer *in gene carriers *(randomized trial)
- Protocol
-  case-only design (observational study)
-  estimate RR
- Baker and Kramer (submitted)
-  nested-case control design
-  estimate risk difference by age

## It’s not what the investigators want--it’s why they can’t get it

- Request
- Identify high-risk subjects for cancer prevention trial
-  increase power
- But more complicated
- Power increases if RR same (low & high risk)
- Power decreases if risk difference same
- Baker, Kramer, Corle (submitted)

## A simple request that led to an international collaboration

- Report on twin study
- Lichtenstein et al NEJM, 2000
- genetics are a minor component of cancer
- Latent class model (based on genetics)
- Age-specific data from Lichtenstein et al
- Results agreed with Lichtenstein et al.
- Baker, Lichtenstein, Kaprio, Holm (in press)

## Final quotes

## Take-home messages

*Take chances, get messy, make mistakes* -The Magic School Bus, television show
*Mathematics is the language of precise thinking –*R. Hamming, mathematician
*In my experience, any really groundbreaking paper had difficulty being accepted, while the mundane sailed through* -Elias Zerhouni, NIH Director