# Presentation: 925723

## Figures of Merit for Evaluating Different Data Analysis Methods and Experimental Designs

**Yudong He, Ph. D.**
- Director, Algorithms & Applications
- Rosetta Inpharmatics, Inc*
- *A Wholly Owned Subsidiary of Merck & Co. Inc
- Kirkland, Washington, USA

- FDA/NCTR Toxicoinformatics Workshop on
**Toxicogenomics Databases, Study Design, and Data Analysis**
- December 4, 2002
- FDA’s National Center for Toxicological Research (NCTR)
- Jefferson, Arkansas, USA

## Applications of Microarray Data“We need to satisfy both camps”.

**To study global changes accompanying changes in cell physiology **
**To relate phenotypes to patterns of overall cellular transcript levels**
**Two application camps: **
**Pattern matching**
**Detect individual gene expression and regulation**

## Evaluation of Methods at Various Levels“We need to establish a set of standards”.

**Evaluation of Methods at Various Levels*****“We need to establish a set of standards”.***

**Gene**

**Experiments**

**FlexJet array**

**Image processing       Slide level analysis     Data set mining**

## Ink-jet Oligonucleotide Microarrays

** ****ink-jet in situ synthesis**
** ****60-mer or different lengths**
** ****25,000 oligos / 1 x 3 inches**
** ****concept to chip in hours**
** ****high flexibility**
** ****2-color hybridization**

**C**
**A**
**T**
**A**
**T**
**T**
**G**
**A**
**C**
**T**
**G**
**G**

**G**
**T**
**C**
**G**
**T**
**T**
**A**
**A**
**G**
**C**
**G**
**A**

**A**
**T**
**C**
**G**
**C**
**T**
**G**
**A**
**G**
**T**
**T**
**T**

**A**

**A**

**G**

**A**

**C**

***Nature Biotech 19: 342 (2001)***

## Scientific Goals of Standard Data Set

- Support analysis methods development
- image processing and background subtraction
- normalization
- gene-specific and probe-specific error models
- similarity metrics
- Support comparison of different microarray experiment design types
- Catalyze community contributions in these areas
- Standard data set + non-parametric figures of merits

## Experiment Description

- 20 human tissue totRNAs
- 204 Flexjet arrays
- Pools and pairings specifically designed to support multiple analysis objectives
- 25k chip design to support multiple analysis objectives

## 25k SDS Chip Design

- 2,487 genes selected.
- 2,466 RefSeq genes represented by 22,657 probes. 1 or 2 probes per exon, 1-40 probes per gene (mean ~ 8).
- 21 alternating splicing genes represented by750 probes. 4-104 probes per isoform (mean ~ 36).
- ~1000 negative control spots
- ~1000 positive control spots for spike-ins

## Experiment Design

- Pool1 + B

- Pool1 + C

- Pool1 + D

- Pool1 + E

- Pool1 + F

- Pool1 + G

- Pool1 + H

- Pool1 + I

- Pool1 + J

- Pool1 + K

- Pool1 + L

- Pool1 + T

- Pool1 + A

- Pool1

-  = 0.3
- = 0.01 - 0.1

- Pool2

- Ring	     20x4
- Chords	       4x4
- Spokes
-   Pool Ref     10x4
-   Distant Ref 10x4
- Controls
-    End-to-end10x2
-    Hyb-to-end  4x2

- 204

- Pool1 + M

- Pool1 + N

- Pool1 + O

- Pool1 + P

- Pool1 + Q

- Pool1 + S

- Pool1 + R

## 88 Non-Control Experiments

**88 Non-Control Experiments**
- (2424 genes passing p<0.01 & |mlratio|>0.3 in >20 experiments)

- against
- distant
- reference

## All replicates grouped

- over >1,000 signature
- genes. The performance
- becomes worse when
- less signature genes
- are used.

- against
- distant
- reference

## Error model provides P-values for all observations of expression difference

- Null and repeated experiments are used to train the error model

- Dynamic Range ~103

- Control: A vs. A

**Log****10****(Intensity)**

**Log****10****(Expression Ratio)**

- Experiment:  A vs. B

*C. Roberts, et al., Science, Feb 4, 2000*

- Dynamic Range ~ 103

**Log****10****(Intensity)**

## False Positives are Reduced with Two-Color Protocol, Fluor Reversal, and Gene-Specific Error Model

- PFA = 7%

- 31 A vs. A Experiments

**200 Randomly Selected Genes**

- PFA = 3%

- PFA = 0.3%

- PFA = 0.8%

**Single Channels**

**One Two-Color Array**

**Fluor Reversed Pair of Arrays**

**Fluor Reversed Pair and Gene-Specific Error Model **

**False positive defined as +/- 40% error**

## FOM#1 Results: Probe Level Detection

**Benefit by**
**using onr fluor**
**reversal pair**

**Benefit by**
**hybridizing two-colored samples **
**on one chip**

**Normalized # of positivesc**

**Normalized # of false positives                  **

## FOM#1 Results: Multiple Level Detections

**FOM#1 Results: Multiple Level Detections**

**Normalized # of positives**

**Normalized # of false positives                  **

## FOM #2  for Pattern Matching= 2N/sum(R12+R21)

- 15,248
- 16,611

## Results from K-Means 

**Results from K-Means **

## FOM#1 Results

**Normalized # of false positives                  **

**Normalized # of positives**

**1**
**  ****2**
**   ****3**
**    ****5**
**    ****8**
**     ****12**
**     ****15**
**18**
**    ****17**

## Comparison of Experimental Design

-                     Real     via        via        via      Real       via       via       via
-                             Spoke  Ring+    Ring-               Spoke  Ring+  Ring-

- Experiments                          Controls

**Estimated error            Standard deviation**
**   ****for log(ratio)                    for log(ratio)**

**1**
**1**
**1**
**1**
**2**
**2**
**2**
**2**
**2**
**3**
**5**
**8**
**18**
**17**
**15**
**12**
**1**
**1**
**1**
**1**
**2**
**2**
**2**
**2**
**2**
**3**
**5**
**8**
**18**
**17**
**15**
**12**

**B**
**C**

**Comparison of Experimental Design**

## Concluding Remarks

- Two figures of merits are introduced
- Rosetta baseline processing is illustrated to be high-performance, including
- image processing
- background subtraction
- normalization
- error modeling
- similarity metric choice
- Advanced methods are being tested

## Contributors/Acknowledgments

- Informatics Department
- Yudong He, Hongyue Dai, Eric Schadt, Guy Cavet, Stephen Edwards, Sergey Stepaniants, Roland Stoughton
- Guided Solutions Department
- Allan Jones, Chris Roberts, Kelly Daniel, Roger Ulrich
- Biology Department
- Dan Shoemaker, Peter Linsley
- North Creek Gene Expression Lab
- Matt Marton et al.