# Presentation: 529358

## Course #412Analyzing Microarray Data using the mAdb System April 1-2, 2008  1:00 pm - 4:00pmmadb-support@bimas.cit.nih.gov

- Day 2
- mAdb Analysis Tools

-  Esther Asaki, Yiwen He

- Use web site: http://mAdb-training.cit.nih.gov
- User Name on your card
- Password on the board

## Agenda

- mAdb system overview
- mAdb dataset overview
- mAdb analysis tools for dataset
- Class Discovery - clustering, PCA, MDS
- Class Comparison - statistical analysis
- t-test
- One-Way ANOVA
- Significance Analysis of Microarrays - SAM
- Class Prediction - PAM
- Various Hands-on exercises

## Class Comparison

- Hypothesis test and two types of errors

- mAdb statistical analysis tools for class comparison
- t-test
- One-way ANOVA
- SAM

**Class Comparison**

## Class Comparison

- Hypothesis test and two types of errors

- mAdb statistical analysis tools for class comparison
- t-test
- One-way ANOVA
- SAM

**Class Comparison**

## Distribution for Expression Data

**Frequency of measurements**

- Center: Mean 

**Signal Intensity**

- Spread: Standard deviation σ

- 50

- 30

- 70

## Sources of Variation in Microarray Data

- Biological variation
- Random
- Stochastic mechanism of gene expression
- Sample heterogeneity
- Patient to patient variation
- Due to the biological process under study
- Technical variation
- Printed probes
- RNA sample extraction
- Labeling efficiency
- Spot size
- Sample distribution on the arrays
- Background signals
- Cross hybridization

**Notes:**
Only the systematic variation caused by the biological process under study is of interest to us. The rest needs to be either removed or controlled.

Technical noise can either be systematic or random. Random ones can be removed by replicates.

Spatial effects caused by printing devices, reagent flow during washing, or uneven scanning.

Dye bias in 2-color arrays
Local signal bias in old affy chips (adjacent placement of probe pairs for each gene)

## Problems with Fold Change

- Genes with high fold change may exhibit high variability among cell types due to natural biological variability for these genes
- Genes with small fold changes may be highly reproducible and should be biologically essential genes
- Some systematic sources of variation are intensity-dependent. Simple, static fold-change thresholds are too stringent at high intensities and not stringent enough at low intensities.

## Take Home Messages

**Take Home Messages**

- Replicates (both biological and technical) are needed to remove random error
- Need normalization to remove systematic variability
- Need robust statistical tests
- Need additional biological validations

**Notes:**
e.g. patient sample, low amount and only biological replicates is possible. On the other hand, if homogeneous sample, e.g. cell line, you may need fewer biological replicates. However, since you have to repeat anyways due to existence of technical errors, you might as well repeat at the biological level so that both bio. and tech. errors are taken care of.
Biological replicates are arrays that use RNA samples from different individual organisms, pools of organisms or flasks of cells, but yet compare the same treatments or control/treatment combination. Multiple samples per treatment group, compensate for intrinsic biological variation in gene expression.
Technical replicates are arrays that use the same RNA samples and also the same treatments.  Thus the only differences in measurements are due to technical differences in array processing. Multiple arrays per sample, compensate for measurement errors and technical variation.
It is highly recommended that more biological replicates are done than technical replicates, especially when dealing with individual organisms rather than a cell line.
Pooling all samples together is NOT recommended.
If the degree of variation between biological replicates is thought to be very high, or not enough RNA can be collected from just one individual, then pooling samples may be beneficial.  It is still important to have more than one independent pool of samples in order to estimate biological variance.  Thus, the sample pools may be thought of as individuals when designing your experiment.
DyeSwap Pairs - Dye-swap every pair of control and experimental samples– this compensates for dye bias.
Sample size calculation

## Class Comparison

- Hypothesis test and two types of errors

- mAdb statistical analysis tools for class comparison
- t-test
- One-way ANOVA
- SAM

**Class Comparison**

## Hypothesis Test

**Before treatment**

**After treatment**

- d

- Null hypothesis

- Alternative hypotheses

- μ1

- μ2

## Spread (Variability) of Measurements

## Two Types of Errors

| Correct decision
 | Type 1 error
False positive |
| --- | --- |
| Type II error
False negative | Correct decision |

- Accept *Ho*

- Reject *Ho*

*Ho* is true

*Ho* is false

**Type I error:** Rejecting the null hypothesis while it’s true;
**Type II error: **Accepting the null hypothesis while it’s not true.

**Two Types of Errors**

## Relation of Type I & Type II Errors

-  Increasing the sample size (number of replicates) will reduce both errors.
-  ***p-value***: the probability (significance value) of observing Xp or bigger under H0.

- Control

- Tested

- Q1=The probability of a type I error   	(false-positive)
- Q2=The probability of a type II error 	(false-negative)

*f**1**(x):* expression in control population
*f**2**(x):* expression in tested population
*x**o* : the observed value of  *x*
*x**0* : the critical (rejection) value of *x*

**Relation of Type I & Type II Errors**

## Class Comparison

- Hypothesis test and two types of errors

- mAdb statistical analysis tools for class comparison
- t-test
- One-way ANOVA
- SAM

**Class Comparison**

## Statistical Analysis

**Goal: **To identify differentially expressed genes, i.e. a list of genes with expression levels statistically and (more important) biologically different in two or more sets of the representative transcriptomes.
- t-test			(1 or 2 groups)
- One-Way ANOVA	(> 2 groups)
- SAM			(1, 2, and more groups)

**Statistical Analysis**

## Data for mAdb One-Group Test

- Design: Two conditions, tumor vs. normal (or treated vs. untreated), labeled with Cy3 and Cy5, respectively.
- Data: Ratio, one group
- Null hypothesis: mean is equal to 1
- Results: A list of genes with ratio significantly different from 1. i.e. Different expression level in the two conditions.
- Note: due to dye bias, it’s better to do a dye swap.

## Data for mAdb Two-Group Test

- Affymetrix
- Normal in group 1 and tumor in group2.
- Paired test if normal and tumor are from the same patient.
- Two-color with common reference
- Normal as common reference with Cy3, two types of tumor (group 1 and group 2) both with Cy5.
- Pooled as common reference, normal and tumor (group 1 and group 2) both with Cy5. Paired if normal and tumor are from the same patient.

## Two-group t-Test

- The t-test assesses whether the means
-  of two groups are statistically different

*The null hypothesis: *

- μ1

- μ2

## t-Test (Cont’d)

**=**

**difference between group means**

**t-Test (Cont’d)**

- Treatment

- Control

**t-statistic**

## Calculating p-Value (t-Test)

- The p-value is the probability to reject the null hypothesis
- (                                ) when it is true (e.g. p=0.0001)
- Calculated based on t and the sample sizes n1 and n2.

## mAdb One-Group Test

**mAdb One-Group Test**

- 1 group statistic analysis automatically selected for a single group dataset

## mAdb Two-Group Test

**mAdb Two-Group Test**

- 2 group statistic analysis automatically selected for a 2 group dataset

## Two-Group t-Test Results

**Two-Group t-Test Results**

## Statistic Results Filtering

## Multiple Group Comparison

|  | Group 1 | Group 2 | ... | Group k |
| --- | --- | --- | --- | --- |
| Gene 1 | μ 1.1 | μ 1.2 | ... | μ 1.k |
| Gene 2 | μ 2.1 | μ 2.2 | ... | μ 2.k |
| ... | ... | ... | ... | ... |
| Gene n | μ n.1 | μ n.2 | ... | μ n.k |

- n: Number of genes/probes
- k: number of groups, k > 2

**Multiple Group Comparison**

## Data for mAdb Multiple-Group Test

- Time course/Dose response
- Normal vs. multiple types of tumor
- For two-color arrays, must have common reference.
- More than two types of tumor/treatments, with normal/untreated as common reference
- Normal, tumor type I, tumor type II, etc. with some common reference.

## Analysis of Variances (ANOVA)

- To compare several population means:

- vs.

## mAdb Multiple-Group Test

**mAdb Multiple-Group Test**

- Multiple group analysis automatically selected for a > 2 group dataset

## ANOVA Results and Filtering

## Hands-on Session 4

- Lab 9
- Total time: 10 minutes

##  Multiple Comparison

- Statistical problems with large-scale experiments
- Many null hypotheses are tested simultaneously in microarray, one for each probe.
- Although p-value cut off (α) of 0.01 is significant in a conventional single-variable test, a microarray experiment for 20,000 gene probes would identify 20,000 x 0.01 = 200 genes just by chance!

## Multiple Comparison Correction

- Probability of false-positive discovery (False Discovery Rate):

-    m: # hypothesis/genes
-    R0: # false positive
-    R: # significant hypothesis

**Multiple Comparison Correction**

## Significance Analysis of Microarrays (SAM)

- http://www-stat.stanford.edu/~tibs/SAM/index.html
- Goal is to select a fairly large number of differentially expressed genes (R), accepting some falsely significant genes (R0), as long as the FDR is low. i.e. R0 is relatively small compared to R.
- For one or two groups, SAM computes a t-like statistic d(i) for each probe i (i=1,2...n), measuring the relative difference between the group means.
- For more groups, SAM computes a F-like statistic.

**Notes:**
Data for 2 groups is same as for t-test, and for multiple groups is same as for ANOVA.

## SAM for 2 groups

**SAM for 2 groups**

## Permutation & the Expected d Values

**Permutation & the Expected *****d***** Values**

## SAM Plot for Delta = 2

- “significantly induced” genes

- “significantly reduced” genes

**SAM Plot for Delta = 2**

## SAM Plot Multiple Groups

## Calculating FDR

- Order the observed d statistics for all n genes so that* d**o**(1) ≤* *... ≤* *d**o**(i)... ≤* *d**o**(n)*.
- Plot the observed *d**o* vs. expected *d**e*
- Select a cutoff value *delta*
- Significant genes (R): | *d**o* - *d**e* | ≥ *delta*
- False genes from a permutation (R0p) : | *d**p* - *d**e* | ≥ *delta*
- Estimate false discovery (R0): median of R0p
- Estimate FDR: R0 / R

## Data for SAM in mAdb

- You can run SAM on data with 1, 2, or more groups
- Experimental design requirements are the same as those for t-test or ANOVA
- Note: SAM assumes that most of the genes in your dataset are NOT changed. So it is recommended that you run SAM on a larger dataset, instead of a small set with mostly significant genes.

## mAdb SAM Data

- Data for Subset: **bl and nb**from Dataset: **Small, Round Blue Cell Tumors (SRBCTs), Nature Medicine Vol 7, Num 6, 601-673 (2001)**
- Filter/Group by Array Property 63 arrays and 2308 genes in the input dataset 20 arrays and 2308 genes in the output dataset. 8 arrays assigned to Group A 12 arrays assigned to Group B
- Filter/Group by Array Property: Group A: Array/Set Name Contains 'bl' Group B: Array/Set Name Contains 'nb'

**mAdb SAM Data**

## mAdb SAM

**mAdb SAM**

## mAdb SAM

## mAdb SAM Results I

**mAdb SAM Results I**

## mAdb SAM Results II

**mAdb SAM Results II**

## mAdb SAM Results III

**mAdb SAM Results III**

## Hands-on Session 5

- Lab 10
- Total time: 10 minutes

## Agenda

- mAdb system overview
- mAdb dataset overview
- mAdb analysis tools for dataset
- Class Discovery - clustering, PCA, MDS
- Class Comparison - statistical analysis
- t-test
- One-Way ANOVA
- Significance Analysis of Microarrays - SAM
- Class Prediction - PAM

## Class PredictionSupervised Model for Two or More Classes

- Prediction Analysis for Microarrays (PAM)
- http://www-stat.stanford.edu/~tibs/PAM
- Provides a list of significant genes whose expression characterizes each class
- Estimates prediction error via cross-validation
- Imputes missing values in dataset

##  Design of the PAM algorithm

- Choose Features

- Cross-validation
- Test errors

- Evaluation of
- Classifier

- Discriminant function

- Training set

- Test set

- Final subset
- of variables

- Best model and
- subset of parameters

** ****Design of the PAM algorithm**

## Calculating the Discriminant Function

- Standardized centroid distance:
- Class average of the gene expression value minus the overall average of the gene expression value, divided by a standard deviation-like normalization factor (NF) for that gene.
-         dik **(centroid distance) = (class k avg – overall avg) / NF**
- Creates a normalized average gene expression profile for each class.

**Calculating the Discriminant Function**

## Reducing the Feature Set

- Nearest shrunken centroid:
- To "shrink" each of the class centroids toward the overall centroid for all classes by a threshold we call Δ.
- Soft threshold:
- To move the centroid towards zero by Δ, setting it to zero when it hits zero.
- After shrinking the centroids, the new sample is classified by the usual nearest centroid rule, but using the shrunken class centroids.

## Shrinking the Centroid

- Threshold Δ = 2.0:
- a centroid of 3.2 would be shrunk to 1.2;
- a centroid of -3.4 would be shrunk to -1.4;
- and a centroid of 1.2 would be shrunk to 0.

- Gene 1

- Gene 2

- Gene 3

**Original centroid**

**Shrunken centroid**

**3.2**

**-3.4**

**1.2**

**1.2**

**-1.4**

**0**

## Reduce Gene Number

- Group A

- Group B

**Reduce Gene Number**

## Sample

-  BL (Burkitt Lymphoma, n1=8)
-  EWS (Ewing, n2=23)
-  NB (neuroblastoma, n3=12)
-  RMS (rhabdomyosarcoma, n4=20)
-  There are  2308 features (distinct gene probes)
-  No missing values in array data sets
-  Each group has an aggregate expression profile
-  An unknown can be compared to each tumor class profile to predict which class it most likely belong

**Sample**

## Class Centroids

**Class Centroids**

- Compare model with new tumor tissues to make diagnosis

## Classifying an Unknown Sample

- Classification is made to the nearest shrunken centroid, in squared distance.

**Classifying an Unknown Sample**

## K-fold Cross Validation

- 10 Group A
- 5 Group B
- 5 Group C

- 1

- 2

- 3

- 4

- 5

- 10 Group A
- 5 Group B
- 5 Group C

- 10 Group A
- 5 Group B
- 5 Group C

- 10 Group A
- 5 Group B
- 5 Group C

- 10 Group A
- 5 Group B
- 5 Group C

- Entire Data Set

- 50 Group A
- 25 Group B
- 25 Group C
- K = 5

**K-fold Cross Validation**

## K-fold Cross Validation

- For each part in turn, the classifier is built on the other K-1 parts then tested on the remaining part.

- TEST

- TRAIN

- TRAIN

- TRAIN

- TRAIN

- 1

- 2

- 3

- 4

- 5

## K-fold Cross Validation 

- TRAIN

- TRAIN

- TRAIN

- TRAIN

- 1

- 4

- 5

- TRAIN

- TRAIN

- TRAIN

- TRAIN

- 1

- 2

- 3

- 5

- 2

- etc....

- TEST

- 3

- TEST

- 4

## Estimating Misclassification Error

- PAM estimates the predicted error rate based on misclassification error, which is calculated by averaging the errors from each of the cross validations.
- The model with lowest Misclassification Error is preferred.

## PAM Results

- Misclassification error

## Prediction Model for SRBCT

**Prediction Model for SRBCT**

## PAM summary

- It generates models (classifiers) from microarray data with phenotype information
- It does automatic gene selection for each models.
- Misclassification errors are calculated with the data for model selection.
- Require adequate numbers of samples in each group

## Hands-on Session 6

- Lab 11, Lab 12 (optional)
- Total time: 15 minutes

## References

- Clustering
- Eisen, et al, Cluster analysis and display of genome-wide expression patterns. *PNAS *1998, 95:14863-14868.
- Tavazoie, et al, Systematic determination of genetic network architecture.*Nat Genet* 1999, 22:281-285.
- Sherlock, Analysis of large-scale gene expression data. *Brief Bioinform* 2001, 2(4):350-62.
- PCA
- Yeung & Ruzzo, Principal component analysis for clustering gene expression data. *Bioinformatics *2001, 17(9): 763-74.
- Statistical Analysis
- Cui & Churchill, Statistical tests for differential expression in cDNA microarray experiments. *Genome Biology* 2003, 4:210
- SAM
- Tusher, Tibshirani and Chu, Significance analysis of microarrays applied to the ionizing radiation response. *PNAS* 2001, 98: 5116-5121
- PAM
- Tibshirani, et al, Diagnosis of multiple cancer types by shrunken centroids of gene expression. *PNAS* 2002, 99:6567-6572

## Other Microarray Resources

- Statistical Analysis of Microarray Data & BRB Array Tools (NCI Biometrics Research Branch) class #410. Offered bimonthly; 4/8-9/08
- Partek, R, GeneSpring classes – training.cit.nih.gov
- Introduction to Principal Component Analysis and Distance Geometry class #407
- Clustering: How  Do They Make Those Dendrograms and Heat Maps – class #406
- Microarray Interest Group
- 1st Wed. seminar, 3rd Thu. journal club
- To sign up: http://list.nih.gov/archives/microarray-user-l.html
- Class slides available on “Reference” page

## mAdb Development and Support Team

- <date/time>

|   John Powell, Chief, BIMAS, CIT
  Lynn Young, Ph.D. |   Esther Asaki*
  Yiwen He, Ph.D.*
  Kathleen Meyer*
  Wenming Xiao, Ph.D.* |
| --- | --- |
|                                  *SRA International contractor |  |
|  |  |

**mAdb Development and Support Team**

## http://madb.nci.nih.gov

- <date/time>

- <date/time>

- <date/time>

- <date/time>

- <date/time>

- <date/time>

**For assistance, remember: **
**madb_support@bimas.cit.nih.gov**

- http://madb.niaid.nih.gov