Summary of the MAQC Project Meeting

**The MAQC Project: Calibrated RNA Samples, Reference Datasets, and QC
Metrics/Thresholds for Microarray Quality Control**

Meeting Date: May 2-3, 2005

Meeting Place: FDA Parklawn Building, 5600 Fishers Lane, Rockville, MD
20857

Summary Date: May 10, 2005

Author: Leming Shi, FDA/NCTR, 870-543-7387, <leming.shi@fda.hhs.gov>

**Day 1: 8:00 AM -- 5:00 PM, Monday, May 2, 2005**

What Makes the MAQC Project Different from Other Efforts?

Felix Frueh (FDA/CDER) gave a brief summary of the aggressive progress
of the MAQC project. Leming Shi (FDA/NCTR) reiterated the goals of the
MAQC project and why the MAQC project is complementary to other ongoing
efforts (*e.g*., ERCC and NIST Gene Expression Metrology Program) aimed
at microarray quality control and standardization. Leming Shi emphasized
again the ultimate importance of the **availability** and
**accessibility** of the two RNA samples used in the MAQC project to
**the general microarray community beyond the MAQC project**. It was
also stressed that for RNA samples from human donors it is essential to
make an RNA batch in a quantity that is **available for the microarray
community to use for several years**. The availability of the same RNA
samples used in the MAQC project, the large reference datasets by
different microarray platforms and other independent technologies, and
the derived QC metrics/thresholds will help individual microarray
laboratories to better assess performance and to avoid procedural
failures. Studies (including those published in *Nature Methods*, May
2005) aimed at microarray standardization and quality control would have
been much more valuable if the same RNA materials were available to the
microarray community.

Leming Shi outlined the goals of the meeting:

1.  To select two RNA samples for the main study from four candidates
    (A. Ambion Brain RNA; B. Ambion Liver RNA; C. Clontech UHRR; and D.
    Stratagene UHRR);

2.  To design the main study; and

3.  To select 1,000 genes for QRT-PCR verification

by reviewing the datasets generated in the MAQC pilot study.

Leming Shi also reiterated the six criteria originally proposed during
the February 11, 2005 MAQC project meeting on RNA sample selection:

1.  Available in large quantity;

2.  Reproducibility in production;

3.  High quality;

4.  Accessibility (commercial source);

5.  Wide gene presence; and

6.  Large fold changes for a number of genes.

The MAQC pilot study was designed to provide quantitative information to
evaluate the candidate RNA samples based on (5) gene presence, (6) fold
change (differential gene expression), and (3) RNA quality.

Presentations on the Analysis of the MAQC Pilot Datasets

Jacques Retief (Affymetrix) and Weida Tong (FDA/NCTR) co-chaired the
morning's session on the analysis of the pilot study datasets. Yaron
Turpaz (Affymetrix), Paul Wolber (Agilent), Mike Wilson (Ambion), Lu
Zhang (Applied Biosystems), Rich Shippy (GE Healthcare), Leming Shi
(NCTR), Walter Liggett (NIST), Rob Clum (Stratagene), and Rick Jensen
(UMass Boston) gave a brief presentation on each organization's analysis
results. Clontech did not provide the analysis results and Laurence
Lamarcq (Clontech) could not attend the meeting. Shawn Baker (Illumina)
could not attend the meeting but provided a short presentation with
analysis results that was shown to the attendees. Each presentation was
followed by \~5 mins discussion. After the presentations, there was a
general discussion for about 30 mins before lunch. It was emphasized
that the ranking of the four RNA samples by quality and gene presence
was straightforward and different data analysis sites reached very
consistent ranking. However, the tricky issue was how to rank the six
sample pairs by *differential* gene expression. The diversity of the
analysis methods applied for identifying genes with differential
expression for the six sample pairs resulted in inconsistent ranking of
sample pairs, as expected. It was decided at the end of the discussion
that instead of directly ranking the four candidate RNA samples, the
selection should be focusing on the ranking of the six sample pairs.

Paul Wolber co-chaired the afternoon's sessions on RNA sample selection
(with Jim Fuscoe of NCTR), sample-mixing strategies (with Uwe Scherf of
CDRH), and independent verification (with Federico Goodsaid of CDER).

It was proposed (by *e.g*. Lu Zhang and Rich Shippy) and accepted by the
MAQC group that the balance of the number and magnitude of up- and
down-regulated genes should also be considered as a criterion for
selecting the best RNA sample pair. All attendees were invited to vote
for their favorite sample pairs based on the following three criteria:

1.  Gene coverage;

2.  Differential gene expression (fold change);

3.  Balance of the number and magnitude of up- and down-regulated genes.

The results are shown in Table 1.

Table 1. Voting results on RNA sample pairs

  --------------------------- ------------- -------------- ---------------
  Criteria                    A-D Pairing   A-B Pairing    C-D Pairing

  Gene coverage               18            5              

  Differential gene           16            9              
  expression (fold change)                                 

  Balance of up- and          19            2              2
  down-regulated genes                                     
  --------------------------- ------------- -------------- ---------------

A second round of voting between A-D and A-B pairs confirmed the
preference of the MAQC group on the choice of A (Ambion brain RNA) and D
(Stratagene UHRR) for the MAQC main study. The MAQC group reiterated
that all the four RNA samples were of good quality compared to samples
one would normally work with in the real biological experiment. The MAQC
group emphasized that the selection of the two RNA samples for the MAQC
study did not imply that these two samples were superior to other
products. Instead, it only reflected the factor that they best met the
criteria set specifically for the MAQC project.

Stratagene has a stock of over 2 grams of UHRR, which is a mixture of
RNAs from 10 human cell lines. Stratagene has demonstrated that the cell
line based UHRR could be produced at reasonable batch-to-batch
reproducibility. Ambion is fully committed to producing the brain RNA
sample with the highest quality and reasonable quantity (*e.g*., each
batch will be enough for the microarray community to use for about three
years). The regeneration of relatively consistent brain RNA sample was
discussed during the MAQC meeting, and Ambion is going to use a fairly
substantial number of donors to minimize the batch-to-batch variation.

During the session on Titration Strategies, Rich Shippy (GEHC) presented
a strategy of mixing two RNA samples for assessing the accuracy of
microarrays and the bioinformatic approach for visualizing the expected
and observed ratios. Scott Pine (FDA/CDER) described the project led by
Karol Thompson (FDA/CDER) that created two rat mixture RNA samples from
four tissue RNAs with defined ratios. Doug Lane (ViaLogy) described an
ongoing titration study aimed at assessing the performance of individual
probes on the Affymetrix platform.

During the session on Verification with Independent Platforms, Lu Zhang
(Applied Biosystems) described a series of criteria for selecting a
subset of 1,000 genes for TaqMan^®^ verification to cover genes with
appropriate distributions in terms of fold change and intensity. During
the discussion session, Federico Goodsaid (FDA/CDER) stressed the
importance of selecting a certain number of genes that do not show
cross-platform ratio concordance from the pilot datasets. Scott
suggested using RefSeq as the basis for identifying the set of genes
that are commonly probed by each commercial platform used in the MAQC
project. For each selected gene, Applied Biosystems will conduct
TaqMan^®^ verification in four replicates. Yuling Luo (Genospectra)
described the QuantiGene^TM^ assay platform for differential gene
expression analysis. Genospectra will provide QuantiGene^TM^ assay data
for 200-400 genes to be selected by the MAQC group.

**Day 2: 8:00 AM -- 12:00 PM, Tuesday, May 3, 2005**

What to Expect from the MAQC Project

The second day of the MAQC meeting was co-chaired by Wendell Jones
(Expression Analysis) and Uwe Scherf (FDA/CDRH) and was focused on the
design of the MAQC main study. Leming Shi (FDA/NCTR) reiterated the
goals of the MAQC project:

1.  To assess the performance that is achievable by the same microarray
    platform within the same laboratory;

2.  To assess cross-laboratory comparability of the same microarray
    platform;

3.  To assess cross-platform comparability of the microarray technology;

4.  To develop a set of metrics/thresholds for the objective assessment
    of microarray performance;

5.  To assess the advantages and disadvantages of various data analysis
    methods;

6.  To provide practical tools (calibrated reference RNA samples,
    reference datasets, QC metrics/thresholds, orthogonal measurements,
    *etc*.) to microarray end-users (including those not involved in the
    MAQC project) to avoid procedural failures and to ensure laboratory
    proficiency;

7.  To provide a foundation for platform providers to further improve
    the microarray technology.

To better design the MAQC main study, the attendees were given a chance
to express their main expectations from participating in the MAQC
meeting:

1.  Jacques Retief (Affymetrix): To develop reference materials
    available in a tube for the quality control of microarray
    hybridizations;

2.  Yaron Turpaz (Affymetrix): To identify sources of variability so
    that confounding factors can be blocked in experimental design;

3.  Scott Pine (FDA/CDER): Regulatory needs for data analysis,
    proficiency tests;

4.  Paul Wolber (Agilent): How different platforms compare; If not
    comparable, identify the main sources of discrepancies; To establish
    reference materials for QC;

5.  Jim Fuscoe (FDA/NCTR): Reference materials for proficiency tests;
    datasets for developing guidelines for microarray data analysis;

6.  Bud Bromley (ViaLogy): To establish microarray performance baseline;
    To ensure that all labs are operating within the norms;

7.  Mike Wilson (Ambion): How to build microarray "standards";

8.  Susan Lundquist (EPA): To help policy decision and risk assessment;

9.  Federico Goodsaid (FDA/CDER): To adequately address technical
    concerns so that we can move to biology;

10. Don Jin (Gene Logic): To demonstrate that consistent data can be
    generated;

11. Walter Liggett (NIST): To get the sources of variation out;

12. Ron Peterson (Novartis): To build confidence in microarray data for
    regulatory submissions;

13. Uwe Scherf (FDA/CDRH): To help regulatory review of IVD's.

Design of the MAQC Main Study (Microarrays)

1.  A survey was conducted and 22 people favored the use of the
    identical protocol recommended by the manufacturer for the same
    platform, whereas five people voted for using any protocol at the
    test site's choice. A decision was made for all test sites of the
    same platform to use exactly the same protocol recommended by the
    manufacturers.

2.  For the same platform, each test site should use the same chip lot
    and the same reagent/kit lot. Data should be generated by one
    operator per test site.

3.  It was decided not to include "day effect" in the design.

4.  Each platform will be tested at 3 sites to be chosen by the array
    manufacturer.

5.  Each of the two samples (Ambion Brain RNA and Stratagene UHRR) will
    be tested in five replicates at each test site.

6.  Two RNA mixtures will be tested. A subcommittee will be formed to
    decide the details on RNA-mixing. The vendors will test the mixing
    first before the test sites perform testing on mixtures.

7.  Each test site is expected to conduct 20 hybridizations (5
    replicates x 4 samples (Brain, UHRR, and two mixtures), and each
    platform provider is expected to contribute 60 arrays for the MAQC
    main study.

8.  Each hybridization should start from independent cRNA
    synthesis/labeling reactions.

9.  cRNA yield, labeling efficiency, and size distribution (Bioanalyzer
    profile) should be recorded by the test sites and cRNA samples
    should also be sent to Ambion for independent testing. Such
    information is expected to help establish QC metrics/thresholds for
    RNA quality control.

Design of the MAQC Main Study (Validation)

1.  Validation by independent platforms such as TaqMan^®^ and
    QuantiGene^TM^ will be conducted.

2.  RNA titration will be used to assess the behavior of a subset of
    genes.

3.  Based on the mapping of probes to RefSeq, about 8K RefSeq sequences
    are in common to the four platforms used in the MAQC pilot study
    (thanks to Scott for this information).

4.  *In silico* analysis will be conducted to finalize the selection of
    genes for independent validation. *In silico* simulation will also
    be conducted to examine the titration behavior of the brain and UHRR
    samples.

Other Issues

Teleconferences will be scheduled once every two weeks to finalize the
design of the MAQC main study and to keep track of the progress of the
project. The timeline of the MAQC project will be determined by the
availability of the larger batch of brain RNA sample to be manufactured
by Ambion. Ambion is figuring out the logistic details and will keep the
MAQC group updated.

MAQC for rodents (mouse and rat) was listed in the meeting agenda but
was not discussed because of the limitation of time. Because of the
importance of rodents to the EPA and FDA, these issues will be actively
explored.

Organizations should address issues such as MTAs individually when
required.

A list of attendees is shown in Table 2.

Table 2. A list of participants of the MAQC project meeting on May 2-3,
2005

  --------- ------------------ -------------- --------- ------------------ ------------ --------- ------------------ --------------
  **No.**   **Organization**   **Name**       **No.**   **Organization**   **Name**     **No.**   **Organization**   **Name**

  1         Affymetrix         Liu, Chunmei   16        FDA/CDER           Thompson,    31        NIST               Liggett,
                                                                           Karol                                     Walter

  2         Affymetrix         Retief,        17        FDA/CDRH           Elespuru,    32        NIST               Satterfield,
                               Jacques                                     Rosalie                                   Mary

  3         Affymetrix         Turpaz, Yaron  18        FDA/CDRH           Scherf, Uwe  33        Novartis           Peterson, Ron

  4         Agilent            Wolber, Paul   19        FDA/CDRH           Tezak,       34        Stratagene         Clum, Rob
                               K.                                          Zivana                                    

  5         Ambion             Setterquist,   20        FDA/CVM            Harbottle,   35        Stratagene         Fischer, Gavin
                               Robert                                      Heather                                   M.

  6         Ambion             Wilson, Mike   21        FDA/NCTR           Fuscoe,      36        Stratagene         Smith, Cheryl
                                                                           James                                     

  7         Applied Biosystems Lee, Kathy Y.  22        FDA/NCTR           Shi, Leming  37        Umass Boston       Jensen,
                                                                                                                     Roderick

  8         Applied Biosystems Zhang, Lu      23        FDA/NCTR           Tong, Weida  38        Umass Boston       Lombardi,
                                                                                                                     Michael

  9         EPA                Gallagher,     24        GE Healthcare      Shippy,      39        ViaLogy            Bromley, Bud
                               Kathryn                                     Richard                                   

  10        EPA                Lundquist,     25        Gene Logic         Jin, Donald  40        ViaLogy            Lane, Doug
                               Susan                                       F.                                        

  11        Expression         Jones, Wendell 26        Genospectra        Luo, Yuling                               
            Analysis                                                                                                 

  12        FDA/CBER           Han, Jing      27        Genospectra        Ma, Yunqing                               

  13        FDA/CDER           Frueh, Felix   28        Luminex            Calvin, Ted                               
                               W.                                                                                    

  14        FDA/CDER           Goodsaid,      29        NCBI               Herman,                                   
                               Federico                                    Damir                                     

  15        FDA/CDER           Pine, Scott    30        NIH/NCI            Kawasaki,                                 
                                                                           Ernest                                    
  --------- ------------------ -------------- --------- ------------------ ------------ --------- ------------------ --------------

\*Those attending the meeting via teleconference are not listed.