e79 Nucleic Acids Research, 2009, Vol. 37, No. 11 dramatic and hitherto underappreciated feature—gene lists derived from a typical microarray experiment tend to have more annotation (i.e. PubMed abstracts) asso- ciated with them than would be expected by chance. This bias can lead to a marked over-representation of many common (and likely uninformative) terms, inter- spersed with terms that appear to convey real biological insight. We have developed several solutions to this issue. The ﬁrst is based on the use of a permutation test, but is hampered by being computationally intensive. Therefore two computationally tractable approaches for performing ORA mining on PubMed abstracts, based on the detection of outliers and the extended hypergeometric distribution, were developed. Here, we describe the unique features of these methods and illustrate their utility by applying them to several diverse biological datasets. MATERIALS AND METHODS Public datasets We used publicly available microarray datasets to evaluate the performance of the ORA methods described in this work. In total, 354 diﬀerent gene lists were collected from 146 scientiﬁc papers, which cover experiments performed on 10 major Aﬀymetrix platforms, including HG-U133A, HG-U133 Plus 2.0, Mouse 430 2.0, Rat 230 2.0, Arabidopsis ATH1, DrosGenome1, Drosophila 2.0, Xenopus laevis, C. elegans and Zebraﬁsh. These gene lists are collectively referred to as the ‘literature gene lists’ and their details can be found in Supplementary Data 2. Two gene lists were selected to evaluate in more detail the performance of the ORA methods presented here: (1) ISG gene list: This gene list was extracted from the gene expression study of Sanda et al. (6). We applied the regularised t-test method (7) to MAS5 expression data and used the false discovery rate (FDR) method to correct for multiple hypothesis testing. Genes that were diﬀerentially regulated following treatment with type I interferon at both 6 and 24 h were identiﬁed (FDR P 0.05). This produces a gene list consisting of 77 interferon-stimulated genes (ISG). (2) Nishimura gene list: This gene list was as reported in Nishimura et al. (8). It contains 685 probesets on the Aﬀymetrix Arabidopsis ATH1 array, representing 679 diﬀerent genes that were diﬀerentially expressed in pmr4 mutant relative to wild-type plants. Text corpus creation The methods described here require the initial creation of a text corpus that connects the textual information stored in PubMed abstracts with genes included in the micro- array analysis. First, we mapped all the genes represented on an array to their corresponding EntrezGene identiﬁers (EGID) based on the mapping schemes provided by the appropriate Bioconductor metadata packages. Then, PubMed articles associated with these genes were obtained from the gene2pubmed ﬁle downloaded from NCBI PAGE 2 OF 11 (ftp://ftp.ncbi.nih.gov/gene/DATA; time stamp: 25 October 2007) in the form of EGID to PubMed identiﬁer (PMID) mappings. PMIDs that are associated with more than one EGID were omitted because, based on manual inspection, these tend to be large-scale sequencing reports that contain information largely irrelevant to gene func- tion. This lack of speciﬁcity can aﬀect the performance of the text mining algorithms. PubMed articles passing this criterion were retrieved from the PubMed database using a customised Perl script implementing modules from the Entrez Programming Utilities (http://eutils. ncbi.nlm.nih.gov/entrez/query/static/eutils_help.html). Upon retrieval, the abstracts were tokenised on white space to produce single-word terms. Any redundancies were removed to produce a unique set of tokens for each gene. Terms composed exclusively of numbers were removed from the text corpus. Then, a simple stemming operation was applied to reduce plural to singular forms (e.g. ‘kinases’ becomes ‘kinase’). Verb tenses were stemmed to their root (e.g. ‘phosphorylates’ and ‘phos- phorylated’ become ‘phosphorylate’). Other more elabo- rate analysis of spelling variants (e.g. catalyze, catalyse) and composite words (e.g. cell cycle, DNA polymerase) were not explored. Porter’s stemming algorithm (http:// tartarus.org/ martin/PorterStemmer/; Perl version, release 1) was adapted for this analysis. Definitions of Chip and List frequencies For each token associated with a given gene list, we cal- culated two values. The ﬁrst one, called Chip frequency, is deﬁned as the number of genes that contains the token of interest on the entire chip (i.e. background). The second value, called List frequency, represents the number of genes that are associated with the token of interest in the query gene list. Classical hypergeometric distribution-based ORA approach Suppose that the total number of genes in the background population is N, of which M are associated with a certain token of interest T. If we select K genes randomly from the entire microarray without replacement, the probability of seeing exactly x genes associated with T in K can be modelled by the hypergeometric distribution (9). Hence, the probability of seeing x or more genes containing token T in a random gene list of K genes can be calculated as the cumulative probability: p ¼ 1 x X i¼0 M i N K N K M i This is a one-sided test for over-representation. In this study, a token is considered signiﬁcantly enriched if its P-value is less than 0.05 after adjusting for multiple hypothesis testing (i.e. at 95% conﬁdence level). Permutation test The fundamental idea underlying this permutation approach is to create random gene list that matches the