e79 Nucleic Acids Research, 2009, Vol. 37, No. 11 for less well-annotated species by using the text-based ORA approaches described here. DISCUSSION We have presented several approaches for mining litera- ture-based information associated with a list of diﬀeren- tially expressed genes (DEG) and to search within them for terms or biological concepts that are signiﬁcantly over- represented. Our initial explorations using the classical hypergeometric distribution revealed a hitherto unex- pected bias in the degree of PubMed annotation asso- ciated with gene lists derived from ‘real’ biological experiments. We hypothesised that gene lists generated from real-life biological experiments are likely to be biased towards older genes (i.e. known for a longer period of time) from more established areas of biology. Indeed, as shown in Figure 1b, most literature-derived gene lists have an overall consensus age that is older than the mean age of a random gene list (in this case 1996). It thus seems that gene lists derived from a typical microarray experiment tends to favour groups of genes and areas of biology that have been studied for longer and have a greater amount of associated published litera- ture. Whilst giving an insight into trends within biological research and the progress of scientiﬁc endeavour, the con- sequence for text-based ORA approaches is an annotation bias that has a negative eﬀect on the performance of simple hypergeometric-based approaches. Both Blaschke et al. (12), Khatri and Draghici (13) have pointed out that such bias constitutes a real problem and should be taken into account during enrichment analysis. However, no solution has been proposed to address this problem and it has generally been overlooked by existing ontological tools that implement ORA. Although illustrated here using PubMed tokens, such bias may have a similar inﬂu- ence on other ORA-based functional analysis tools that mine diﬀerent annotation resources. To address this annotation bias we have implemented three diﬀerent approaches to ORA using PubMed tokens, based on a permutation test, an outlier detection method, and the use of the extended hypergeometric distribution. The latter two are computational tractable and this enabled us to benchmark them against 354 literature- derived gene lists. We ﬁnd that tokens plausibly relevant to each study are often called signiﬁcantly enriched, whilst the apparent over-representation of common terms due to annotation bias are successfully avoided. The results produced by the proposed methods generally show a good concordance in most analyses that we have per- formed. These tools provided a similar but distinct insight into the themes over-represented in a gene list compared to the results from undertaking ORA using GO terms, and can be successfully applied not only to well-annotated spe- cies but also to model species such as Arabidopsis. Evaluating the performance of any exploratory approach such as those proposed herein is a challenging task because it is diﬃcult to ﬁnd datasets for which the ground truth is known. We have therefore undertaken a more focused approach to assess the performance of the PAGE 10 OF 11 proposed methods. Speciﬁcally, we focused on gene lists based on the HG-U133A array, and compared the outcome from Outlier and ExtendedHG with those obtained from a standard ORA approach that mines GO terms. The biological relevance and plausibility of the over-represented tokens and GO terms were then assessed against the perceived biology of the original pub- lication. These set of data are presented as Supplementary Data 3. For literature gene lists derived from other arrays, their token- and GO-based ORA results are readily acces- sible via our website for review by researchers with the relevant biological background. Several groups have undertaken the challenge of incor- porating literature-based information into data mining algorithms to interpret the underlying biological signiﬁ- cance of a list of DEGs (12,14,15); their approaches diﬀer fundamentally from our methods. The closest in spirit to ours is the GEISHA (Gene Expression Information System for Human Analysis) system devel- oped by Blaschke et al., which evaluates the signiﬁcance of terms associated with a gene cluster by comparing their frequency of abstracts with the frequency of abstracts containing these terms in diﬀerent gene clusters. However, the online version of this system was only imple- mented for E. coli and yeast. Therefore, it has not been possible to perform a direct comparison between this tool and our methods. Like other ORA approaches, our methods require an initial selection of DEGs by an arbitrarily chosen cut-oﬀ threshold. A major criticism to such ‘threshold-based’ approach is that diﬀerent choices of the cut-oﬀ value will produce diﬀerent lists of DEGs and alter the result of the enrichment analysis. Moreover, many genes with moderate but meaningful expression changes may be dis- carded by the selected threshold regardless of their relative position in the ranked list, leading to a loss in statistical power. In recent years, an alternative mode of analysis that does not involve an initial gene selection step has been proposed. Examples of these include Gene Set Enrichment Analysis (GSEA) (16–18) and Functional Class Scoring (FCS) (19). These methods consider the dis- tribution of a functionally deﬁned group of genes in the ranked list of genes and allow adjustments for their cor- relation structure. While a few studies have shown that such threshold-free approach enables the detection of more subtle functional categories that were overlooked by ORA (19,20), Manoli and coworkers (21) found that ORA produced more consistent results than GSEA with respect to the concordance between analyses on DEG obtained by diﬀerent statistical methods from three pros- tate cancer data sets. Although it would be computation- ally challenging in scale, it may be possible to develop threshold-free methods that can accommodate annotation bias and thus be applied to the mining of PubMed tokens and we are currently exploring this question. The methods described here depend on a corpus of articles relevant to the genes being studied (e.g. all genes appearing on an array), and an index that links the articles to the appropriate genes. We used the manually curated citations provided by NCBI to retrieve the relevant gene- related PubMed abstracts. Although such curation