e79 Nucleic Acids Research, 2009, Vol. 37, No. 11 PAGE 8 OF 11 Figure 2. A scatter plot of Chip versus List frequencies for tokens in the ISG gene list. Each data point represents an abstract term. Terms that were identiﬁed as signiﬁcantly enriched (i.e. Bonferroni P 0.05) in the ISG gene list by using the Outlier method are circled and the adjacent numbers corresponding to their rankings. Chip (y-axis) represents the number of genes associated with each term on the whole chip. List (x-axis) represents the number of genes associated with each term in the ISG gene list. The log 2-transformed List and Chip frequencies are plotted. ExtendedHG: identify over-represented terms in PubMed abstracts using the extended hypergeometric test The second method, called ExtendedHG, is a parametric approach based on the extended hypergeometric distribu- tion. The key concept underlying this approach is that annotation bias will cause common, non-speciﬁc terms to have higher probabilities of being selected than expected by chance. Therefore the sampling procedure is biased, with the token frequency distribution following the extended hypergeometric distribution. The degree of bias is measured by the odds ratio, which is equivalent to the probability ratio of seeing a token of interest over other tokens simply by chance, and a P-value for each token can therefore be calculated. ExtendedHG produced results similar to the permuta- tion test (Table 2). All 27 tokens identiﬁed as over- represented in the ISG gene list by ExtendedHG are bio- logically plausible, while those non-speciﬁc words that were called signiﬁcant by the classical hypergeometric test-based approach such as ‘synthesis’, ‘molecule’, ‘after’ were not selected. As with Outlier, a typical runtime for ExtendedHG is 20–30 s when applied to a 500-gene list. A comparison of the rankings between the top 100 most signiﬁcant terms in ISG gene list shows that, despite minor diﬀerences in rank order, there is a good concor- dance between Outlier and ExtendedHG (Supplementary Data 1 Figure S2). False positive rates under the hypothesis From the speciﬁcity perspective, an ideal ORA method should not ﬁnd any signiﬁcant terms in a random gene list. To estimate the false positive rates associated with the proposed methods, we created 1000 random gene lists by randomly sampling 50–2000 unique genes from the HG-U133A array and analysed them with Outlier and ExtendedHG. The false positive rate of Outlier ranges from 0.18 to 1.84, with shorter gene lists (<300 genes) being more susceptible to false positives. This is because the Z-scores distribution from the outlier detec- tion procedure tends to show a slight negative skewness for short gene lists, but is closer to being normally distrib- uted for longer gene lists (see Supplementary Data 4). ExtendedHG shows a low false positive rate (<0.01) even for short gene lists. Comparison with existing ORA tools Using the ISG gene list as the benchmarking dataset, we observed a good agreement between the biology associated with the enriched GO terms reported by the functional annotation tool DAVID 2.0 (http://david. abcc.ncifcrf.gov/home.jsp) and the PubMed abstract terms produced by our methods, with concepts related to immune response highly ranked by both approaches (Tables 2 and 3). As an illustration of the limitations of ontologies such as GO we noted that none of the signiﬁ- cant GO terms gives an indication of the involvement of interferon, thus demonstrating how mining of PubMed abstracts can potentially reveal additional biological insight that is not possible by mining pre-deﬁned ontologies alone. Performance across different species Outlier and ExtendedHG can be readily extended to other species for which an associated corpus of PubMed