Table 2. A comparison of the results from diﬀerent methods when applied to the ISG gene list PAGE 7 OF 11 Abstract term INTERFERON IFN IFN-BETA ANTIVIRAL IFN-ALPHA INTERFERON-ALPHA OLIGOADENYLATE ISG ISRE DSRNA HLA-CLASS HLA-A HLA-B INDUCIBLE ENCEPHALOMYOCARDITIS STOMATITIS OAS HLA-G MXA EVASION INNATE TAPASIN VIRAL INFECTION OR-C LYMPHOBLASTOID IFN-GAMMA IMMUNITY IMMUNE TREAT MHC VIRUS Nucleic Acids Research, 2009, Vol. 37, No. 11 e79 Chip 414 245 71 176 114 59 18 14 31 60 11 30 25 1068 16 52 10 10 11 65 363 12 892 1177 5 239 443 387 1275 1817 353 1408 List 46 35 18 23 19 14 8 7 9 11 6 8 7 37 6 9 5 5 5 9 21 5 32 36 4 16 22 20 35 40 17 34 Bonferroni P-value Permutation <0.0484 <0.0484 <0.0484 <0.0484 <0.0484 <0.0484 <0.0484 <0.0484 <0.0484 <0.0484 <0.0484 1 1 <0.0484 <0.0484 <0.0484 0.0968 1 1 <0.0484 0.0484 1 0.0484 0.0968 <0.0484 <0.0484 0.1936 0.0968 0.6776 <0.0484 1 1 Outlier 9.81 10 34 8.90 10 19 5.60 10 11 2.24 10 8 3.00 10 8 6.08 10 8 2.26 10 6 1.41 10 5 2.06 10 5 0.0001 0.0002 0.0003 0.0024 0.0025 0.0036 0.0036 0.0145 0.0145 0.0257 0.0258 0.0311 0.0427 0.0447 0.0477 0.0717 0.0872 0.1113 0.1799 0.4363 0.9165 1 1 ExtendedHG 2.12 10 31 1.35 10 25 1.34 10 14 1.03 10 13 2.49 10 12 5.85 10 10 8.29 10 6 9.05 10 5 1.62 10 5 1.03 10 5 0.0015 0.0005 0.0048 1.29 10 7 0.0136 0.0013 0.1047 0.1047 0.1624 0.0074 1.35 10 5 0.2407 4.25 10 6 9.61 10 6 0.5133 0.0004 6.38 10 5 0.0002 0.0003 0.0033 0.0115 0.0089 Abstract terms that are identiﬁed as over-represented by the corresponding method are shown in bold. The cutoﬀ used is P 0.05 after Bonferroni correction. 100 000 randomisations were performed for the permutation test. 4840 tokens were being tested during the permutation test and the best possible Bonferroni P-value attainable is 10–5 4840 = 0.0484. Any term with an empirical P-value less than 10–5 is provisionally assigned a value of <10–5, and the corresponding Bonferroni P-value is set to be <0.0484. The unadjusted P-values and other details (e.g. Z-scores and odds ratio) can be found in Supplementary Data 1, Table S1. Chip is the number of genes that contains a given term on the entire chip; List is the number of genes that are associated with a given term in the ISG gene list. permutations carried out. In this analysis, the best possible Bonferroni P-value attainable is 10–5 4840 = 0.0484. This could be improved by increasing the number of per- mutations, but as with many permutation-based methods, this approach is extremely computationally intensive, requiring six hours on a standard desktop computer to analyse the ISG gene list. To address this issue we devel- oped two computationally eﬃcient methods capable of handling the annotation bias problem. Outlier: an empirical outlier detection method for finding over-represented terms in PubMed abstracts This method exploits observations made when plotting the number of genes that contains a token of interest on the entire chip (the Chip frequency) versus the number of genes that are associated with a token in the query gene list (the List frequency). On such plots (see Figure 2 for an example) we observed that there are typically a set of biologically plausible tokens that lie far away from the main data cluster formed by the remaining tokens and appear as outliers. We reasoned that most tokens in a gene list would be largely irrelevant to the biology under study (e.g. common English words), thus forming the background cloud of points, with enriched and thus potentially interesting tokens appearing as outliers. We further reasoned that the annotation bias eﬀect should, in principle, also aﬀect the background distribution, and thus an outlier-detection approach may intrinsically compensate for the underlying annotation bias. To test this we developed an outlier detection method denoted here as Outlier. The detailed methodology is described in the Materials and Methods section but brieﬂy, we use a plot such as Figure 2 and calculate a Z-score for each token based on the local Chip mean and standard deviation. Using this method, 24 tokens were called signiﬁcantly over-represented in the ISG gene list (Table 2), all of which appear relevant to the biology of an interferon-regulated response and are very similar to those tokens produced by the permutation- based method. However, Outlier is much more computa- tionally eﬃcient than the permutation-based test (typically runtimes of 20–30 s on a desktop PC).