e79 Nucleic Acids Research, 2009, Vol. 37, No. 11 PAGE 6 OF 11 Figure 1. The relationship between annotation bias and gene age. (a) 52 gene lists from the HG-U133A chip were collated from published lit- erature and for each of these equivalently sized random gene lists were created. The numbers of PMIDs associated with them were calculated and plotted against the size of the gene lists. Both axes are on loga- rithmic scale. (b) A mean age was calculated for each of the 52 litera- ture gene lists by averaging the consensus ages of its constituent genes. Fold-change in PMID was calculated by dividing the number of PMIDs associated with a literature gene list by the average PMID count in an equivalently sized random gene list. The vertical dashed line represents the mean age of a random gene list, which is 1996 in this case; the horizontal dashed line represents the level at which there is no diﬀerence between the numbers of PMIDs associated with the literature and random gene lists. there is an excess of annotation inherent with highly anno- tated gene lists. This in turn provides an explanation for the distorted token frequency distribution and skewed hypergeometric P-values that result in a mixture of speciﬁc and non-informative terms to be called signiﬁcantly over- represented by the classical hypergeometric distribution- based ORA approach. Annotation bias and consensus gene age A ‘well-studied’ gene may in part reﬂect one that has been known for many years, thus allowing a substantial corpus of literature regarding it to be accumulated. To investigate the possible eﬀect of this aspect of the history of recent scientiﬁc research we determined the ‘age’ of each gene represented on the HG-U133A chip. Here, gene age is deﬁned as an approximate measure of how long a gene has been known and researched upon relative to other genes, and should not be confused with the molecular timescale of evolution in the genome. We derived a con- sensus age for each gene represented on the HG-U133A chip based on two criteria: (1) when the gene was ﬁrst cited in the published literature and (2) when the gene was ﬁrst integrated into the OMIM and HGNC public databases. The consensus gene age was computed as the average of these two measures (see ‘Materials and Methods’ section for more details), and ranges from year 1939 to 2007. In the context of this analysis, a gene with a consensus age of 1990 implies that it was discovered in approximately 1990, so it is considered older and has been studied for longer compared to a gene with a consensus age of 1998. We stratiﬁed all the genes by their consensus age and compared the amount of PMIDs associated with them. As expected, younger genes that have only recently been described have markedly fewer PMIDs associated with them; whereas older genes are generally better studied and cited by more PMIDs (Supplementary Data 1 Figure S1). This eﬀect seen on individual genes is also translated into an eﬀect on the mean age of genes in bio- logically derived gene lists. As can be seen in Figure 1b, there is a strong trend whereby those gene lists that show excess PMID annotation are also those whose constituent genes have been known for longer. Overcoming annotation bias by permutation test Our initial attempt to address the eﬀects of annotation bias used a permutation test that makes no assumptions about the underlying data distribution. The signiﬁcance of a token was assessed by calculating an empirical P-value based on the creation of 100 000 random gene lists, each of which was matched for the number of genes and the amount of associated PMIDs. Tokens with List frequency equal to 1 were removed before they were subjected to permutation test because a token can only be useful in deﬁning relationships among genes if it is shared by at least two of them. After this ﬁltering, 4840 tokens remained for testing. This approach produced an improvement over the classical hypergeometric distribution-based approach when tested on the ISG gene list, insofar as it successfully retained those biologically plausible terms such as ‘inter- feron’, ‘IFN’, ‘antiviral’, whilst no longer called those less-speciﬁc terms as signiﬁcant (Table 2). However, one limitation of this approach is the precision of the P-values, the smallest of which is determined by the number of