PAGE 3 OF 11 experimentally derived gene list not only in the number of genes but also the amount of associated PMIDs. This is achieved by replacing the abstracts for each gene in the experimentally derived gene list with other abstracts selected randomly (without replacement) from the text corpus. As such, the number of genes and abstracts (hence PMID) for each random gene list are kept the same as that of the experimentally derived gene list. This permutation procedure is repeated n = 100 000 times. An empirical P-value of over-representation can then be calculated for each token in the gene list as the fraction of times its frequency in the random gene list (r) is equal to or greater than that seen in the experimentally deﬁned gene list (x): P value ¼ ðNumber of permutations for which ri n xiÞ : In this study a token is considered signiﬁcantly enriched, if its P-value is less than 0.05 after Bonferroni multiple hypothesis correction. Outlier: Outlier detection-based ORA This method is motivated by the observation that, on a scatter plot of Chip versus List frequencies, there are a set of biologically plausible terms that deviate substan- tially from the main data cluster and appear as outliers (see Figure 2 for an example). An explanation underlying this observation is that the majority of the tokens in a gene list will not be over-represented because they are either: (i) common words or non-speciﬁc biological terms that are shared by most abstracts, or (ii) rare words that are not biologically interesting. These will form the main cloud of data points. On the other hand, biologically plausible terms are associated with more genes in the gene list and have diﬀerent token frequency distributions, thus appear to be separated from other observations in the background. We developed an outlier detection procedure that determines within a group of tokens corresponding to the same List frequencies any tokens that have lower than expected Chip frequencies. Formally, we derive a Z-score for each token based on its Chip frequency and infer the statistical signiﬁcance of this Z-score against the normal distribution as follows. (1) Local mean and standard deviation (SD) estimation: The Chip and List frequencies were log-transformed to base 2. Then, all tokens in the gene list were stratiﬁed into groups according to their List frequen- cies. Consider a group of n tokens corresponding to List frequency L and their Chip frequencies are given by (x1, . . ., xn). For this group of tokens, the para- meters that deﬁne the outlier region, i.e. mean and SD, are estimated based on the following: If n 10, mean and SD are calculated directly from (x1, . . . , xn). If 1 < n < 10, we borrow information from neigh- bouring observations to give a better estimation of the local mean and SD. This is done by capturing (10 – n) tokens from the adjacent group for which the Nucleic Acids Research, 2009, Vol. 37, No. 11 e79 corresponding List frequency is less than L and com- bine the Chip frequencies from them with (x1, . . . , xn). Then, a local mean and SD are derived from the combined data. (2) Local mean and SD smoothing: The means and SDs estimated in step (1) are smoothed as a function of List frequency by ﬁtting polynomial curves to the frequency data (see panels (a) and (b) in Supplementary Data 1 Figure S5). Locally smoothed mean and SD are computed for each group based on the ﬁtted values derived from the best-ﬁtting curves. The purpose of smoothing is to stabilise the variance in the data so that we can ﬁnd representative mean and SD for calculating Z-score. The eﬀect of smoothing is shown in panels (c) and (d) in Supplementary Data 1 Figure S5. (3) Z-score and P-value calculation: The Z-score for token i is Zi ¼ x ð i sdi iÞ , where x is the Chip frequency, is the locally smoothed mean, and sd is the locally smoothed SD. The Z-score reﬂects the number of standard devia- tions an observed Chip frequency is above or below the local mean. A P-value was calculated from the Z-score using the normal distribution (for a justiﬁca- tion for this approach see Supplementary Data 4). A token is labelled as an outlier and considered over-represented in the gene list if it has a signiﬁcant P-value (Bonferroni P 0.05) and a negative Z-score (indicative of a lower Chip frequency than the local mean). ExtendedHG: extended hypergeometric distribution-based ORA The extended hypergeometric distribution, also known as the Fisher non-central hypergeometric distribution, is a generalization of the classical hypergeometric distribution where the sampling procedure is biased (9–11). Assume that we draw out n balls without replacement from an urn containing N balls, of which m1 are red and m2 are white. The balls have diﬀerent weights, where the weight for each red and white ball is w1 and w2, respectively. When sampling is unbiased (i.e. w1 = w2), the balls have equal probability of being taken (i.e. p1 = p2) and the results will follow the classical hypergeometric distribu- tion. However, if sampling is biased such that the proba- bility of taking ball of one colour is proportional to its weight but independent of the other balls, then the number of balls of a particular colour drawn will follow the bino- mial distribution: xi binomialðmi, piÞ,i ¼ 1,2: On the condition that the sum of the independent bino- mial variables is ﬁxed (i.e. P xi ¼ n), the number of red balls in our sample x will follow the extended