W144 Nucleic Acids Research, 2009, Vol. 37, Web Server issue Figure 2. Parameter estimation. The number of abstracts in the back- ground and training sets has an impact on the ROC area for various biomedical topics. The y-axis shows the mean ROC area after leave- one-out cross validations over 10 random background sets using 1000 training set abstracts (left column), or 10 bootstrapped training sets using the rest of Medline as background set (right column). Additionally, the probability of correct ranking of a random pair of abstracts, one relevant and one irrelevant, is calculated from the area under a ROC curve. This is provided to allow future comparisons with other algo- rithms (Figure 1C). Finally, the list of discriminative words with corresponding weights is given in decreasing order of importance (Figure 1D). Parameter estimation and validations In principle, the larger the size of both training and back- ground sets, the better the precision of MedlineRanker. However, the use of large datasets can result in longer computation times. Fortunately, as we show for various topics (Figure 2) it is possible to obtain approximately optimal ROC area using relatively small training and background sets of one hundred to one thousand abstracts. This number of abstracts is also suggested as a minimum because the performance drops oﬀ sharply when fewer abstracts are used as input. Changes in ROC area between topics reﬂect their heterogeneity of word content. A ﬁrst benchmark of MedlineRanker by manual vali- dation was performed on abstracts related to Arabidopsis thaliana to retrieve host–pathogen interactions. The train- ing and the test set were deﬁned by existing MeSH anno- tations (see ‘Materials and Methods’ for details). The manual evaluation showed only one false positive within the top hundred abstracts (Table 1 and Supplementary Table 1). Furthermore, within this subset, only 17 abstracts were properly tagged with both ‘Arabidopsis’ Table 1. Number of true positives in manually evaluated sets Abstracts selection TOP100 TOP50 TOP25 TOP10 Host–pathogen interactions 99 (99%) 49 (98%) 25 (100%) 10 (100%) Phosphorylation-dependent mechanisms 71 (71%) 41 (82%) 19 (76%) 9 (90%) Manual validations on 200 abstracts. The ranking of two topics was manually validated. The ﬁrst topic, host–pathogen interactions, was used to rank abstracts related to Arabidopsis thaliana. The second topic, phosphorylation-dependent molecular processes, was used to rank abstracts from three PPI databases (HPRD, MINT and DIP). The proportion of true positives was calculated from the manual validation of the best 100 (TOP100), 50 (TOP50), 25 (TOP25) and 10 (TOP10) abstracts. and ‘Host–Pathogen Interactions’ terms in the MeSH database. Although MedlineRanker is not speciﬁcally designed to detect dependency between topics, we illustrate another benchmark of this method in ranking abstracts related to two dependent topics. We applied MedlineRanker to rank abstracts from three public PPI databases according to their relevance to phosphorylation-dependent molecu- lar processes and evaluated manually the results (see ‘Materials and Methods’ for details). Results showed that within 100 evaluated abstracts, the proportion of true positives correlated with the stringency of the selec- tion (Table 1 and Supplementary Table 2). The proportion of true positives increased from 0.71 to 0.90 in the best one hundred and the best 10 abstracts, respectively. We con- sidered the results satisfactory, taking into account that MedlineRanker uses exclusively the word content in the abstract rather than attempting to derive the meaning of it. For example, to determine whether an abstract describes an event of protein phosphorylation depending on a PPI or, conversely, a PPI that depends on an event of protein phosphorylation, requires deeper semantic analysis. DISCUSSION While the biomedical literature contains millions of refer- ences in the Medline database, it is likely that in many particular situations the topics of interest for a given user will be described in thousands or tens of thousands of abstracts and only a handful of those abstracts will be relevant for this user. Finding those relevant abstracts for a given biomedical domain using the main search engine available for Medline, i.e. keyword-based Boolean PubMed, requires a set of words related to the topic of interest and this necessitates domain-speciﬁc knowledge. And then, even for an expert user, dealing with numerous unranked results may be detrimental for the selection of relevant papers. Retrieval of interesting documents can beneﬁt from text mining tools that do not require expert knowledge from the user and that are able to order the results by relevance.