present study (see Figure 1). These applicants published a total of 6,087 papers (articles, letters, notes, and reviews) prior to application (publication window: from 1984 to the application year in 2001 or 2002) and 3,632 papers subsequent to application (publication window: from the application year in 2001 or 2002 to the beginning of 2007). The papers published prior to application received an average of 46.56 citations (median = 23) and the papers published subsequent to application an average of 11.15 citations (median = 4) (citation window: from publication year to the beginning of 2007). In the citation search for the applicants’ papers we included self- citations, because (1) it is not expected that the number of self- citations varies systematically for the papers published by the approved and rejected applicants, and (2) the number of self- citations of a publication can be modeled in the multiple regression analysis (the results of which are reported in the following) using the number of authors of a manuscript [22]. As Herbertz [23] shows, a greater number of authors is associated with a greater number of self-citations of a publication [24]. The bibliographic data of the applicants’ papers (published prior and subsequent to application) were taken from the SCI and were double-checked in the Medline database (provided by the National Library of Medicine, NLM, Bethesda, MD, USA) and with the applicants’ lists of publications. For the careful process of evaluation and cleaning, the bibliographic data were imported into a FileMaker database and matched to the information arising from the EMBO selection process (e.g., the committee’s decision) [25]. To undertake the statistical analyses, two datasets (one for the LTF applicants and the other for the YIP applicants) were exported from the database to the statistical package Stata [26]. By using these datasets, the relationship between the judgments of the EMBO selection committee (approval or rejection of applications) and standard bibliometric indicators was evaluated in hindsight of the committee’s decisions. In other words, we evaluated the committee’s decisions with the following bibliometric indicators: (1) number of papers that were published subsequent to application, (2) citation counts for papers that were published prior and (3) subsequent to application. Statistical procedure Bibliometric studies have demonstrated that factors other than scientific quality have a general influence on citation counts [15]: Citation counts are affected by the number of co-authors [27] and the length [28] of a paper as well as the size of the citation window [29]. That means there is a positive correlation between citation counts and the number of co-authors and the size of a paper as well as the length of the citation window. By considering these factors in the statistical analysis, it becomes possible to establish a meaningful and adjusted co-variation between decisions made by peer review and the bibliometric data gathered for the applicants. We performed six multiple regression analyses (three for each programme), which reveal the factors that exert a primary influence on the number of papers published and citation counts. Both models predicting citation counts took the number of pages and the number of co-authors of each paper as independent variables into account besides the decision variable (dichotomous variable: 0 = rejected, 1 = approved). The publication years of the papers were included in the models predicting citation counts as exposure time [30, pp. 370–372]. We used the exposure option provided in the statistical package Stata [26] to take into account the time that a paper is available for citation. The violation of the assumption of independent observations by including citation counts of more than one paper per applicant was considered in the models by using the cluster option in Stata. This option specifies that the citation counts are independent across papers of different applicants, but are not Measuring Peer Review Quality necessarily independent within papers of the same applicant [31, section 8.3]. For each of the independent variables included in the regression models, we checked for the presence of multicollinearity by calculating variance inflation factors and tolerances [32]. The results of these analyses showed no evidences of multicollinearity. Both outcome variables (number of papers and citations) are count variables. They indicate ‘‘how many times something has happened’’ [30, p. 350]. The Poisson distribution is often used to model information on counts. However, this distribution rarely fits in the statistical analysis of bibliometric data, due to overdispersion. ‘‘That is, the [Poisson] model underfits the amount of dispersion in the outcome’’ [30, p. 372]. Since the standard model to account for overdispersion is the negative binomial [33], we calculated in the present study negative binomial regression models (NBRMs) [34]. A second type of problem in the statistical analysis of count data occurs ‘‘when observations with outcomes equal to zero are missing from the sample because of the way the data were collected’’ [30, p. 381]. The statistical analysis of citation counts in the present study is based on a sample of those applicants who published at least one paper. Non-publishers were excluded, because they had not published any paper that could have been cited. Since zero-truncated count models (or zero-truncated negative binomial models, ZTNBMs) are designed for data ‘‘in which observations with an outcome of zero have been excluded from the sample’’ [30, p. 382], we calculated this model type if non-publishers were among the applicants in the sample (it was a necessary requirement for the model calculation to add the value 1 to each citation number to avoid zero citations). The publication and citation data gathered for the applicants were analyzed using cycles of model specification, estimation, testing, and evaluation. We began with Poisson and then tested for negative binomial. Testing and evaluation include residual analyses and goodness-of-fit measures [35]. Results Did the EMBO peer review process actually achieve its goal of selecting the best young scientists? The findings in Figure 2 do not provide clear evidence that it did. The figure shows box plots for number of papers published subsequent to application (graphs A and D), univariate distributions of the median number of citations per paper per year published prior to application (graphs B and E) and univariate distributions of the median number of citations per paper per year published subsequent to application (graphs C and F). The distributions in each graph of the figure are presented separately for approved and rejected LTF and YI programmes applicants. Graph B shows, for example, that each of the papers published in 1993 by approved LTF applicants received a median of 21 citations, whereas each of the papers published in 1993 by rejected applicants received a median of 18 citations since publication until 2006. Even if in Figure 2 (1) for every publication year, the papers published by the approved LTF applicants prior to application were more often cited than papers published by the rejected applicants (graph B) and (2) approved LTF and YI applicants had published more papers subsequent to application than rejected LTF and YI applicants (graphs A and D), the median citation counts for the papers published subsequent (both programmes, graphs C and F) and prior (YI programme, graph E) to application do not demonstrate this consistent trend of an advancement for approved applicants. Regression analyses based on bibliometric data for the applicants to the LTF programme Table 1 shows a description of the variables that were included in the (zero-truncated) negative binomial regression models PLoS ONE | www.plosone.org 3 October 2008 | Volume 3 | Issue 10 | e3480