publications are considered as well as when we restrict our analyses to the group that we know has continued a career in academic science. Extent of type I and type II errors in EMBO committee peer review Since in every grant or fellowship peer review process some good proposals are rejected and some bad proposals are accepted due to random error or systematic bias [36], it is instructive to calculate the extent of erroneous decisions [37]. In type I error (also called false positive error), the EMBO selection committee concluded that an applicant had the scientific potential for promotion and was approved, when he or she actually did not, as reflected in an applicant’s low scientific performance subsequent to application. Type I errors lead to the over-estimation of the applicant’s future performance, i.e. the selected applicant will perform on the same level or below the average of the rejected group. In type II error (also called false negative error), the committee concluded that an applicant did not have the scientific potential for promotion and was rejected, when he or she actually did as reflected in a high scientific performance subsequent to application. Type II errors lead to the under-estimation of the applicant’s future performance, i.e. the rejected applicant will perform on the same level or above the average of the selected group [38]. In order to consider both performance measures for each applicant (paper numbers and citation counts) in the determina- tion of the error types for the EMBO peer review process, we used the h index that was recently proposed by Hirsch [39]. This index is an original and simple new measure incorporating both quantity and impact of publications in one single number: ‘‘A scientist has index h if h of his or her Np papers have at least h citations each and the other (Np2h) papers have fewer than h citations each’’ [39, p. 16569]. A series of studies could demonstrate that a scientist’s h index is highly correlated with his or her paper numbers and citation counts [40]. According to Hirsch [39] an h index of 20 after 20 years of scientific activity characterizes a successful scientist. An h index of 40 after 20 years of scientific activity characterizes outstanding scientists, likely to be found only at the top universities or major research laboratories and an h index of 60 after 20 years characterizes truly unique individuals. As the results of Bornmann and Daniel [38,41] show, the h index can not only be used to measure the performance of scientists after a long career, but also that of young scientists. The authors found that the mean h index for successful applicants (arithmetic mean = 3.84, median = 3) for post doctoral research fellowships was statistically significantly higher than the mean h index for non-successful applicants (arithmetic mean = 2.72, median = 2) and that the applicants’ h index values correlate significantly with their publication and citation numbers. The box plots in Figure 3 show the distributions of the applicants’ h index values. In agreement with the results reported above, the median h index for approved applicants is larger than that for rejected applicants, although the h index of both approved and rejected applicants significantly vary around the median values (see the boxes and the outliers in the figure) [42]. Among rejected applicants are scientists who have an h index that is higher than the median value for approved applicants, an indication of type II, i.e. false negative, errors. Among approved applicants we find scientists who have an h index lower than the median value for rejected applicants, an indication of type I, i.e. false positive, errors. For the determination of the extent of type I and type II errors in the peer review we categorized the decision of the selection committee to approve applicants with an h index equal to or Measuring Peer Review Quality smaller than the median value for rejected applicants as type I error. Type II errors were defined as the rejection of applicants with an h index equal to or higher than the median of approved applicants (see Table 5). Based on these definitions, we calculated the extent of type I and type II errors in the peer review processes for the LTF and YI programmes. 54% (LTF programme) and 69% (YI programme) of the committee’s decisions can be called correct according to our definition (see Table 6). The further percentages in the tables clearly reveal that in both programmes the selection committee made type II errors more frequently than type I errors. This means that approximately one-third of the applicants (39% and 28%) was rejected but later went on to demonstrate the same or greater scientific performance than applicants that were approved. Less than one-tenth of the applicants (7% and 3%) was approved but was subsequent not as successful as or on the same level as an ‘‘average’’ rejected applicant. However, when interpreting the frequencies of correct and erroneous decisions, it must be taken into consideration that the extent of errors is generally dependent on the approval and rejection rates of the peer review process [38]. If the rejection rate is low, there is less risk of under-estimation, i.e. type II error. In contrast, if the approval rate is low, only few approvals are at the risk of being over-estimated, i.e. type I error. Due to scarce financial resources on one side and a large number of applicants on the other side, the present grant peer review system is especially open to type II errors [43,44]. With approval rates of 20% (in 1998 for the LTF programme) and 13% (in 2001/2002 for the YI programme), the distributions in Table 6 are therefore hardly surprising. In order to gain an impression of the actual extent of erroneous decisions in the EMBO peer review, we included in Table 6 the proportion of type I errors within the approved group and the proportion of type II errors within the rejection group. The results show that the error rates within approved and rejected groups are between 26% and 48%, whereby again the extent of type II errors exceeds the extent of type I errors in both programme. The tables also point out that the extent of both under- and over-estimations of the applicants’ scientific performance is lower for the YI programme than for the LTF programme. Discussion Since ‘‘peer review can … [build,] jeopardize or destroy research efforts and careers of innovative investigators’’ [45, p. 34] and the advancement of scientific knowledge builds essentially on an efficient peer review system [1], the quality of each peer review process in science is of great importance. In this comprehensive study we investigated the committee peer review performed by EMBO for the selection of post doctoral fellows and young investigators. The results of the regression analyses show that the mean scientific performance of approved applicants is higher subsequent to application than the mean performance of rejected applicants. That means, there is a statistically significant association between selection decisions and the applicants’ scientific achievements, if quantity and impact of research publications are used as a criterion for scientific achievement. However, as the results of the regression analyses have not been validated with independent data, there is a need for validation to generalize the findings. In the interpretation of the results of the regression analyses it cannot be ruled out that the applicants who received funding from EMBO may have published more subsequent to application because they received funding and not necessarily because the PLoS ONE | www.plosone.org 7 October 2008 | Volume 3 | Issue 10 | e3480