Measuring Peer Review Quality Table 5. Type I and type II errors as well as correct decisions in EMBO peer review. Applicant’s scientific output Applicant’s scientific output is high Applicant’s scientific output is low doi:10.1371/journal.pone.0003480.t005 Decision of the selection committee Approval Correct: the h index is higher than the median h index for rejected applicants Type I error: the h index is equal to or lower than the median h index for rejected applicants Rejection Type II error: the h index is equal to or higher than the median h index for approved applicants Correct: the h index is lower than the median h index for approved applicants above the average selected applicant). The magnitude of the under-estimation error (type II error) is a function of the success rate, i.e. scarce funding will lead to the rejection of a sizable number of worthy candidates, or reversely, an increase in success rate will reduce this error type, while increasing the risk of over- estimation (type I errors). In fact, reducing one cause for one error type (e.g., by increasing the approval rate) automatically increases the risk for the other error type. Not surprisingly both types of errors are smaller for the YI programme. 3% of the applicants have been over-estimated vs. 28% who have been under- estimated, indicating that it is easier to predict the future performance of more advanced scientists. This decrease in error rates is most likely due to the longer publication history of advanced scientists and the resulting improved view on the consistency of results produced by the scientist under evaluation. We should also note that the applicants to the EMBO programmes are not representative of the respective post doctoral and young group leader communities at large, since they have to fulfill stringent eligibility criteria that already pre-select for high performers. Applicants to the post doctoral fellowships must have published at least one first author article in an international peer- reviewed journal, and applicants to the YI programme must have published at least one last author publication from their own independent laboratory, thereby demonstrating the ability to produce and publish independent research results. It is therefore not surprising that, given the low success rates for both programmes, the selection procedure tends to underestimate a substantial percentage of applicants. Our review of the literature revealed that other studies on peer review also report the occurrence of errors of this kind in selection decisions. Thorngate, Faregh, and Young [44], for example, Table 6. Proportions of type I and type II errors in the decisions of the EMBO peer review for the LTF and YI programmes. Error type Correct decision Type I Type II Total Type I Type II Errors among approvals Errors among rejections 362 48 258 668 48 258 doi:10.1371/journal.pone.0003480.t006 LTF programme YI programme absolute in percent absolute in percent 54 7 39 100 37 (n = 130) 48 (n = 538) 204 10 83 297 10 83 69 3 28 100 26 (n = 39) 32 (n = 258) comments as follows on the grants peer review of the Canadian Institutes of Health Research (CIHR, Ottawa): ‘‘Some of the losing proposals are truly bad, but not all; many of the rejected proposals are no worse than many of the funded ones … When proposals are abundant and money is scarce, the vast majority of putative funding errors are exclusory; a large number of proposals are rejected that are statistically indistinguishable from an equal number accepted’’ (p. 3). According to Cole [11], the two types of errors can also take place in the journal peer review process: leaving aside speculation regarding the number of articles submitted versus available space for journal publication in the natural and social sciences, respectively, ‘‘physics journals prefer to make ‘Type I’ errors of accepting unimportant work rather than ‘Type II’ errors of rejecting potentially important work. This policy often leads to the publication of trivial articles with little or no theoretical significance, deficits which are frequently cited by referees in social science fields in rejecting articles. Other fields, such as sociology in the United States, follow a norm of rejecting an article unless it represents a significant contribution to knowledge. Sociologists prefer to make Type II errors’’ (p. 114). We are aware of only four studies that investigated the quality of peer review for the selection of young scientists, only one of which included an analysis of the subsequent publication output of the applicants [46]: Melin and Danell [47] examined the peer review process for the Individual Grant for the Advancement of Research Leaders (INGVAR) of the Swedish Foundation for Strategic Research (SSF, Stockholm). Their analyses of the ‘‘publication histories’’ of 40 applicants show – in contrast to the results of the present study – only slight mean differences in scientific productivity between approved and rejected applicants. Similar results are reported by van den Besselaar and Leydesdorff [48] who evaluated the peer review process of the council for social scientific research of the Netherlands Organization for Scientific Research (Den Haag). However, the results of both studies are not directly comparable since they focused on highly selected applicants, i.e. besides the approved only the best rejected applicants. Large performance differences between accepted and rejected applicants would have been a surprise for these samples. Bornmann and Daniel [38,41,49] investigated committee peer review for the post doctoral fellowship programme of the Boehringer Ingelheim Fonds (B.I.F.). The authors analysed the bibliometric performance of close to 400 applicants prior to application. The results are in agreement with the findings of the present study. Hornbostel et al. [46] studied applications to the German Research Foundation’s (DFG, Bonn) Emmy Noether programme. The programme funds young researchers in the late post doctoral and early group leader phase. The results show only minor differences in number of publications and citation counts between approved and rejected applicants. It can be speculated that the high success rate of applications (52%) in combination with stringent eligibility requirements have contributed to this result. PLoS ONE | www.plosone.org 9 October 2008 | Volume 3 | Issue 10 | e3480