Does the Committee Peer Review Select the Best Applicants for Funding? An Investigation of the Selection Process for Two European Molecular Biology Organization Programmes Lutz Bornmann1*, Gerlind Wallon 2 , Anna Ledin2¤ 1 Professorship for Social Psychology and Research on Higher Education, ETH Zurich, Zurich, Switzerland, 2 European Molecular Biology Organization (EMBO), Heidelberg, Germany Abstract Does peer review fulfill its declared objective of identifying the best science and the best scientists? In order to answer this question we analyzed the Long-Term Fellowship and the Young Investigator programmes of the European Molecular Biology Organization. Both programmes aim to identify and support the best post doctoral fellows and young group leaders in the life sciences. We checked the association between the selection decisions and the scientific performance of the applicants. Our study involved publication and citation data for 668 applicants to the Long-Term Fellowship programme from the year 1998 (130 approved, 538 rejected) and 297 applicants to the Young Investigator programme (39 approved and 258 rejected applicants) from the years 2001 and 2002. If quantity and impact of research publications are used as a criterion for scientific achievement, the results of (zero-truncated) negative binomial models show that the peer review process indeed selects scientists who perform on a higher level than the rejected ones subsequent to application. We determined the extent of errors due to over-estimation (type I errors) and under-estimation (type 2 errors) of future scientific performance. Our statistical analyses point out that between 26% and 48% of the decisions made to award or reject an application show one of both error types. Even though for a part of the applicants, the selection committee did not correctly estimate the applicant’s future performance, the results show a statistically significant association between selection decisions and the applicants’ scientific achievements, if quantity and impact of research publications are used as a criterion for scientific achievement. Citation: Bornmann L, Wallon G, Ledin A (2008) Does the Committee Peer Review Select the Best Applicants for Funding? An Investigation of the Selection Process for Two European Molecular Biology Organization Programmes. PLoS ONE 3(10): e3480. doi:10.1371/journal.pone.0003480 Editor: Scott R. Evans, Harvard School of Public Health, United States of America Received June 23, 2008; Accepted September 18, 2008; Published October 22, 2008 Copyright: 2008 Bornmann et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Funding: The authors have no support or funding to report. Competing Interests: The authors have declared that no competing interests exist. * E-mail: bornmann@gess.ethz.ch ¤ Current address: Royal Swedish Academy of Sciences, Stockholm, Sweden Introduction Peer review is a cornerstone of science [1,2]. It is the oldest metric used to assess scientific work by which a jury of experts is asked to evaluate the undertaking of scientific activity from an intra-scientific perspective [3,4]. Active research scientists who are familiar with the kind of research being proposed are the best judges of the prospective impact of a research proposal on science [5]. However, critics doubt that peer review is a valid assessment instrument [6,7]. Cole and his colleagues [8] concluded in their highly influential study on grant peer review at the National Science Foundation (NSF, Arlington, VA, USA) that ‘‘the fate of a particular application is roughly half determined by the charac- teristics of the proposal and the principal investigator, and about half by apparently random elements which might be characterized as ‘the luck of the reviewer draw’’’ (p. 885). Against this background, every scientific institution that uses peer review should ask whether the peer review system implemented fulfills its declared objective to select the best science and the best scientists. We investigated two programmes of the European Molecular Biology Organization (EMBO, Heidelberg, Germany) for the promotion and support of highly talented young scientists in the life sciences to answer this question. Established in 1966, the Long-Term Fellowship (LTF) pro- gramme has gained an excellent reputation in the scientific community (see http://www.embo.org/fellowships/long_term. html, Access: June 12, 2008). The fellowships are awarded for a period of up to two years and are intended for advanced post doctoral research. The Young Investigator (YI) programme has been supporting outstanding young group leaders in the life sciences in Europe since 2000 (see http://www.embo.org/yip/ index.html, access: June 12, 2008). The programme targets researchers who have established their first independent labora- tories normally four years before the assessment in an European Molecular Biology Conference (EMBC, see http://www.embo. org/embc/, Access: September 6, 2007) member state. The evaluation procedure for applicants to both programmes comprises of an interview with an EMBO member expert in the area of the applicant’s research and an evaluation by all members of the programmes’ selection committees. Each committee member PLoS ONE | www.plosone.org 1 October 2008 | Volume 3 | Issue 10 | e3480