Even if the findings of this study show that the committee peer review performed by EMBO selected applicants who subsequently to selection did higher impact scientific research than rejected applicants, we still do not know whether the organisation is supporting ‘‘scientific excellence’’. This question can be answered only by comparing the research performance of approved and rejected applicants with international scientific reference values [49]. Vinkler [50,51] recommends a worldwide reference standard for the bibliometric evaluation of research groups: ‘‘Relative Subfield Citedness (Rw) (where W refers to ‘world’) relates the number of citations obtained by the set of papers evaluated to the number of citations received by a same number of papers … dedicated to the respective discipline, field or subfield’’ (p. 164) [52]. Wuchty, Jones, and Uzzi [27] define highly cited work ‘‘as receiving more than the mean number of citations for a given field’’ (p. 1037), that is, with Rw.1. Neuhaus and References 1. Ziman J (2000) Real science. What it is, and what it means. Cambridge, UK: Cambridge University Press. 2. Marsh HW, Jayasinghe UW, Bond NW (2008) Improving the peer-review process for grant applications-reliability, validity, bias, and generalizability. American Psychologist 63: 160–168. 3. Geisler E (2000) The metrics of science and technology. Westport, CT, USA: Quorum Books. 4. Hemlin S (1996) Research on research evaluations. Social Epistemology 10: 209–250. 5. National Institutes of Health (2000) Recommendations for change at the NIH’s Center for Scientific Review: phase 1 report, panel on scientific boundaries for review. Bethesda, MD, USA: Center for Scientific Review (CSR). 6. Ross PF (1980) The sciences’ self-management: manuscript refereeing, peer review, and goals in science. Lincoln, MA, USA: The Ross Company. 7. Bornstein RF (1991) The predictive validity of peer-review: a neglected issue. Behavioral and Brain Sciences 14: 138–139. 8. Cole S, Cole JR, Simon GA (1981) Chance and consensus in peer-review. Science 214: 881–886. 9. Smith R (1988) Problems with peer review and alternatives. British Medical Journal 296: 774–777. 10. Harnad S (2007) Open Access scientometrics and the UK Research Assessment Exercise. In: Torres-Salinas D, Moed HF, eds. Proceedings of the 11th Conference of the International Society for Scientometrics and Informetrics. Madrid, Spain: Spanish Research Council (CSIC). pp 27–33. 11. Cole S (1992) Making science. Between nature and society. Cambridge, MA, USA: Harvard University Press. 12. van Raan AFJ (2005) For your citations only? Hot topics in bibliometric analysis. Measurement: Interdisciplinary Research and Perspectives 3: 50–62. 13. Garfield E (2002) Highly cited authors. Scientist 16: 10. 14. Jennings CG (2006) Quality and value: the true purpose of peer review. What you can’t measure, you can’t manage: the need for quantitative indicators in peer review. 15. Bornmann L, Daniel H-D (2008) What do citation counts measure? A review of studies on citing behavior. Journal of Documentation 64: 45–80. 16. Joint Committee on Quantitative Assessment of Research (2008) Citation statistics. A report from the International Mathematical Union (IMU) in cooperation with the International Council of Industrial and Applied Mathematics (ICIAM) and the Institute of Mathematical Statistics (IMS). Berlin, Germany: International Mathematical Union (IMU). 17. Lokker C, McKibbon KA, McKinlay RJ, Wilczynski NL, Haynes RB (2008) Prediction of citation counts for clinical articles at two years using data available within three weeks of publication: retrospective cohort study. British Medical Journal 336: 655–657. 18. Dewitt TW, Nicholson RS, Wilson MK (1980) Science Citation Index and chemistry. Scientometrics 2: 265–275. 19. Bornmann L, Daniel H-D (2008) Functional use of frequently and infrequently cited articles in citing publications. A content analysis of citations to articles with low and high citation counts. European Science Editing 34: 35–38. 20. Evidence Ltd (2007) The use of bibliometrics to measure research quality in UK higher education institutions. London, UK: Universities UK. 21. Ledin A, Bornmann L, Gannon F, Wallon G (2007) A persistent problem. Traditional gender roles hold back female scientists. EMBO Reports 8: 982–987. 22. Bornmann L, Daniel H-D (2008) Selecting manuscripts for a high impact journal through peer review: a citation analysis of Communications that were accepted by Angewandte Chemie International Edition, or rejected but published elsewhere. Journal of the American Society for Information Science and Technology 59: 1841–1852. 23. Herbertz H (1995) Does it pay to cooperate? A bibliometric case-study in molecular-biology. Scientometrics 33: 117–122. 24. Leimu R, Koricheva J (2005) What determines the citation frequency of ecological papers? Trends in Ecology & Evolution 20: 28–32. Measuring Peer Review Quality Daniel [53] propose for chemistry and related fields such as biology and life sciences reference values that are based on the fields/ subfields of the Chemical Abstracts database (CA, Chemical Abstracts Services, CAS, Columbus, OH, USA). In CA each paper is assigned individually to a field/ subfield. As Bornmann and Daniel [22] succeeded in applying this approach on the evaluation of the peer review process (of the journal Angewandte Chemie-International Edition), we will compare in a future study the publication impact of the EMBO applicants with international scientific reference values. Author Contributions Conceived and designed the experiments: LB GW AL. Performed the experiments: LB. Analyzed the data: LB. Wrote the paper: LB GW. 25. Aksnes DW (2008) When different persons have an identical author name. How frequent are homonyms? Journal of the American Society for Information Science and Technology 59: 838–841. 26. StataCorp (2007) Stata statistical software: release 10. College Station, TX, USA: Stata Corporation. 27. Wuchty S, Jones BF, Uzzi B (2007) The increasing dominance of teams in production of knowledge. Science 316: 1036–1039. 28. Bornmann L, Daniel H-D (2007) Multiple publication on a single research study: does it pay? The influence of number of research articles on total citation counts in biomedicine. Journal of the American Society for Information Science and Technology 58: 1100–1107. 29. Daniel H-D (1993/2004) Guardians of science. Fairness and reliability of peer review. Weinheim, Germany: Wiley-VCH, Published online 16 July 2004, Wiley Interscience, DOI: 10.1002/3527602208. 30. Long JS, Freese J (2006) Regression models for categorical dependent variables using Stata. College Station, TX, USA: Stata Press, Stata Corporation. 31. Hosmer DW, Lemeshow S (2000) Applied logistic regression. Chichester, UK: John Wiley & Sons, Inc. 32. Chatterjee S, Hadi AS (2006) Regression analysis by example. 4. ed. New York, NY, USA: Wiley-Interscience. 33. Hausman J, Hall BH, Griliches Z (1984) Econometric models for count data with an application to the patents R and D relationship. Econometrica 52: 909–938. 34. Hilbe JM (2007) Negative binomial regression. Cambridge, UK: Cambridge University Press. 35. Cameron AC, Trivedi PK (1998) Regression analysis of count data. Cambridge, UK: Cambridge University Press. 36. Jayasinghe UW (2003) Peer review in the assessment and funding of research by the Australian Research Council. Greater Western Sydney, Australia: University of Western Sydney. 37. Johnson VE (2008) Statistical analysis of the National Institutes of Health peer review system. Proceedings of the National Academy of Sciences 105: 11076–11080. 38. Bornmann L, Daniel H-D (2007) Convergent validation of peer review decisions using the h index: extent of and reasons for type I and type II errors. Journal of Informetrics 1: 204–213. 39. Hirsch JE (2005) An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States of America 102: 16569–16572. 40. Bornmann L, Daniel H-D (2007) What do we know about the h index? Journal of the American Society for Information Science and Technology 58: 1381–1385. 41. Bornmann L, Daniel H-D (2005) Does the h-index for ranking of scientists really work? Scientometrics 65: 391–392. 42. Bornmann L, Wallon G, Ledin A (2008) Is the h index related to (standard) bibliometric measures and to the assessments by peers? An investigation of the h index by using molecular life sciences data. Research Evaluation 17: 149– 156. 43. Freeman R, Weinstein E, Marincola E, Rosenbaum J, Solomon F (2001) Careers-competition and careers in biosciences. Science 294: 2293–2294. 44. Thorngate W, Faregh N, Young M (2002) Mining the archives: analyses of CIHR research grant applications. Ottawa, Ontario, Canada: Psychology Department, Carlton University. 45. Stehbens WE (1999) Basic philosophy and concepts underlying scientific peer review. Medical Hypotheses 52: 31–36. 46. Hornbostel S, Bo¨hmer S, Klingsporn B, Neufeld J, von Ins M (in press) Funding of young scientist and scientific excellence. Scientometrics. 47. Melin G, Danell R (2006) The top eight percent: development of approved and rejected applicants for a prestigious grant in Sweden. Science and Public Policy 33: 702–712. PLoS ONE | www.plosone.org 10 October 2008 | Volume 3 | Issue 10 | e3480