Grant Review: American Idol or Big Brother?

There is little doubt that the current NIH grant review system is not perfect. But this is true for any nondeterministic process, including scientific manuscript review, content consensus in collaborative web environments, the refeering of sports events, and debates surrounding issues of health or education. Any process that has broad ramifications and requires a frequent, repetitive, and community-based decision will have to be consistent and fair but also agile and dynamic.Pagano argues that some NIH grant reviewers may resemble judges on the TV talent show American Idol (Pagano, 2006xPagano, M. Cell. 2006; 126: 637–638Abstract | Full Text | Full Text PDF | PubMed | Scopus (8)See all ReferencesPagano, 2006). However, the modifications to NIH grant review that he proposes (prescreening all grant applications, a shorter format, a more general future research plan, and elimination of face-to-face reviewer study sections) may swing the pendulum in the other direction, resulting in judgements like those on the TV reality show Big Brother. There are three primary reviewers for each NIH grant proposal, but usually the entire review panel is engaged in the discussion, assessment, and final summary of each proposal. A diverse panel is critical for evaluating the scientific quality of the proposals, impact of the work, and relevance of the findings. If there are study sections where Cowellesque reviewers highjack the discussion, perpetuate negative comments about the application, and mislead the entire panel into an inappropriate decision, this seems to be a problem with the composition of the panel rather than with the current evaluation system per se. “The nasty reviewer” would not “always win” if the panel acted as a democratic board of conscientious reviewers. There is a large pool of registered NIH reviewers and an even larger pool of qualified scientists who could be recruited to grant review panels. Like jury duty and term limits on public service jobs, there must be a regular turn over of reviewers, administrators, and grant applications to ensure that the process runs smoothly and consistently.Pagano's calculation of the annual cost of NIH grant review ($50 million) is less than 0.2% of the total NIH operational budget ($29 billion). This figure is less than the operating costs for a regular company or the percentage costs associated with managing large mutual funds, academic institutes, or government institutions. For example, the administrative costs of the University of California are about 4% to 5% of the annual operating budget of $11.4 billion. Although I agree with Pagano that assembling an NIH grant application takes significant efforts on the part of the principal investigator and his/her collaborators and requires administrative support, I disagree that it could take 6 months. A good team with the right idea and matching expertise should be able to deliver a solid application within 2–8 weeks, depending on the scope of the project, subject area, and investigator time commitment.Pagano proposes scaling back face-to-face interactions among grant reviewers by increasing discussions by phone or email and only using the written critiques of the primary reviewers, unless there is a rare complication requiring a teleconference. Taking away the face-to-face aspect of NIH grant review, reducing proposal page limits, and relying on a limited written communication will, in my opinion, adversely affect the review process. It will enhance the effects of “α-dominant” reviewers, introduce a bias in favor of large labs and research consortia, and increase the error rate in assessment, resulting in a negative impact on scientific research (Marks, 2006xMarks, A.R. J. Clin. Invest. 2006; 116: 844CrossRef | PubMed | Scopus (41)See all ReferencesMarks, 2006).Scientific progress is not determined solely by “funding” and “publication” as Pagano claims. Funding and publication are merely byproducts of the scientific discovery process and are neither necessary nor sufficient to ensure the quality of research. Often the impact of a new scientific technique, idea, or discovery is not realized until years after it has been communicated. There is a need to accommodate diverse grant applications (Carnes et al., 2005xCarnes, M., Geller, S., Fine, E., Sheridan, J., and Handelsman, J. J. Womens Health. 2005; 14: 684–691CrossRef | Scopus (15)See all ReferencesCarnes et al., 2005) that seek financing for research that is short- or long-term, low or high risk, small or large scale, basic or clinical, and led by junior or senior investigators (Drazen and Ingelfinger, 2003xDrazen, J.M. and Ingelfinger, J.R. N. Engl. J. Med. 2003; 349: 2259–2261CrossRef | PubMed | Scopus (11)See all References, Kotchen et al., 2004xKotchen, T.A., Lindquist, T., Malik, K., and Ehrenfeld, E. JAMA. 2004; 291: 836–843CrossRef | PubMed | Scopus (69)See all References). We need to avoid the blind use of publication counts or total funding support in prospective and retrospective research evaluations, including grant reviews.We have to collectively think of a new strategy to avoid the occasional resemblance of the NIH grant review system to the evaluation of amateur singers on American Idol (“nasty reviewer always wins”). However, we also need to avoid the pitfalls of Pagano's proposed strategy where a potential Big Brother ranking system could result in subjective judgements about grant applications and a bias in favor of large labs and established investigators.