American Idol and NIH grant review--redux.

Michele Pagano's provocative Correspondence discussing ways to improve NIH grant review contains points that deserve additional consideration (Pagano, 2006xPagano, M. Cell. 2006; 126: 637–638Abstract | Full Text | Full Text PDF | PubMed | Scopus (9)See all ReferencesPagano, 2006). In the spirit of “fair and balanced” reporting (to use an expression of the TV network that the article references), I would like to offer an alternative view.There is no doubt that the NIH grant submission and review process is time consuming and cumbersome. Nevertheless, it should not be dismissed and replaced by a process that could aggravate existing problems and create new ones. Pagano proposes implementing presubmission enquiries; expanded versions of these pre-proposals passing the first hurdle then would be submitted. These proposals would be shorter than current NIH grant applications, with a less detailed description of the proposed research plan. It is debatable whether such shortening will be useful or will simply lead to a further “dumbing-down” of individual applications, given that consideration of the overall feasibility of the proposed research would be left to the reviewers' imagination. Junior investigators who often lack an extensive track record of achievements would be at a disadvantage. Pagano's most significant proposal is to conduct review meetings through virtual electronic study sections, in most cases with very limited discussion. This process emulates the review process of scientific manuscripts.There is no dispute that Pagano's suggestions if implemented would result in considerable financial savings for the NIH. The proposed review format, however, would not alleviate the fundamental problem that Pagano so vividly describes by comparing an average NIH study section meeting to the American Idol TV talent show. Rather, the proposed changes would exacerbate this problem. Based on my own 10+ years of experience as a grant reviewer for the NIH and for other federal and non-federal agencies, I do not dispute that the Cowellesque behavior described by Pagano occurs on rare occasions, but it is not common practice. Study section members with behavioral patterns of American Idol judges are not taken seriously by their fellow members, who each have full and equal voting rights and enter a numerical score for each application at the end of the discussion. Discussion is necessary for panel members to appreciate why reviewers like or dislike certain applications and how their score was derived. Electronic interactions or even teleconferences are not adequate. Every effort is made in the current system to streamline the review process. Applications that fall in the lower half are removed after minimal discussion, sometimes at a teleconference prior to the actual meeting. If a mistake has been made, “streamlined” applications can be resurrected by any panel member at the meeting and will receive the benefit of a full discussion. At the actual meeting, Scientific Review Administrators (SRA) and/or the committee chairs do their utmost to keep succinct discussions regarding proposals where there is agreement on their relative merit. Discussions, however, are absolutely necessary even in such cases because they ensure consistent scoring within a study section. This is imperative as even minor variations in numerical scores have a dramatic impact on the final percentile ranking. This ultimately determines whether a proposal will get funded or needs to be revised and resubmitted.What works for the review of scientific manuscripts cannot be applied to the grant review process, particularly not when funding levels are at the dismal levels that we are experiencing today. A manuscript rejected by a journal with a 10% to 15% acceptance rate (similar to the funding rates of most NIH institutes) can still be successfully published in a different journal, often without substantive revisions. Similar solutions do not exist for a grant proposal that is not funded by NIH; loss of funding generally entails immediate and dire consequences for the lab and its personnel. Our frustration with the current funding situation will not be solved by fundamentally changing the grant review process in a way that will make “mistakes” even more commonplace.The process could be helped, however, by the NIH taking immediate steps to streamline the application process rather than forcing us to adopt an archaic electronic grant submission system that creates a significant additional burden for each applicant and their institution. Next year, RO1 grants will have to be submitted through a Windows-only software system—applicants who use other computer platforms are out of luck and are left with investing their dwindling grant dollars in the purchase of expensive software or additional hardware, or else they will have to log on to a Windows server, which necessitates an active internet connection. Users will have to wade through a panoply of strange new forms in addition to the standard NIH-specific forms. The forms have already been changed once and now we are facing another round of alterations, not to mention a user manual that is inches thick. Institutions, after painstakingly checking each of these forms, will then upload them onto a “secure” central server. As a consequence of the vastly increased time institutions require to deal with these new procedures, applicants will lose at least a week to finalize their proposals. In this case, user-friendly electronic submission procedures that have been successfully implemented years ago by almost every major scientific journal would definitely be worth a look.The morale and effectiveness of grant reviewers is not helped if they are forced to download and print on their own time and expense proposals that they are to review. Other recently introduced bureaucratic annoyances for reviewers include required registration as government contractors: a lengthy, two-step application process that requires additional follow-up phone calls requesting to be removed from junk email lists, as well as annual re-registration. Moreover, reviewers are now forced to make travel reservations on the cheapest nonrefundable tickets, entailing additional personal expenses if flights need to be changed at the last minute in case a study section takes longer that anticipated.Nevertheless, as reviewers, we need to continue to do our best to provide a fair and serious review of the grant proposals that we are charged to evaluate. It is up to us to resist the temptation to act like celebrity judges on some tacky TV show. Most of us take our mission as reviewers seriously, and we realize that the current process, which is inherently expensive and time consuming, serves an important purpose, even though we occasionally check our emails during meetings. As a community, we have to work harder to make a more convincing case to politicians and taxpayers alike that the mission of the NIH is endangered by the current severe funding restrictions.