Peering into review.

Funds awarded after peer review of grant applications by committees established by the National Institutes of Health (NIH) directly affects the careers of many biomedical scientists. The scientific tastes and judgments of peer review committees also shape fields by favoring specific questions and approaches to particular scientific opportunities that the committees favor. Such a process naturally generates a considerable amount of discussion in the scientific community. As someone who spent most of his scientific career as a review administrator at the NIH it distresses me that these discussions are usually based on ill-founded anecdotes and misinformation.This is fueled by the secretiveness in which the NIH shrouds the review system in a misguided attempt to maintain confidentiality and is coupled with the protective egotism by which applicants insulate themselves from painful criticism. I hope to dispel some myths and misperceptions regarding peer review of NIH grant applications.The scientific review administrators (SRAs) at the NIH are not necessarily ‘failed scientists with personality disorders,’ an epithet with which I was greeted when I first came to NIH. Despite the fact that they are often the conduits of bad news, this is not the highlight of their day. They are individuals promoting the mission of the NIH and facilitating the scientific efforts of others. This is not always an easy task given the arcane, almost Byzantine, regulation and policy constraints. Equally limiting is a bureaucratic culture that rarely rewards innovation. Nevertheless, SRAs at the NIH know the grant process thoroughly and are happy to guide and advise applicants when necessary. Although they see this task as important, applicants must seek them out. Investigators should call NIH administrators early on when preparing applications and often thereafter.Probably the most important, and certainly the most visible, responsibility of the SRA is nomination of scientists to become members of the review group. Although the NIH Director makes the appointment, nomination is tantamount to appointment, as long as the political requirements that ensure gender equity, and geographic and racial diversity are satisfied.There is a tendency to see the reviewers as nitpickers whose strength is negativity. Nothing is further from the truthAlthough the higher echelons at the NIH trumpet the mantra of ‘the best and the brightest,’ my experience does not bear this out. In over 20 years, not once was I questioned about the scientific credentials of nominees or my rationale for selecting them. However, it was not unusual for the approval of my nominations to be held up for months because someone (usually, unidentified) in the chain of command thought that too few women, minorities or citizens of North Dakota were included. In one case, I had a Nobel laureate, and in another, a Fellow of the Royal Academy, sitting on their hands while I haggled over the political correctness of the nominees to a review group.You might question the wisdom of placing such an important responsibility in the hands of the SRA. A look at study section rosters (available via the NIH homepage) is unlikely to settle the issue. Most review groups have at least some outstanding members; the noticeable problem is that, in general, the quality within any roster is variable as is the quality among the various committees. However, a few rosters will be seen to be outstanding from top to bottom. My judgment is that the SRA is in the best position to make the selections. Having been involved in the review of innumerable applications, (s)he should be aware of the significant open questions and impediments to progress in a particular field and who are the major players. (S)he can consult the leading scientists in the US. Thus, weaknesses of the rosters are not due to the level at which the decisions are made, but to failures in quality control in selecting SRAs. Some people, even those at the highest levels at the NIH, believe that a contributing factor is that the best people are refusing to serve. I can only respond that, in 22 years, only three or four people refused service on the Cell Biology Study Section.The second most important role played by the SRA is the creation of an environment at the study section meeting that encourages the reviewers to see beyond their own experiments and biases. Creating such an environment is difficult but once established, it tends to be self-perpetuating. Reviewers leave feeling that they have spent time fruitfully and that they have learned as much as they taught.But researchers outside the study section tend to view the meetings more negatively. As only about a third of the applications reviewed at one meeting are funded, at least two thirds of the applicants are disappointed. There is a tendency to see the reviewers as nitpickers whose strength is negativity. Nothing is further from the truth. Reviewers sacrifice at least two weeks of productivity in preparation for every study section that they attend. Like all scientists, their passion is good and challenging science. They want to be excited, to be wowed by the applications and by the discussions that they engender — that is their intellectual payoff for the hard work and angst engendered by study section service. Reviewers do not begin by looking for reasons to denigrate an application; they look for substance, for reasons to say, ‘this work must be done — if successful it will surely have a great impact.’ Much is said about the conservatism of study sections but in my experience this is mainly a reflection of the conservatism of applicants.Most applications are distressingly similar. The organisms or systems may change but the questions and approaches barely differ. A novel application, well argued, stands out like a bright beacon and is rewarded by study section members.There is another important factor that motivates study section members to be fair. Participating in the meeting are some of the influential practitioners in their field who know who is competing and who is collaborating. These are colleagues whose respect it is imperative study section members maintain. The upshot is that reviewers give their competitors the benefit of the doubt and collaborators are held to a very high standard. Few wish to look the fool by seeming to be self-serving or gullible.The NIH peer review system is not perfect, nor do I believe that it is perfectible since it depends to such a high degree on the personalities and talents of people. It is, as many have said, the best system we have for distributing the largesse provided by the public. It works best when all participants recognize that it is a human endeavor with all the strengths and weaknesses that that entails.