Science with a punch

There are changes afoot in the way that US granting agencies, most notably the NIH, evaluate proposals. Change can be a good thing, and it is probably high time that the NIH reassessed how grants are evaluated. Frankly, I was becoming blase about the entire granting process. It seemed that once I became inured to the low probability of funding, competition from the mega-labs, and the shift towards preferring science that can be called ‘medically relevant’, all of the fun had been taken out of writing grants and reading the subsequent scathing evaluations. Now, the evaluation of science will include a more precise and specific ranking of several individual aspects of a proposal, one of which is the (real or imagined) significance or impact of the proposed research. Ah … life is worth living again!How can one argue with the notion that significant science should be funded and insignificant science should not? Although explicit ranking of the impact of the proposed research seems to be a great idea, surely reviewers have been overtly or covertly considering significance when rating grants all along. So why has this criterion now come out of the closet and how will it be applied evenly to all grant applications?One major reason for the change in the method of evaluation is that a mechanism for provoking reviewers into giving out a wider range of rankings seems to be needed, since so many grants score high and so few can be funded. If the evaluation can be broken down into smaller quanta there is a chance that the overall scores may have greater variability. But how can we put a number on such an inherently unpredictable aspect of science as its future significance? Yes, some projects have obvious impact, but one of the attractions of this business is the fact that there are times when biology is determined to undermine your nice, tidy, and oh-so-logical model in a new, fascinating and wholly unpredicted way. Apparently routine questions are forever coming up with unexpected answers. Will the new system reduce our ability to check and recheck the truth of our assumptions?Moving to the personal level, how can I assess the likely impact of my own research? How will I know that my work will affect the concepts or methods that drive my field, and how can I be sure that I am working on an important problem? I increasingly suspect that the fact that I think my science is interesting and important is irrelevant to the misguided people out there who don’t share my views. Is there a generally acceptable way to define high-impact science?One simple-minded criterion is to gauge significance using the publication record: the number of papers published, and where they are published. This criterion is convenient and readily quantifiable, with the added bonus that the editors and reviewers of the ‘correct’ journals do a good deal of the dirty work by deciding what areas of science are significant and which papers are likely to have an impact on the field. Perhaps in the future journals will be able to buy the right from the NIH to affix the label “The place where high-impact scientists publish!” to the front cover. But are we really happy with the idea that these faceless editors and reviewers who already have so much power will now have more? Are they really the best people to decide the overall scientific direction of the country? Perhaps.It's hard to guess which journal the paper is going to end up in before the work is even done, however. So can we define high-impact science by the area of research? Maybe the NIH should provide a list of high-impact topics, to alleviate confusion for those of us who might be unwittingly considering working in medium or low impact areas. In assembling this list, the NIH may want to consider whether high-impact science is work that even the man on the street has heard of, having an appeal not solely confined to the rarified circles of the scientist. By this criterion, research to improve the quality of the beers produced by the major brewers in the US would probably be the ultimate in high-impact biological science.However we decide to define high-impact science, there is one problem that nobody seems to have recognized yet. We need to provide therapy for people who are addicted to science of lesser significance. I envision a high-quality, extremely confidential clinic (the Watson & Crick Clinic for Significant Science) modeled along the lines of the Betty Ford treatment center for substance abusers. The anonymous patients would be offered a series of seminars alerting them to the warning signs of medium- or low-impact science, and would be taught to avoid self-destructive tendencies such as an affection for un-trendy research. This approach alone would greatly improve the perceived quality of science in many labs almost overnight.Of course it would be a good idea to shift the way that people develop projects in such a way as to make it more likely that truly significant work will be done. But will this change in NIH reviewing guidelines help or hinder this goal? I suspect the latter.