# Problems with Academia and the Rising Sea

**Severe problems with the biomedical research process**

GiveWell has recently been investigating ways to improve biomedical research. When I discovered GiveWell's research I was shocked by how severe and comprehensive the problems with the field seem to be:

From a [conversation](http://www.givewell.org/files/conversations/Ferric%20Fang%2003-01-13%20(public).pdf) with Ferric Fang:

_Because scientists have to compete for grants, they spend a very large fraction of their time fundraising, sometimes more than 50% of their working hours. Scientists feel_ \[_strong_\] _pressure to optimize their activities for getting tenure and grants, rather than for doing good science._ 

From a [conversation](http://www.givewell.org/files/conversations/Elizabeth%20Iorns%20conversation%2002-26-13%20(public).pdf) with Elizabeth Iorns:

_Researchers are rewarded primarily for publishing papers in prestigious journals such as Nature, Science and Cell. These journals select for papers that report on surprising and unusual findings. Papers that report on unsound research that is apparently exciting are more likely to be published than papers which report on less exciting research that is sound._

_There is little post-publication check on the soundness of papers’ findings, because journals, especially prestigious ones, generally don’t publish replications, and there is little funding for performing replications._

\[…\]

_Pharmaceutical companies such as Bayer and Amgen have studied the frequency with which studies are reproducible by trying to reproduce them, and they have found that about 70% of published papers in the areas that they considered don’t reproduce._

\[…\]

_Because many published results are not reproducible, it is difficult for scientists to use the published literature as a basis for deciding what experiments to perform._

\[…\]

_As things stand, the pharmaceutical industry does replications, however, these are generally unpublished. Because a given lab doesn’t know whether other labs have found that a study fails to replicate, labs duplicate a lot of effort._

From a [conversation](http://www.givewell.org/files/conversations/Ken%20Witwer%2002-28-13%20(public).pdf) with Ken Witwer:

_Dr. Witwer published a study in Clinical Chemistry examining 127 papers that had been published in between July 2011 and April 2012 in journals that ostensibly require that researchers deposit their microarray data. He found that the data was not submitted for almost 60% of papers, and that data for 75% of papers were not in a format suitable for replication._

The above remarks give the impression that the problems are deeply entrenched and mutually reinforcing. On first glance, it seems that while one might be able to make incremental improvements (such as funding a journal that publishes replications), prospects for big improvements are very poor. But I became more hopeful after learning more.

**The Rising Sea**

The great mathematician Alexander Grothendieck [wrote](http://www.math.jussieu.fr/~leila/grothendieckcircle/mclarty1.pdf) about two approaches to solving a difficult problem:

_If you think of a theorem to be proved as a nut to be opened, so as to reach “the nourishing ﬂesh protected by the shell”, then the hammer and chisel principle is: “put the cutting edge of the chisel against the shell and strike hard. If needed, begin again at many diﬀerent points until the shell cracks—and you are satisﬁed”._

\[_…_\]

_I can illustrate the second approach with the same image of a nut to be opened. The ﬁrst analogy that came to my mind is of immersing the nut in some softening liquid, and why not simply water? From time to time you rub so the liquid penetrates better, and otherwise you let time pass. The shell becomes more ﬂexible through weeks and months—when the time is ripe, hand pressure is enough, the shell opens like a perfectly ripened avocado!_

_A diﬀerent image came to me a few weeks ago. The unknown thing to be known appeared to me as some stretch of earth or hard marl, resisting penetration … the sea advances insensibly in silence, nothing seems to happen, nothing moves, the water is so far oﬀ you hardly hear it …. yet it ﬁnally surrounds the resistant substance._

When a nut seems too hard to crack, it’s wise to think about the second method that Grothendieck describes.

**Alternative Metrics**

I was encouraged by GiveWell’s subsequent conversations, with David Jay and Jason Priem, which suggest a “rising sea” type solution to the cluster of apparently severe problems with biomedical research.

In brief, the idea is that it may be possible to create online communities and interfaces that can be used to generate measures of how valuable researchers find research outputs, and which could be used for funding and tenure decisions, thereby rewarding producing the research outputs that other researchers find most valuable. If incentives become aligned with producing valuable research, the whole system will shift accordingly, greatly reducing the existing inefficiencies.

From a [conversation](http://www.givewell.org/files/conversations/Jason%20Priem%2003-13-13%20(public).pdf) with Jason Priem

_Historically, the academic community has filtered academic outputs for interest by peer review and, more specifically, the prestige of the journals where papers are published. This model is inadequate relative to filtering mechanisms that are now in principle possible using the Internet._

_It is now possible to use the web to measure the quality and impact of an academic output via alternative metrics (altmetrics)_ _such as_

*   _How many people downloaded it_
*   _How much it has been discussed on Twitter_
*   _How many websites link to it_
*   _The caliber of the scientists who have recommended it_
*   _How many people have saved it in a reference manager like Mendeley or Zotero_

_This is similar to how Google generates a list of webpages corresponding to a search term, since you can benefit from PageRank-type algorithms that foreground popular  content in an intelligent fashion._

\[…\]

_There’s been a significant amount of interest from funders and administrators in more nuanced and broader measures of researcher impact than their journal publication record._ \[…\] _Algorithmically generated rankings of researchers’ influence as measured by the altmetrics mentioned previously could be an input into hiring, tenure, promotion, and grant decisions. ImpactStory and other providers of alternative metrics could help researchers’ aggregate their online impact so that they can present good summaries of it to administrators and funders._

From a [conversation](http://www.givewell.org/files/conversations/David%20Jay%2003-14-13%20(public).pdf) with David Jay

_Commenting systems could potentially be used to create much more useful altmetrics. Such altmetrics could be generated for a scientific output by examining the nature of the comments that scientists make about it, weighting the comments using factors such as the number of upvotes that a comment receives and how distinguished the commenter is._

_The metrics generated would be more informative than a journal publication record, because commenters give more specific feedback than the acceptance/rejection of a paper submitted to a given journal does._

\[…\]

_If scientists were to routinely use online commenting systems to discuss scientific outputs, it seems likely that altmetrics generated from them would be strong enough for them to be used for hiring, promotion and grant-making decisions (in conjunction with, or in place of, the traditional metric of journal publication record)._

\[…\]

_David Jay envisages a future in which there is_ \[...\] _A website which collects analytics from other websites so as to aggregate the impact of individual researchers, both for their own information and for use by hiring/promotion/grant committees._

The viability of this approach remains to be seen, but it could work really well, and illustrate a general principle.

**About the author:**I worked as a research analyst at GiveWell from April 2012 to May 2013. All views expressed here are my own.