Dataset research

All good questions. It's called an "impact score" because it's intended to
measure, "...the likelihood for the project to exert a sustained, powerful
influence on the research field(s) involved." It's based on 5 individual
criteria and on reviews from 3 reviewers. The individual institutes and
centers (ICs) do not assign percentiles--instead, much of the peer review
system is managed by a third party within NIH called the Center for
Scientific Review (CSR). The method they use to assign percentiles is not
published, but generally they say, that.. "A percentile is the approximate
percentage of applications that received a better overall impact/priority
score from the study section during the past year." ICs do not receive the
base percentile information or any of the raw data, and it's destroyed
after each funding round. NIMH and another IC are currently trying to lobby
for getting the raw data (but this is the federal government and processes
are slow to change).

The percentiles are given within study sections. Study sections tend to
specialize in certain scientific areas, so grant applications are not
randomly assigned. The issue here is, since humans are making the
determination of what grants go where, it's not entirely clear whether
they're actually doing a good job and whether they assign things
consistently. This is a huge problem since we rely on peer review to help
select the most promising science.

I think giving ICs only the overall impact score and percentile is
problematic since some of the most innovative science is controversial. I'm
trying to finagle my way into getting scores from the individual reviewers
to pluck out highly discrepant reviews. We have the things each reviewers
says (saved as PDFs), which can be used for a sentiment analysis (maybe).

I agree that percentiles destroy so much of the information. Not many
people ask very many questions here, so they haven't had much of a reason
to freely give out this type of information.(Until now, ha ha!).

I'm actually working with NHLBI on a publication (which basically
reproduces their study using our data), so I have their code. Thankfully
they also use R. Here's a link to their publication:
http://circres.ahajournals.org/content/early/2014/01/09/CIRCRESAHA.114.302656







On Fri, Jul 25, 2014 at 3:14 PM, Aaron Schumacher <notifications@github.com>
wrote:

> In dataset_research/jamieresearch.md:
>
> > +
> > +
> > +**How do you acquire and load the dataset into  R ? (Include code.)**
> > +
> > +
> > +
> > +```
> > +final_nimh<- read.csv(C:/Users/doylejm/final_nimh.csv)
> > +
> > +```
> > +
> > +**What are some simple statistics describing the dataset?**
> > +
> > +
> > +
> > +- Mean percentile rank: 11.67
>
> So all grant applicants are peer reviewed and this results in an impact
> score which is then translated to a percentile rank - is that right?
>
> Why is it called an impact score?
>
> Not a huge deal, but do you know which percentile method is used?
>
> Are these percentiles across all time, or across one application period,
> or across one grant category/period, etc.? But always within individual
> "study sections", yes? Are applications randomly assigned to study
> sections? Are acceptance patterns consistent with this?
>
> I'm always concerned about using percentiles because they destroy so much
> information in potentially misleading ways. You may want to consider a
> shape-preserving transformation like z-scoring or normal percentiles.
>
> Is there a reference to the NHLBI study? Do they have their data and code
> available?
>
> Rather a nasty selection bias problem - will be interesting to see what
> comes of it!
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/ajschumacher/gadsdc2/pull/43/files#r15419219>.
>