# Presentation: 134202

## Decision Making Research and NIH Peer Review

- Jane Steinberg, Ph.D.

**National Institute of Mental Health**
**National Institutes of Health**

## What is Decision Research?

- The systematic study of how humans come to conclusions or a course of action.
- Tests the bounds of rational decision making with very interesting results.
- And 2 Nobel Prizes in economics

**Notes:**
Herb Simon In 1978, he received the Alfred Nobel Memorial Prize in Economic Sciences, and in 1986, the National Medal of Science. 
Although he never took an economics course, Princeton psychologist Daniel Kahneman received the Nobel Prize in economic sciences in 2002 

## What is Decision Research?

**Interesting results showing that:**
**providing information or instructions is not enough in complex or high-risk decision making**
**humans turn to certain heuristics or biases to simplify complex decisions that may lead to non-optimal decisions**
**these biases can be minimized or used to strengthen decision making and behavior change**

**Notes:**
These findings have been replicated across a number of content areas such as....

## Personal Finances

**Notes:**
Gambling, Investing, selecting a finance package

## Medical Decision Making

**Notes:**
DIAGNOSIS-Improving decisions in mental illness
TREATMENT DECISIONS- determine treatment and also how to help people follow difficult treatment regimens. 
PREVENTION: High-risk behaviors, Diet & nutrition,  sun protection

## Security Issues

**Military protocols**

**Screening procedures**

**Notes:**
Picture of NIMH COUNCIL being screened before heading to campus.

One other fairly reliable finding...

## Decision Research

**People are much happier using their heuristics or biases or intuitions than a structured decision making tool.**

**Notes:**
For instance, I always prefer having an interview with a candidate, even though I am very aware that better decisions are made by the paper credentials alone.

Even so....

## Why Not Peer Review?

## Decision Research and Review

**NIH Peer Review can be conceptualized  as a series of judgments, each one a complex decision**
**Individual decision: a reviewer prepares a critique and assigns a draft score**
**Group decision: the review meeting and consideration of reviewers comments**
**Individual decision: a reviewer assigns a score**

## Decision Research in Peer Review

**RGA consultants told us that reviewers are giving us a lot of information, but we weren’t making the most of it. **
**NIH implemented one recommendation that was based on  work in judgment and decision research.**

**Notes:**
Review of Grant Applications Committee

We’ve already moved in this direction....
* or even letting them make the most of it.

## Decision Research in Peer Review

- RGA consultants explained that the more  structure and simplification provided in a complex decision, the more accurate the decision.
- They recommended that reviewers address each criterion specifically in their reviews, rather than simply identifying strengths and weaknesses.
- This was implemented and is the current process in grant review.

**Notes:**

Breaking the complex task in to smaller, easier decisions and ensuring that the criteria NIH wanted were actually used.

**What else would be helpful to try or to evaluate? What else does decision science tell us?**
**RGA experts recommended going one step further in breaking down the complex individual decision: scoring by criterion.**
**Many possibilities...**

## Review Criteria

**Significance**
**Innovation**
**Approach**
**Investigators**
**Environment**

**Notes:**
Rate each one 1-5 and then take the average for a reviewer’s overall score
Let each of the five scores stand
Decide that the first three deserve more emphasis and assign differential weights, as we do in NIH contract reviews.

The important thing to know is that although the idea of NIH weighting criteria systematically has many detractors, the individual reviewers are doing just this in their heads. Like asking them to do a multifactor regression analysis in their head—so this is just the type of complex judgment that is particularly susceptible to heuristic shortcuts. 

It would also give program staff more information for funding decisions. Say there is an effort to look for innovation. As one of the five criteria, the program officer could give additional weight to this factor. 

##  Heuristics

**Accessibility**
**Framing**
**Anchoring**
**Preference for discussing shared information**

**Notes:**
Acc-Familiarity breeds attraction, less accessible criteria are likely to suffer from interference
Novel ideas or traits may figure in less in the overall decision
Good evidence that people are more comfortable with richer evaluations (more details) than 	shorter ones. Given individual differences in reviewer style, workload, and time devoted to a 	review, is this type of variability something that should be minimized? 

Framing – In a rationale world, one would expect that irrelevant information has no bearing on decision making. But, it turns out that decisions are very context specific. 
McNeil, Pauker, Sox, and Tversky-induced different hypothetical choices between surgery and radiation by describing outcome as either short-term survival or immediate mortality. Significant preference for surgery with a 90% short term survival rate  than radiation with 10% immediate mortality. Just as pronounced among experienced physicians as it was among patients.	

Our instructions to reviewers can be tested to strengthen their focus on identifying important or innovative research. 

## Conclusions

**People and groups use heuristics to make easy work of difficult decisions. **
**Most are successful, but some systematically provide less than optimal decisions.**
**Thirty years’ worth of decision research has identified how to minimize these biases and strengthen decision making.**
**The review process involves very complex decisions that may benefit from applying this knowledge base.**