Center for Scientific Review

Peer Review Advisory Committee Meeting

National Institutes of Health

U.S. Department of Health and Human Services

May 16, 2005

The second meeting of the Peer Review Advisory Committee (PRAC) convened
at 8:15 a.m. on Monday, May 16, 2005, at the Bethesda Marriott,
Bethesda, Maryland. The entire meeting was held in open session. Dr.
Brent Stanfield and Dr. Jeremy Berg presided as Co-Chairs.

## Members

Jeremy Berg, Ph.D., Co-Chair Edward Pugh, Jr., Ph.D.

Brent Stanfield, Ph.D., Co-Chair Louise Ramm, Ph.D.

Faye Calhoun, Ph.D. Anne Sassaman, Ph.D.

Robert Hammond, Ph.D. Beverly Torok-Storb, Ph.D.

Joe Martinez, Ph.D. Matt Winkler, Ph.D.

*Ad Hoc Member*

Antonio Scarpa, M.D., Ph.D.

Dr. Karl Malik was the Executive Secretary for the meeting.

# Welcome and Approval of January 2005 PRAC Meeting Minutes

Dr. Berg welcomed members and participants to the second meeting of PRAC
and asked the members to briefly introduce themselves.

Dr. Stanfield asked for approval of the minutes from the January 2005
PRAC meeting.

Dr. Anne Sassaman moved to approve the minutes. Her motion was seconded
and approved unanimously.

**Core Values of NIH Peer Review**

Dr. Alan Willard, Chair of the Review Policy Committee (RPC) and Chief
of the Scientific Review Branch of the National Institute of
Neurological Disorders and Stroke (NINDS), made a presentation to
initiate discussion on NIH peer review core values. When PRAC members
discussed this topic in January 2005, Dr. Stanfield asked Dr. Willard to
convene a small group to articulate these values. The Core Values
Subcommittee of the Peer Review Committee was subsequently formed, with
representatives from the RPC and the Extramural Programs Management
Committee.

Dr. Willard noted that the definition of core values evoked much
interest and passion by the committee, which produced a summary of its
actions and outcomes that was provided to PRAC members. He then said
that members of the core values committee used different words to
describe core values, but they all agreed on two concepts: scientific
and technical competence, and fairness and objectivity.

[Scientific and Technical Competence]{.underline}: Achieving scientific
and technical competence requires that requests for NIH support be
evaluated by groups of qualified individuals who collectively have (1)
the appropriate scientific and technical expertise to evaluate the
critical aspects of the requests, (2) the capability to make objective
judgments in a broad scientific context, and (3) the flexibility and
receptivity to address new areas of science and new technologies. A
commitment to scientific and technical competence should also govern all
CSR efforts from the receipt and referral of grant applications, to the
management of reviews, and to the release of summary statements.

[Fairness and Objectivity]{.underline}: Managing conflicts of interest
poses the biggest challenge in achieving fairness and objectivity. Dr.
Willard stressed that the committee carefully chose the word "manage" to
acknowledge that anyone who has the relevant expertise to review an
application potentially has some level of conflict of interest.
Achieving this core value also requires keeping the review process
independent from the many interest groups and others that support NIH
funding for particular areas of research.

Other important factors to achieve fairness and objectivity include (1)
encouraging informed participation by all members of a review group; (2)
applying the review criteria appropriately and consistently throughout
the meeting, from the morning of the first day to the afternoon of the
last; (3) implementing policies and procedures consistently and equally
for all applications;

\(4\) maintaining confidentiality of both the applicant's ideas and the
reviewer's opinions;

\(5\) making information about the peer review process available so that
the process is as transparent as possible; and (6) monitoring the
process to ensure fairness and objectivity.

*Discussion*

Dr. Edward Pugh praised the Core Values Committee's effort as much
needed and timely and asked how to make the best of its insights as CSR
moves forward. Dr. Willard said that having principles stated up front
and conspicuously can help in guiding future discussions, particularly
future PRAC discussions such as assessing the possible tradeoffs of
in-person vs. electronic review meetings.

Dr. Faye Calhoun augmented her comments with several PowerPoint slides.
She suggested six core values: (1) assure the transparency of the review
process; (2) maintain and support the credibility of and confidence in
publicly funded biomedical research and the products from it;

\(3\) improve the quality of research through the discussions that occur
and the feedback provided to the applicant; (4) gain unprejudiced,
evidence-based decisions from independent reviewers without conflicts
with the applicant investigator; (5) enhance the breadth and depth of
reviewers' knowledge; and (6) maintain an efficient and effective
process.

Dr. Robert Hammond praised the Core Value Committee's summary draft, but
suggested inserting language about the purpose of a review; e.g.,
stating that the purpose is to allow the Institutes and Centers (ICs) to
make the most informed funding decisions means that peer review must be
an objective process without interference. Dr. Willard said that his
committee's summary draft intended to capture this purpose when it
mentioned the need to "manage other influences."

Dr. Berg noted that perhaps the definition of "peer" is needed, as it is
defined differently in many contexts. Dr. Stanfield said that the NIH
peer review process differs from other institutions in many ways,
including what constitutes a peer, the ability for those who receive NIH
grants to sit on a review, and the separation between program and review
functions.

Dr. Louise Ramm warned against letting reviewers see their job as making
funding decisions. She noted that reviewers and scientific review
administrators (SRAs) sometimes seem to know pay lines and, thus, what
priority scores may be funded. She suggested that the core values aim to
prevent this tendency. Dr. Anne Sassaman concurred with Dr. Ramm on the
importance of communicating the difference between priority scores and
funding decisions, as well as the different responsibilities in the
process.

Dr. Norka Ruiz Bravo praised the work on core values and expressed
support that the core principles remain broad so they will be useful as
changes are proposed. Dr. Pugh agreed and suggested that referring to
core values could protect the peer review system from attacks.

Dr. Joe Martinez discussed operationalizing the values. For example,
overloading study sections goes against a core value as presented. Dr.
Willard stressed the need for consistency not only within a meeting, but
also from one meeting to the next.

Dr. Antonio Scarpa said that the speed at which review results are
communicated to an investigator is very important, especially for new
investigators. He asked how timing fits in with the core values. Dr.
Willard said that the committee discussed timing as part of fairness.
Policies require SRAs to prepare summary statements for the best
applications first. However, this priority conflicts with timely
communication of results to applicants. Dr. Peter Guthrie, SRA, CSR
Molecular, Cellular and Developmental Neuroscience Integrated Review
Group (IRG), said ICs deal with applications differently, which could
affect reviewer consistency.

Dr. Berg asked about the status of efforts to evaluate Internet-Assisted
Review and to investigate the correlation between pre-meeting scores and
post-meeting outcomes. Dr. Stanfield said that the data had not been
analyzed, but may shed light on whether and how people change their
minds. Dr. William Benzing, Deputy Chief of the CSR Brain Disorders and
Clinical Neuroscience IRG, said that he has seen that scores often
change at a meeting, and the change can go either way.

Dr. Alexander Politis, Chief of the CSR Infectious Diseases and
Microbiology IRG*,* stressed thoroughness as a core value, noting that a
thorough review cannot take place without experts, but an expert review
can take place without being thorough. He stated that doing away with
study section meetings might jeopardize thoroughness.

Dr. Beverly Torok-Storb praised the work of the Core Values Committee.
She questioned the use of the term "competence," which could be equated
with "satisfactory" reviews vs. "good or excellent" reviews. Dr. Willard
acknowledged that much discussion had focused on what constitutes a
"competent review," and that technical and scientific expertise was just
one part. Dr. Calhoun commented on how reviews fit into Institutes'
funding decisions. She said that a serious discussion across NIH would
be critical and could change the way study sections operate.

Dr. Hammond referred to Dr. Willard's presentation and suggested adding
language that all sources of information should be utilized, including,
where appropriate, Institute staff in order to have an informed review.
He said that having more information strengthens the SRA's role.

Dr. Andre Premen, Assistant Director in the CSR Division of Receipt and
Referral, said that program staff behavior could influence how reviewers
evaluate applications, directly or indirectly. Program staff attend
review meetings as guests of the review staff to observe, provide
feedback to applicants, and pick up on new areas of science, but not to
participate or influence the process. Dr. Stanfield said a partnership
is needed between program and review staff.

Dr. Stanfield suggested getting input on the core values from outside
stakeholders, as well as asking the Core Values Committee to formulate
an Implementation Plan for consideration at the next PRAC meeting. In
the meantime, he suggested that these principles serve as interim
guiding principles for peer review.

**Quantitative Aspects of NIH Peer Review**

Dr. Stanfield said that a study on the topic of quantitative aspects of
peer review had begun. He then provided some background on NIH rating of
grant applications.

***Rating of Grant Applications***

[History]{.underline}: Dr. Stanfield explained how grant applications
have been rated over the past 55 years and focused on efforts to change
the system in the 1990s. He reminded PRAC members that NIH rating of
grant applications began in 1950 with a five-point scale. In the 1970s,
the Division of Research Grants mapped priority scores onto a
standardized distribution curve out of concern that the scores were not
comparable across study sections. In 1980, the National Heart, Lung, and
Blood Institute suggested percentiling, which was adopted as NIH's
normalization convention in 1987.

In 1996, and NIH Rating of Grant Application (RGA) Committee looked at
the review criteria, which were not as focused as the criteria used
today. The RGA Committee was composed of NIH staff and a member of the
National Science Foundation (NSF) and was aided by experts in the fields
of psychometrics and decision-making. This committee made four primary
recommendations: (1) reformulate the review criteria, (2) conduct
reviews criterion by criterion, (3) provide separate numerical reviews
for each criterion, and (4) eliminate the global rating of scientific
merit.

The RGA Committee also criticized the polarity of the rating system, in
which 1 is best and 5 is worst. They also suggested using a scale of 0
to 7, or at most 0 to 10; anchoring the rating scale only at the ends;
ensuring that the scores standardize on each criterion within a reviewer
and average across reviewers; and reporting scores out to two
significant digits, rather than one. They also said that an overall
score, if necessary, should be the average of the individual scores on
the different criteria, rather than a separate overall score. In 1996,
the RGA Committee's recommendations went to the Peer Review Oversight
Committee (PROG).

[Review criteria]{.underline}: PROG put forward four review criteria for
consideration. The NIH Director at that time, Dr. Harold Varmus,
modified them slightly into the five criteria used today: significance,
approach, innovation, investigator, and environment.

PROG did not recommend the RGA's other suggested changes, except that
well-defined criteria should lead the review. However, Dr. Stanfield
noted that almost 10 years have passed, and it might be time to revisit
some of these issues.

Dr. Stanfield then introduced Dr. Valen Johnson to report on a study he
is conducting on how reviewers rate grant applications. Dr. Johnson is
the author of *Grade Inflation: A Crisis in College Education*, which is
based on data gathered at Duke University.

***Analysis of Scientific Review Group Data: A Proposal***

Dr. Johnson stressed the need to interpret ratings properly so as not to
compromise the integrity of the peer review system.

[Scientific questions]{.underline}: Dr. Johnson raised four questions:
Do rating criteria vary systematically between panel members within and
between Scientific Review Groups (SRGs)? If so, how do these variations
affect scoring ranges provided to non-reading members of the panels? Do
personality traits, such as persuasiveness, differentially affect how
non-readers score applications? How do such effects combine to influence
the summary score of a proposal, and how might scoring procedures be
changed or modified to minimize these effects?

[A hypothetical example]{.underline}: Dr. Johnson discussed the
hypothetical review of four proposals by nine reviewers. In his model,
the reviewers had different "rater thresholds," by which he meant that
some tend to score more stringently than others. He then showed how
different rater thresholds could affect the mean score of proposals and
their ranking. In the example he presented, the order of ranking of
proposals by each rater was in exactly the opposite order implied by the
mean of the reviewers scores. This difference was explained by the fact
that different raters used different thresholds.

The variation in rater thresholds is common to nearly all rating
schemes, he said. People have different ways of subjectively rating
proposals, which is why multiple raters are needed. He said that the
effects are exacerbated if the raters who score more stringently are
more persuasive.

[Statistical Analyses]{.underline}: To better estimate a proposal's
merit, Dr. Johnson said rater thresholds, precision, and persuasiveness
must also be estimated. Dr. Johnson said a statistical model can adjust
for different thresholds, and he illustrated how a baseline latent
variable model can do so.

Dr. Johnson called the NIH system more complex than other multirater
scoring systems, since proposals are not read by all the raters. The
non-readers receive a report of a proposal from the reading members in
the study group and must assume that the score that a reader has given
matches their interpretation of what would merit the same score. Rater
precision refers to the group's perception of how accurate the reader
was in his or her review of the application. A model could be
constructed that reflects the fact that raters may shift their scores
towards their own mean. However, to implement this kind of system or
make recommendations about how to change the NIH scoring scheme, it
would first be necessary to see how sensitive the model's assumptions
are.

[Conclusions]{.underline}: According to Dr. Johnson, potentially serious
and undetected biases may affect funding decisions. Such biases, if
present, can be detected and quantified. Statistical modeling may
suggest mechanisms for improving the collection and interpretation of
SRG scoring data.

*Discussion*

Referring to Dr. Johnson's example, Dr. Martinez suggested that perhaps
what is at issue is that the numbers are being used as an interval,
rather than an ordinal, scale. The relative rankings remained the same,
so perhaps these rankings could be reported out of a committee, rather
than trying to develop a more complicated model. Dr. Johnson pointed out
that the scores contain more information; i.e., there is a greater
difference between a 1.2 and a 1.9 than a 1.2 and a 1.3, although both
cases show the same first and second rankings. In response to a question
about the scoring data in his example, Dr. Johnson said that they were
not actual data. He said that the study he is conducting will find out
whether panel-rating data do or do not reflect the patterns in the
example.

Dr. Ruiz Bravo said that the presentation suggested two options: either
acknowledge that the system is not perfect because of the human judgment
factor and figure out a way to deal with it, or apply statistics or
other methodologies to put more rigor into the system. She said that she
would prefer three bins---yes, no, and maybe---with statistical rigor
focused on the middle bin, but that this concept would change NIH
culture.

Dr. Torok-Storb reminded members that three groups hold stakes in the
NIH review process: the ICs, the applicants, and the reviewers. The ICs
use the review process to inform its decision-making. The applicants use
the feedback to help them figure out what was wrong in the application.
The reviewers are charged with preparing the review, but Dr. Torok-Storb
urged CSR to develop a way for reviewers to communicate their
assessments better and more easily. In addition, she said that too many
applications are being triaged. She agreed with Dr. Ruiz Bravo that
perhaps three levels would be more useful.

Dr. Ramm asked Dr. Johnson how the number of study sections in which a
reviewer participates affects the threshold. Dr. Johnson said that the
model shows thresholds with each reviewer rating two proposals. He
stressed he is using the model to study the process, not reform it.

Referring to Dr. Stanfield's presentation about global scores, Dr. Ramm
said that, in some cases, the overall global score reflects the
"gestalt" of everything proposed. She agreed that some sort of binning
might be useful, so that an Institute can make decisions based on
balancing the portfolio and other general interests, rather than on
small, incremental differences in scores.

Dr. Ruiz Bravo noted that a binning experiment conducted by the National
Institute of General Medical Sciences was controversial with the
extramural community and was discontinued.

Dr. Pugh observed that some subjectivity is unavoidable, so the issue
becomes using checks and balances to avoid excess subjectivity. He
suggested determining how the system is doing before rushing to change
it. The system to train reviewers is tremendously important to make
scales less incommensurate between reviewers. He suggested setting up an
experiment to see how big the problem is.

Dr. Martinez identified two aspects of the review process that do not
seem to adhere to best scientific practices and principles: having
people make 40 distinctions on a scale, and using the ratings as an
interval scale. He suggested changing the scores to 1, 1.5, 2, and so
on, rather than tenths. He also said that using the ratings as an
interval scale cannot produce a mean.

Dr. Berg noted that IC program staff only receive overall scores; thus,
they do not know the relative weights of the different factors that
individual reviewers use. Scores on individual criterion from the
reviewers who have read the application, in addition to an application's
overall score, could help in program judgment.

Dr. Sassaman asked whether the Panel on Scientific Boundaries for Review
made any recommendations about assessing reviewer behavior. Dr.
Stanfield said that he did not think there was a call for such an
assessment, but that he would check. Dr. Torok-Storb noted that, when
she was on a National Cancer Institute (NCI) subcommittee, individual
scores were recorded and reviewers had to write a dissenting opinion
when their scores fell outside the range recommended by the readers.

Dr. Tom Tatham, SRA, CSR Biobehavioral and Behavioral Processes IRG,
asked

Dr. Johnson to comment on the definition and concept of persuasiveness
and the role that it plays in his model. In some circumstances, Dr.
Tatham argued, it is reasonable for reviewers to be differentially
influenced by a particular reader's evaluation. For example, when
evaluating fellowship applications, the views of a professor known for
his or her work with postdoctoral fellows would carry more weight than a
beginning professor. Dr. Johnson said that the data do not distinguish
whether a person is more expert or just more persuasive. One way to get
at that would be to see how reviewers change their scores. The goal at
this stage, he stressed, is to see if some readers have more influence
than others.

Dr. Eileen Bradley, Chief, CSR Surgical Sciences, Biomedical Imaging,
and Bioengineering IRG, said that the model did not factor in diversity
of scientific expertise, although it is a large factor in how people
look at applications. For example, a surgeon and an engineer might
evaluate a new device differently, based on their experience and
expertise.

Dr. Olivia Bartlett, Chief, NCI Research Programs Review Branch, said
that the Department of Defense (DoD) has provided an overall score and
criterion scores to program managers and to the DoD equivalent of NIH
Councils for the past 10 years. She also observed that, in her 18 years
experience with review, differences between scores are usually resolved
in discussion, which reinforces the value of a meeting.

Dr. Stanfield emphasized that no one is suggesting that the current
system does not provide good information, but that there is value in
studying it to see if it can be improved. Dr. Ruiz Bravo concurred.
While it might be an imperfect system, she said, it is probably the best
system not just in the United States, but probably in the world. It is
important for scientists to have facts and data, she said, which is
behind the idea of studying reviewer behavior.

**NIH Peer Review: Where Should We Go From Here?**

Dr. John J. McGowan, Director of the Division of Extramural Activities
at the National Institute of Allergy and Infectious Diseases (NIAID),
spoke on making the peer review system more efficient. He said that his
presentation describe past, current, and future attempts to expedite the
cycle.

*Expectations*

Dr. McGowan said that the system is under stress from three factors
needed to make it work effectively: the people in the system, the system
itself, and the resources available. Flat budgets increase the pressure.
The research community has expectations of peer review both as
applicants and as reviewers. Applicants wonder about when and if they
will get funding, and, if not, want feedback. Reviewers wonder whether
their time is well spent relative to other pressures. They also wonder
how they can advise an applicant in terms of a future application. The
role as a mentor, though not strictly a peer review function, is
incorporated into the research culture.

Scientists within NIH, including review and program staff, also hold
expectations. The SRA wants to ensure that the best science was
identified and appropriate criteria used to assess the merit and
feasibility of the proposal. The SRA must ensure that reviewers comply
with NIH policies and procedures. Meanwhile, program staff are looking
to see how the science identified maps to their portfolios. They are
looking for insights from the review meetings that are not captured in
the critiques going back to the applicants.

*Review of Pilots*

Dr. McGowan noted that pilots or proposed changes to achieve
efficiencies must be judged against core values. He then introduced a
graphic of the review process, beginning with receipt and ending with a
grant award, to look at various ideas to streamline the process.

[Referrals]{.underline}: In terms of referral and assignment, adding
more staff would speed up the process, but the resources are not
available for more hiring. Another option is to streamline the process.
For example, 4 years ago, a prototype of a tool to scan applications was
tested, similar to one used by *Nature*. This tool helps find potential
reviewers, identify conflicts of interest, make study section
assignments, and refers applications to the appropriate IC. However,
funds are not currently available to implement the prototype, and NIH is
currently focusing its knowledge management technology efforts on other
critical efforts. Developing this technology or hiring more staff could
cut the process from 3 months to 1 month. Dr. McGowan explained that
many SRAs have built their own local solutions to streamline their work
using Access or other software.

[Meeting preparation]{.underline}: Dr. McGowan reported on several
pilots to streamline meeting preparations. In one, reviewers selected
the applications they would review in a process that was moderated by
the SRA. These reviewers chose applications closest to their interest
and field of expertise.

Another pilot in 1995 allowed Internet submission of critiques and
scores before a meeting, which has been in use for a number of years and
was wrapped into eRA in 2001. In these Internet-Assisted Reviews (IARs),
scores and critiques are viewed on a secure Internet site.

To eliminate a meeting, reviewers would access the content via phone,
secure Web blog, or e-mail. However, Dr. McGowan said that eliminating a
meeting may be too cumbersome when a study section needs to consider 80
to 100 applications. Reviewers have reacted favorably to IAR, though
more extensive evaluation would need to be done of the costs and
benefits. The bottom line, according to Dr. McGowan, is that IAR could
cut perhaps two months, but reviewers still need 6-8 weeks to review
applications, and SRAs still need time to manage the conflicts of
interest.

Dr. McGowan said that it takes time and commitment to move pilots into
an integrated system and process. In addition, extensive evaluations are
needed of any pilot before one is widely implemented.

[At the meetings]{.underline}: Dr. McGowan discussed increasing
efficiencies in the meeting itself by using three reviewers and virtual
meetings as is sometimes done by NSF. Another option is to triage 70
percent vs. 50 percent of the applications. A third option is to group
applications differently---for example, high, medium, and low bins, with
full summary statements for the "medium" bin and critiques for the
others. Another pilot tested was selective discussion. Because there is
usually high concurrence on most scores before a meeting, discussion
could focus on disagreements or challenges to the literature.

[Production]{.underline}: Dr. McGowan then discussed what he termed
"production aspects" to streamline review. Instead of having the SRAs
write review resumes or summaries, primary reviewers could be asked to
write or dictate resumes at the meeting. In addition, abbreviated
amendments may streamline the process. According to Dr. McGowan, amended
applications represent the bulk of the work in the system. Dr. McGowan
discussed a pilot he conducted in the 1990s, when some applicants who
did not receive funding were allowed to choose to resubmit a revised
application as usual or respond to the issues raised by the study
section. If they chose the latter, the SRA sent them a letter describing
the primary concerns reviewers had about their application. The
applicant had 4-6 weeks to respond, thereby bypassing the receipt and
referral process.

About 60 percent of the applications that went through the abbreviated
process received funding, but applicants who were not funded were
dissatisfied because they did not have scores and summary statements.
Their reaction and the added burden on the SRAs and reviewers led to the
demise of this pilot.

[Councils]{.underline}: Dr. McGowan explained an expedited Council
review pilot in 1994 that shaved 3-4 months from the award process by
having Council members validate grants that were within the funding
parameters before their meeting took place. Unexpectedly, only about 40
percent of the principal investigators who went through this expedited
process accepted an early award.

[Hyper-accelerated review for ongoing clinical trials]{.underline}: Dr.
McGowan described a process called the hyper-accelerated review, which
NIAID initiated and jointly launched with CSR in 1999. The goal is to go
from receipt to award in 3-4 months so NIH could rapidly add needed
studies to ongoing clinical trials. The process, however, is very
resource-intensive and the number of applications involved was very
small.

*Discussion*

Dr. Hammond noted that accelerating amended applications feeds into the
earlier discussion about scoring individual criteria. If an application
does not make the cut because of its significance, the applicant could
not turn the application around in a hurry. Another concern was whether
amended applications are subject to a "ratcheting effect," in which
scores only improve.

Dr. Matt Winkler asked how many of these different experiments have been
implemented, since many appeared to work. Dr. McGowan replied that many
ICs conduct expedited Council reviews, and that IAR has been
mainstreamed. Dr. Winkler suggested that IAR might reduce the influence
of particularly persuasive reviewers. Dr. McGowan said that some members
who used pilots involving IAR thought it prevented bias of one's own
scores.

Dr. Winkler also asked about continuous submissions. Dr. Sassaman said
that the practicalities of the process make continuous evaluation
difficult. She agreed that many pilots require either additional
authorities or resources. She noted that people's increasing comfort
with the Internet has made some review-related activities on the
Internet more acceptable now than 10 years ago.

Dr. McGowan brought up the relative value of the review meeting. The NIH
culture, he said, favors in-person meetings. But perhaps this belief
needs to be measured against the different ways that people now receive
information. He stressed that he is not arguing in favor of doing away
with meetings, but there is no study assessing their relative value.

Dr. Olivia Bartlett said that NCI recently discontinued an accelerated
review process because it became overburdening to the SRAs.

**Pilot: Online Review of R21 Applications**

Dr. Daniel McDonald, Chief, CSR Musculoskeletal Oral and Skin Sciences
(MOSS) and Renal and Urological Sciences (RUS) IRGs, reported on an
online pilot special emphasis panel (SEP) that he recently convened. He
first identified two objectives behind the study: to assess a
potentially more resource-efficient review paradigm, and to relieve the
review burden on the standing skeletal biology study sections in MOSS.
This IRG was formed in 2003, as part of the overall reorganization of
CSR's study sections, with the anticipation that the two study sections
in this area would each review about 60 applications each round. In
fact, the number of grant proposals assigned to each panel immediately
and significantly exceeded these expectations, eventually approaching
about twice that number per study section. A pilot involving online
reviews was developed to find a way to relieve the burden on those
panels.

The pilot targeted R21 applications as well as a small number of R15s
and R03s for a total of 58 applications. Two review panels were formed:
one reviewed 30 applications, while the other reviewed 28. Dr. McDonald
recruited 45 of the 46 reviewers needed via e-mail. Reviewers were each
given a maximum of five written assignments. The reviewers generally had
5-6 weeks to do their evaluations and then submitted their critiques via
IAR.

The entire review was conducted online. To determine which applications
were to be triaged, he developed an initial list of the less competitive
half, based on the preliminary IAR scores. He submitted the list to the
panelists and allowed them 24 hours to respond. As per normal review
procedure, any reviewer, regardless of their specific assignment status,
was permitted to request that a particular proposal be subjected to a
full group discussion, rather than going unscored.

CSR information technology (IT) staff set up individual primary folders
for each application. Panel members had a secure way to log in, but they
were blocked from any with which they might have a conflict. Subfolders
contained the preliminary critiques, discussion, and submission of final
recommended scores at the end of the discussion. A 5-6 day message board
was opened. Dr. McDonald and the respective study section chairs
monitored the discussions closely. At the end of this \"chat\" phase,
the panel members were informed of the range of preliminary scores
assigned. The following day, all reviewers entered their final scores.

The 46 reviewers had a total of 194 assignments. Dr. McDonald noted that
21 of the 58 applications went unscored, which is about the same
percentage that would have been expected to have occurred normally.
Focusing on the 37 that were scored, there were 137 assignments and an
average of 3.3 reviewers assigned to each scored application.

*Outcomes*

Dr. McDonald reviewed the outcome of the comments that occurred during
the \"chat\" phase. A little less than four people commented on each
scored application, as compared to an estimated 4.6 who participate
during a normal hotel-based study section meeting. From 2 to 14 comments
were received per proposal, with 7 as the average. Assigned reviewers
made an average of 5.9 comments per application. Most reviewers scored
most, if not all, of the applications, rather than just their assigned
applications. Dr. McDonald presented data which suggested that the
Internet-only panels tended to distribute scores more effectively than
the two parent panels. The data analysis suggests that the range of
initial scores affected the number of comments, but that the final
scores were not closely related to that number. The range of final
scores from the assigned reviewers was much narrower than their initial
scores.

*Conclusions and Reflections*

Dr. McDonald shared some advantages and shortcomings of Internet review,
but he noted he would probably revise his list after feedback from the
parties involved.

[Advantages]{.underline}: The chief advantages were that these reviews
afforded more flexibility and convenience. Dr. McDonald was also able to
recruit some reviewers who normally would not have participated, and CSR
saved about \$25,000 in travel costs.

[Shortcomings]{.underline}: The major shortcoming was that very few
people other than the assigned reviewers involved themselves in the
discussion. There may be added security vulnerabilities, even with the
IT staff's efforts. The lack of real-time, face-to-face interaction is a
significant issue. The process also was time-consuming for all,
including the SRA, chair, and panelists. Finally, this process
constrains the size of the meeting. Dr. McDonald suggested the
possibility of merging the Internet-only approach with other approaches,
such as teleconferencing or a more focused face-to-face meeting.

*Discussion*

Dr. Martinez asked whether participants had an avatar, a graphic that
people create of themselves to participate in chat rooms, which can help
personalize discussion. He also noted that Internet meetings can take
place in real time. Dr. McDonald said that avatars were not used, but
individuals who submitted comments were identified by name. A real-time
online meeting will be tried soon, Dr. Martinez said that his experience
with online meetings has been positive, particularly if they mean no
airline travel. He also asked Dr. McDonald about the overall outcome.
Besides the issue with the comments, Dr. McDonald said that the outcome
in terms of scoring and other factors compares with regular panels.
Moreover, he said, more discussion might occur over time as people
became more familiar with the technology. Dr. Martinez recommended
exploring the process.

In response to a question from Dr. Ramm, Dr. McDonald said that program
staff had access to comments after the message board was closed, but not
while it was active. In addition, he had what he called a fairly
expected response to his e-mail about which applications to triage: a
few people replied that they agreed with the list, a few recommended
some changes, and the rest did not respond.

Dr. Sassaman lauded this attempt to creatively deal with resource
limitations. She speculated that perhaps an online review could save
time overall, if travel time were included and as people became more
familiar with the system. She suggested using this method with a subset
of applications in combination with other study section activities.

Dr. Martinez cited other benefits of Internet reviews. Reviewers could
potentially take on more assignments because they do not have to spend
time traveling. In addition, perhaps the money saved from the \$12 to
\$15 million CSR spends each year on travel could go to hire more
people.

Dr. Scarpa also said that he saw more advantages than shortcomings. A
major advantage is the ability to recruit a great reviewer who would not
travel to a meeting, particularly those in clinical areas. Another cost
savings to consider is the cost of people leaving their business. While
there might be less discussion online, the discussion could potentially
be more valuable as people have 2 or 3 days to read the application and
contribute to the discussion.

Dr. Pugh congratulated the people involved in the pilot. He saw healthy
feedback in the system. He said it would be interesting to see the final
versus initial scores; ideally, scores would have moved in both
directions to show the effect of the discussion.

Dr. Torok-Storb said that she would not be discouraged by the quantity
of comments if the quality was high. Dr. McDonald agreed, saying that
one could hypothesize that people may be more focused in their comments
because they need to type them. On the other hand, people sitting around
the table like hearing what is happening in the field from colleagues.
Dr. Calhoun called the pilot a promising process, but urged CSR to think
carefully about the applications and study sections on which it is used.
She suggested picking a study section that is not too broad to initially
pilot and iron out the process.

Dr. Kirt Vener, Chief, Special Review and Logistics Branch, NCI, said
that NCI has done similar online-only experiments with the review of 346
loan repayment applications by 140 reviewers. About 80 percent of
reviewers asked agreed to participate, and the reviewer community was
highly receptive to the concept.

Dr. Benzing praised the presentation. He said that he thought that
breaking the applications down into smaller, discrete areas of science
would increase comments by the participants.

Dr. McDonald agreed that there would have been more involvement by the
group as a whole. Part of the pilot involved simulating the panels he
normally prefers, which are broader based scientifically, so he conceded
that he may have created some of his own problems.

Dr. Olivia Bartlett asked how an SRA could serve as designated Federal
official in an online chat, as it is not possible to monitor a chat room
24/7. Dr. McDonald responded that he and the chairs monitored the chats
as frequently as possible, although they would have had to diffuse an
inappropriate comment after the fact. Dr. Bartlett recalled that a
capacitor effect was discussed in the Peer Review Policy Committee:
people load comments during the day, and the SRA reviews what is posted
and releases it at 5 p.m. Dr. Pugh suggested that a knowledge management
system would catch 75 percent of the untoward comments or flag comments
for further review.

Dr. Tatham asked whether the server logs could tease out the extent to
which non-assigned reviewers read the materials pertinent to each
application. The technology used, Dr. McDonald replied, would not
provide this information. Dr. Stanfield suggested a pilot in which
participants certify that they have read the discussion on the
applications before voting.

**New Investigators**

***Possibilities for Compressing the Review Cycle for New
Investigators***

Dr. Donald Schneider, Director, CSR Division of Molecular and Cellular
Mechanisms, presented the work of a small group that looked at ways to
compress the review cycle for new investigators. He referred to NIH
Director Dr. Zerhouni's remarks at the January 2005 PRAC meeting, where
he highlighted his and the extramural community's concern that the
review cycle---15 months from initial submission to possible award---is
too long for new investigators, and in some cases can kill careers.

[Defining a "new investigator":]{.underline} Dr. Schneider noted that
the definition of "new investigator" has changed. Until 1998, the
R29/First Independent Research Support and Transition mechanisms
identified new investigators as people 5 years out or less from their
degrees. Applicants now self-identify via a check box on Form 398. In
some reporting, NIH also defines a new investigator as someone with no
previous R01 or similar competing grant support. Self-identification can
create confusion, such as when senior investigators who have had funding
elsewhere check that they are "new investigators" since this is their
first time applying to NIH.

Currently, reviewers are advised to judge the applications in a manner
appropriate to the applicants' career stage. SRAs also provide
guidelines in mailings to reviewers and orientation at the start of
every review meeting. Dr. Schneider showed how SRAs can easily access
information on how a study section has judged new investigators.

[How new investigator applications fare]{.underline}: Dr. Schneider
presented data from 2001 through 2004 on how R01 applications of new
investigators fared in CSR study sections. About 15 percent of them
scored in the 20^th^ percentile or better, about the same as experienced
investigators with new projects. Experienced investigators with
applications for renewals scored best---about 34 percent of these
applications scored in the 20^th^ percentile or better.

[Resubmitting in the next cycle]{.underline}: The big question is
whether the system could be changed so that new investigators can
resubmit an application in the next cycle. Dr. Schneider "worked
backwards" from study section meeting dates to show that it would
require a tight schedule with little room for slack, but it is possible.
AIDS research applications are now considered on a compressed cycle,
although only about 700 of the several thousand applications received in
each cycle are classified as AIDS applications. About 25 percent of AIDS
new investigators choose to resubmit in the very next cycle. About 40
percent of the new investigators who are successful upon revision say
they could probably have completed their revisions in 2 weeks.

Dr. Schneider presented data showing the effect of how long new
investigators had their summary statements on whether or not they
revised them for the very next cycle and how the length of time seemed
to affect their scores. The amount of time they had the statement did
have an effect on whether or not they resubmitted in the next cycle, but
it did not seem to affect their score after revision. There was
significant improvement in all the scores.

Dr. Schneider said the analysis shows that new investigators could
benefit from getting their summary statements faster. One way to do this
would be to change priorities: instead of preparing summary statements
for the best applications first, summary statements could first be sent
to new investigators. Another possibility would be to send out raw
reviews right away.

Dr. Schneider highlighted challenges to implementing these changes:

-   When a section meeting was held would give some applicants more time
    (for example, early June versus late July), and thus a greater
    advantage, to resubmit in the next cycle;

-   Which mechanisms to target for this process would have to be
    determined;

-   Fairness to experienced investigators would have to be considered;

-   Fairness to SRAs also would have to be considered. They would have
    to receive applications, assign them, find reviewers, and set up a
    meeting in less than 2 weeks.

Dr. Schneider noted that knowledge management and other electronic tools
could help shorten the process. In addition, if receipt dates are
changed because of electronic business practices, there could be a
special date for amended applications and quicker revision.

[Implications for NIH]{.underline}: Dr. Schneider said that complex
interdependencies within NIH affect potential changes. For example, NIH
policy is to prepare summary statements in the order of best-scored
first. However, the data suggest that new investigators can judge
whether they should go ahead and try for a quick revision. He agreed
that the cycle should be shortened, and the challenge is finding a way
to do it, whether just for new investigators or for everyone. In
conclusion, he said submitting an application on February 1 and
resubmitting it in the fall for the following January Council meeting
would shorten the process from 15 to 11 months.

*Discussion*

Dr. Stanfield said that, when summary statements are released "on
paper," they go to the ICs to send to applicants. The amount of time
this takes varies. However, applicants who use the NIH Commons have
quicker access to their summary statements. Dr. Schneider said he found
no evidence that new investigators fared worse if they rushed their
applications back in a revision.

***Staff/Council Interventions Supporting New Investigators***

Dr. Craig Jordan, Director of the Division of Extramural Activities of
the National Institute on Deafness and Other Communication Disorders
(NIDCD), spoke about how NIDCD addresses new investigators under the
F31, F32, R03, and R01 mechanisms.

*F31 and F32*

When NIDCD initiated the pre-doctoral F31 mechanism in 2000, the intent
was a rapid turn-around: 4 months to receive an award or 3.5 months to
receive a summary statement. This expedited cycle was extended to
postdocs applying for an F32. NIDCD uses an existing review committee
that meets three times a year, with SEPs set up for conflicts of
interest. After the meeting, summary statements are prepared first for
the fellowships, which allows pre- and postdoctoral candidates more time
to revise. It also gives more time for the fellowship applications to go
to the Training Board.

The Training Board, made of up staff members across the extramural
programs, meets several days after release of the final summary
statements to make second-level determinations. Generation of pay memos,
sign-off by the Budget Office, and hand-off to the Grants Management
Branch to begin processing the awards all takes place at the Training
Board meeting to expedite the process.

Dr. Jordan showed a time line for a sample fellowship application. The
application was first submitted in December 2002. It required two
amendments in May 2003 and August 2003. The applicant was ultimately
successful and was funded in February 2004. Similar applications that
only needed one amendment were funded in August 2003.

*R03 and R01*

[R03]{.underline}: NIDCD uses the R03 exclusively for unestablished
investigators. He explained the evolution of the program since 1990. Its
goal is to support an investigator to be able to compete for his or her
first, full-fledged independent research grant. The number of
applications received each year has ranged from 96 to 153. The
percentage awarded funds has fluctuated slightly, but Dr. Jordan
characterized the Institute as having a strong ability to support these
individuals.

[R01]{.underline}: In September 2000, the Council began a new process to
look at new investigator R01 applications that fell slightly beyond the
pay line. NIDCD slightly modified the NIH definition of new investigator
to cover investigators within 10 years of their terminal degree,
excluding clinical training, as a way to filter out senior people who
had gotten their support elsewhere.

Dr. Jordan reviewed the information from the CSR Web page about how new
investigators should be evaluated in reviews. NIDCD efforts have
included the following:

-   High program priority: Eighty percent of NIDCD's grant funds are
    awarded based on priority scores. Its Council looks at other
    criteria, such as support for new investigators, when it makes
    recommendations for award the other 20 percent.

-   Coding and tracking: Because of the definition used by NIDCD, the
    Institute developed an in-house coding and tracking system.

-   Protection from cuts in time or amount: When budgets must be
    reduced, new investigator awards are protected in terms of funding
    and duration.

-   Special Council consideration: Staff contacts new investigators
    before a Council round and gives them the opportunity to respond to
    the critiques in their summary statements.

[Applicant letters]{.underline}: Dr. Jordan discussed the letters that
new investigators can submit in response to their reviews. NIDCD staff
contact all new investigators and give them one of three recommendations
depending on their reviews/scores: (1) their application scored very
well and an applicant response letter is probably not necessary; (2)
their application did not score well and a letter probably would not
help answer the concerns; or (3) their application scored in the middle
and a letter may help. Staff spends most of their time with this middle
group, and this group submits most of the letters. Council members
discuss each letter.

*Discussion*

In response to a question from Dr. Sassaman, Dr. Jordan said that NIDCD
did not yet have data on how new investigators fared going from a Type 1
to a Type 2 R01. Awarding the R03 helps a new investigator get an R01,
but Dr. Jordan said there seem to be other factors as well.

Dr. Sassaman said that the National Institute of Environmental Health
Sciences has a transition award for postdocs of \$50,000 for 3 years
once they get an academic appointment.

Dr. Martinez said NIDCD's system for Council review of letters was
innovative, but asked the reaction of other R01 investigators. Dr.
Jordan said he has not heard anything negative. Although he did not have
the numbers, he said that new investigators are doing a bit better than
experienced investigators with new ideas, but not as well as experienced
investigators with Type 2 renewals. He stressed that priority scores are
not changed with the letters and that every new investigator is
contacted about the opportunity to write a letter.

Dr. Jordan responded to a question from Dr. Calhoun about whether the
letter process is used with requests for applications (RFAs) by saying
that it is not, but NIDCD's Council can recommend moving the funding
line or jumping over an application. He noted that this latitude is
available to all program staff and Institutes.

In response to a question from Dr. Stanfield, Dr. Jordan said that NIDCD
only uses this process for applications for which NIDCD has the primary
assignment. NIDCD may convert the application to serve as the primary,
but not if it has already gone to Council.

Dr. Torok-Storb said that the different regulations that affect the R03s
from Institute to Institute make it difficult to keep track of and to
identify the best vehicle for a new investigator. She recommended
streamlining the information so that investigators could better target
their applications. Dr. Stanfield said that the different ways that the
mechanism is used is challenging for reviewers, as well. Dr. Jordan
explained that the SRAs provide the different guidelines to the
reviewers and may ask a program officer to explain an initiative at the
review.

Dr. Torok-Storb asked Dr. Jordan the reasoning behind giving greater
weight to funding new rather than experienced investigators. He replied
that the decision is based on the 1997 report that not enough new
investigators enter the system to replace outgoing ones. Dr. Ruiz Bravo
added that she did not know the "right" number, but that the general
feeling is that there are not enough; in addition, postdocs often talk
about the difficulty of getting first funding. She said two reports from
the National Research Council studied this issue, but they were not in
perfect harmony. One report, *Bridges to Independence*, recommended
moving funds from research to National Research Service Award Fellows.
The other report, an evaluation of NIH training conducted by the
National Research Council every four or 5 years, recommended keeping the
number of fellows constant. Dr. Stanfield said data, though somewhat
dated, suggest that the critical period for a new investigator to become
established as a full-fledged member of the research community is at the
time of first competitive renewals, rather than the first grant.

Dr. Peter Guthrie commended NIDCD's approach, but noted that NIDCD has
about 100 applications in a year---about the same amount that CSR deals
with in a single meeting. CSR has tried to accelerate the process by
separating fellowship applications from the regular R mechanism.
However, resource limitations must be kept in mind in accelerating the
process.

**Oversight of CSR Study Sections that Review Small Business
Applications**

Dr. Elliot Postow, Director of the CSR Division of Biologic Basis of
Disease, began his presentation by noting that, in 2003, the CSR
Advisory Committee examined the possibility of chartering small business
study sections. The reasons were to "regularize" their activities so
they are more like R01 study sections, provide additional oversight, and
increase the consistency of reviews. The most persuasive reason against
chartering was the difficulty in obtaining long-term commitment from
reviewers who come from small businesses. Another reason was the
variability in expertise needed from one meeting to the next because of
the range of science covered by Small Business Innovative Research
(SBIR) study sections. Recidivism---the probability that a person had
attended two of the last three meetings---was close to that of chartered
study sections. If SBIR study sections were chartered, however, new and
significant burdens would be placed on committee management staff as
well as other review staff.

Dr. Postow added that he had found no truth to the "urban myth" that
some people serve on SBIR sections indefinitely.

The question that Dr. Postow said he would focus on is how to provide
additional oversight and consistency of review in SBIR and Small
Business Technology Transfer Program (STTR) study sections without
chartering them.

*SBIR/STTR Study Sections*

The number of these applications has grown from about 1,400 a round in
2000 to about 2,200 a round in 2004--2005, and they represent about 13
percent of the CSR workload. Dr. Postow compared the demographics of
SBIR/STTR reviewers to those in chartered study sections. The percentage
of women reviewers on the SBIR/STTR study sections (21.7 percent) is
lower than on chartered study sections (27.8 percent). The average age
is about the same. The percentages of minorities and of underrepresented
minorities are lower for SBIR/STTR study sections than for chartered
study sections. The geographic distribution is only slightly different.

Dr. Postow said that chartered study sections have reviewers with
greater seniority, though this is in part because of the nature of
SBIR/STTR applications. Finally, Dr. Postow said about 16.7 percent of
the reviewers are from small businesses, and 1.5 percent are from large
businesses, which is lower than the 25 percent that NIH would like to
achieve. However, this figure is probably an underestimate because
reviewers with both academic and business affiliations are listed with
their academic affiliation.

Only 15.6 percent of Phase I small business applications received
priority scores of less than 200. However, 25.9 percent of the Phase II
small business applications score better than 200, which corresponds
more closely with the R01 average of 28 percent and R21-R03 average of
22 percent that score 200 or better.

Dr. Postow concluded that the environment in which SBIR/STTR
applications are reviewed is similar to that found in study sections
reviewing R01 grant applications.

*Two Possible Oversight Mechanisms*

Dr. Postow proposed two oversight mechanisms: (1) an annual "state of
the study section" report to the IRG chief and division director, and
(2) working group evaluation of all small business study sections in a
division in a concerted and consistent way.

[State of the Study Section Report]{.underline}: The SRA would prepare
the report with an emphasis on the previous year's activities. The
report would discuss the types of applications received, workload,
changes in workload, and relationships with other small business study
sections. He envisioned that the report would also discuss the customers
and ICs the study section supports, raise any issues to CSR that the SRA
feels should be addressed, and discuss reviewer participation, including
demographics. The report could also capture the small business
experience of study section members who list themselves on the roster
with academic affiliations. The report could also cover end users and
the length of service and prior service of the chairs and members.

[Working Group]{.underline}: The small business working group would be
conducted on a 5-year basis and begin by developing questions for
individual evaluators of study sections. A minimum of three evaluators
would conduct the evaluations over three meetings. The plan would be to
select evaluators after the meeting, so that they are part of the study
section and not perceived as outsiders. Dr. Postow said that most
evaluators would be experienced reviewers, with an occasional new
reviewer to evaluate how new reviewers are trained and integrated into
deliberations. The evaluators of the individual study sections would
meet by telephone to discuss their observations and develop a report.
Then the division director would choose a representative from each group
to develop a report about all that division's SBIR study sections. The
report would ultimately come to PRAC, as well as to the CSR Director.

*Discussion*

Dr. Winkler thanked Dr. Postow for the reasonable proposal to address
some concerns that he had about the SBIR/STTR study sections. He
suggested characterizing the current system as "not broke" and that
these controls would make sure that the system stays that way. Dr.
Stanfield noted that applicants want to know the expertise of the
reviewers, and the report could provide that information. Dr. Winkler
agreed that more transparency would be welcomed.

Dr. Pugh suggested that the SBIR review process might lend itself to a
pilot study to compare online and face-to-face reviews, as might
fellowship applications. He also noted that the proposals advanced by
Dr. Postow could require a good bit of extra work for the IRG directors
and the SRAs. Dr. Postow agreed that SBIR would lend itself to
electronic review, even more so than fellowships. He said that the
working group evaluations would not add extra work, as these evaluations
would have occurred in any event.

Dr. Hammond said that Dr. Postow's description of SBIR study sections
makes them sound like a hybrid between chartered study sections and
SEPs. He wondered whether it would be feasible to have chartered study
sections with a small group of members and have temporary members coming
on board as needed. Dr. Postow replied that the most convincing reason
against chartering 2 years ago was the unwillingness of small business
people to commit to 4 years.

Dr. Calhoun said that 4 years may not be appropriate in some cases.
Perhaps the term for a core group could be 2 years for small business
and 3 years for academics.

Dr. Vener described the SBIR process at NCI, where the selection of
reviewers is driven by the science. The greater the flexibility in the
mechanism to bring in reviewers, the better, considering that the
science is so fast-moving.

**Flat Rate Reviewer Reimbursement**

Mr. David Whitmer, CSR Executive Officer, spoke about changes in the
Scientific Review Evaluation Awards (SREA), the funds that NIH uses to
reimburse travel, lodging, per diem, honoraria, and meeting costs for
study section meetings. They currently fall under the "grant-other" line
in the budget, but will be changed to the research management services
(RMS) budget. The total CSR cost is about \$47 million per year, about
\$12 million of which is airline travel.

*Office of Management Assessment Report*

The NIH Office of Financial Management asked the Office of Management
Assessment (OMA) to review the program. The resulting OMA study, which
was released in May 2004, found inadequate internal controls for
oversight and accountability; high risk of fraud, waste, and abuse;
unreliable, incomplete, or inaccurate data; and inconsistent treatment
of similar costs. Based on the report, OMA recommended---

-   Centralizing program administration in CSR, which already processes
    about two-thirds of the payments, with an SREA point of contact in
    each IC;

-   Developing a flat-rate payment system to reimburse reviewers so that
    they would not have to submit receipts;

-   Issuing electronic payments through the Office of Financial
    Management;

-   Funding the program through RMS funds to retain eligibility for
    government travel rates;

-   Transferring funds from grants to RMS in the fiscal year (FY) 2006
    budget;

-   Appointing a group to implement these recommendations.

Mr. Whitmer said that the Extramural Activities Work Group and the
Management and Budget Working Group reviewed and endorsed the OMA report
and recommendations, and the NIH Steering Committee approved the
recommendations.

*Proposed Changes and Next Steps*

According to Mr. Whitmer, the transfer to the RMS line has spurred
looking for ways to save money. One way is through blanket purchase
agreements with hotels. These agreements would produce terms more
favorable to NIH, get SRAs out of hotel negotiations, and save NIH about
\$1.2 million in local taxes. The flat rate reimbursement also will free
up staff who have been processing vouchers. Reviewers will not have to
provide receipts and vouchers. Finally, NIH will negotiate lower travel
management fees. A model is being developed with electronic deposit of
funds into reviewers' accounts a few days after a meeting.

The SREA Transition Team, which Mr. Whitmer chairs, was established to
make these changes. The team includes representation from the extramural
scientific community, as well as different NIH staff, with the task to
put together a system that has no negative impact on peer review while
saving money. All data will go through the IMPAC II system, rather than
the current four different data sources. Subcommittees are looking at
different aspects of the changes, including a Peer Review Impact
Subcommittee. Next steps include the following:

-   Finalizing logistics of the flat rate payment, which will consist of
    the honorarium, per diem, and \$160 per trip to cover incidentals;

-   Completing the hotel contract, which a contractor will negotiate
    with the approximately 75 hotels that NIH has used in the D.C. area;

-   Determining how to administer the program, whether through a
    contract or the Office of Financial Management;

-   Creating NIH-wide financial management systems for tracking and
    accountability;

-   Conducting outreach to the reviewer community.

*Discussion*

In response to a question from Dr. Martinez about airline travel, Mr.
Whitmer explained that reviewers could use the Government rate, but that
some reviewers report they can find less costly flights on their own.
The team is trying to create flexibility. Dr. Martinez related his
experience with the NSF model. He warned that the quality of hotels
assigned to reviewers have been a problem.

Dr. Ramm said that she thought reviewers would like the fact that
payments will be more timely, but was concerned about transferring funds
into RMS and the effect of site visits on tight funds. She also
expressed concern about how ICs would control their funds. Mr. Whitmer
said that the SREA budget will go to the Extramural Activities and the
Management and Budget Working Groups to determine the resources that
should be dedicated to the program. He also said that costs under the
HHS travel contract would equal about 7 percent of the total paid to
reviewers.

A cost-benefit analysis will determine if it is better to contract out
or manage services in-house.

Dr. Sassaman agreed that there were advantages to the new system for
both IC staff and reviewers, but she praised the flexibility in the
existing system. She requested that the hotel negotiations include high
standards for speakerphones and other communication technology.

Mr. Whitmer said that the contract would include chains with hotels
throughout the country. If the decision were to use the NIH events
management contract as a vehicle, there would be task orders and the
contractor would have to meet specified requirements

Dr. Scarpa stressed that the quality and location of the hotel are top
on the list of reviewer satisfaction. Dr. Berg brought up the tax
ramifications so that reviewers do not pay taxes on reimbursed expenses.
Mr. Whitmer said an IRS 1099 reflects honoraria only.

Dr. Calhoun asked about the leftover funds from the 53 current SREA
accounts. Mr. Whitmer said that a plan to draw down that money, about
\$700,000 to \$800,000, is being worked out. Unfortunately, he said,
funds cannot be carried over from one year to the next.

Dr. David Maslow, an SRA at NCI, asked whether an evaluation plan will
look at the effect of hotels and reimbursements on recruiting reviewers.
Mr. Whitmer said users of the system are involved in the transition
team, and that cutting costs will not result in lesser-quality hotels.
Dr. Ruiz Bravo stressed that, as the era of governance of this trans-NIH
budget is launched, the peer review principles must be kept in mind.

Mr. Whitmer said that the committee also recommended that NIH use a
direct "CAN citation," whereby the IC sponsoring the review pays for the
services actually used, rather than an estimate. He also said that the
adjustments to the \$160 flat rate will be made in the future by
collecting data from reviewers, such as how many need to file for an
exception.

**NIH Extramural Loan Repayment Programs**

Dr. Stanfield introduced Dr. Robert Hammond, PRAC member and Director of
the Division of Extramural Activities of the National Institute of
Diabetes and Digestive and Kidney Diseases (NIDDK). Dr. Hammond also
chairs the NIH Extramural Loan Repayment Program (LRP) Oversight Panel.
Dr. Stanfield noted that Dr. Hammond was retiring, which he called a
great loss to NIH because of Dr. Hammond's work across NIH as a
consensus builder.

Dr. Hammond explained that the Extramural LRP is a contract mechanism
that allows NIH to repay, at least in part, outstanding educational debt
to encourage postdocs to stay in research. There is no Council review or
summary statements, but the principles of peer review are followed.
There are five LRPs, two available across NIH and three specific to
Institutes.

*Trans-NIH LRPs*

[About the Programs]{.underline}: The two trans-NIH LRPs---Clinical
Research and Pediatric Research---use a totally electronic process. In
exchange for a 2-year commitment to a research career, NIH repays up to
\$35,000 per year of qualified educational debt, an additional 39
percent to cover Federal taxes, and, in some cases, state taxes. The
programs are open to U.S. citizens and permanent residents with a
doctorate. At the time of the actual award, the qualifying debt must be
equal to or in excess of 20 percent of the applicant\'s annual income.
The applicant must be engaged in qualified research responsive to the
specific LRP for at least 2 years at 50 percent effort or above, and the
research must be supported by a nonprofit or U.S. Government entity.

[Clinical Research LRP]{.underline}: Dr. Hammond reviewed the Clinical
Research LRP for FY 2004. About 1,200 new and 200 renewal applications
were submitted, and slightly more than 50 percent of the new and 90
percent of the renewal applications were funded. The five LRPs' success
rate is comparable. More than half went to applicants with M.D. or
M.D./Ph.D. degrees, and most were within 5 years of receipt of their
degrees. Currently, ICs are reviewing applications for FY 2005 as
contract proposals. Dr. Hammond said that reviewer feedback on the
Internet review process is generally positive, and an evaluation of the
entire program is being developed.

Dr. Hammond reviewed the six criteria, which focus on the individual's
abilities and potential as a researcher.

Applicants with competitive scores are then asked to provide the
required loan documentation to ensure eligibility. Those with
noncompetitive scores can contact the Institute for feedback. There is
no appeal, but applicants can reapply. Each IC has a pre-announced
portion of the \$59 million NIH LRP set-aside. Strong applications not
funded in a particular Institute go into a pool, where another Institute
might pick them up. The administrative cost is \$2,100 per successful
application and \$1,100 per unsuccessful application. Other challenges
include finding reviewers who understand the goals of the program. He
added that the amount awarded for Type 2 renewals is under review.
Currently, applicants do not have to meet the 20 percent income
threshold. Besides cutting into the amount available for Type 1
applications, Dr. Hammond observed that it is very time-consuming for
the reviewers to review a lot of Type 2 applications for very small loan
repayment amounts.

*Discussion*

Dr. Torok-Storb praised the program and asked whether D.V.Ms. are
eligible. Dr. Ramm replied that the necessary legislation to include
these scientists in the clinical LRP has been put forward, but so far
without success. Researchers with D.V.Ms. are eligible for all other
LRPs. Dr. Torok-Storb noted many talented D.V.Ms. would stay in clinical
research if they did not have to repay huge loans.

In response to a question from Dr. Torok-Storb, Dr. Hammond explained
that evaluation forms go to the program director, who can provide
feedback to the applicant. Applicants do not receive the form, in part
because the score, while the most important factor, is not the sole
factor.

Dr. Ramm praised Dr. Hammond for his contribution to the LRPs and
thanked him and CSR for making appropriate referrals to her Institute.
She also asked whether the Loan Repayment Office tracks awardees. Dr.
Hammond said they report quarterly during the 2-year commitment. It was
noted that a phase I evaluation of the program is underway.

**Final Business and Discussion**

*New Oncology Study Section*

Dr. Stanfield explained that "housekeeping" matters from CSR are
occasionally brought before PRAC, such as the creation of new study
sections. He asked Dr. Syed Quadri, Chief of the CSR Oncological
Sciences IRG, to explain a new study section in his IRG. Dr. Quadri said
that 13 study sections were created in 2002 for this IRG. Two of them,
Drug Discovery and Molecular Pharmacology and Development Therapeutics
have grown so much that a SEP, Basic Mechanism of Cancer Therapeutics,
was created, as suggested by a working group. At this point, 220
applications are distributed among the three sections. He reviewed their
guidelines and said that all three are pleased with how things are
going. He requested PRAC approval to charter the Basic Mechanism of
Cancer Therapeutics Study Section.

Dr. Ruiz Bravo asked how a study section, rather than a SEP, is created.
Dr. Quadri replied that the decision is based on the workload and the
science. As the number of applications grows, the SRAs first may form a
SEP. After several meetings, if it seems necessary, they might propose
turning the SEP into a chartered study section. Working groups are
formed for this purpose, and they report to PRAC. Dr. Ruiz Bravo
requested a list of working groups so PRAC is aware of upcoming issues.

In response to questions from Drs. Calhoun and Torok-Storb, Dr. Suzanne
Fisher, Director of the CSR Division of Receipt and Referral, named
other study sections that address drug and vaccine development. Dr.
Schneider noted the growth in the chemistry and basic bioengineering
areas. In the next year, working groups will be formed to look at these
areas. Dr. Quadri added that the Roadmap Initiative and program
announcements are also considered to make sure that applications will
fit the study sections in the longer term.

*Closing Business*

Dr. Stanfield asked PRAC members to recommend additional members for
PRAC and thanked members for the suggestions they have already provided.

He said that a part of the community has requested that PRAC discuss how
CSR and NIH in general handle member conflict applications. If PRAC
members agree, this item will be on the agenda at the next meeting. Dr.
Stanfield also urged members to give him or Dr. Berg any feedback about
the meeting.

[Action items]{.underline}: He then suggested summing up the action
items that came from the day's meeting. He said he welcomed additions to
the following initial list:

-   Core values: Work needs to continue on developing the core values,
    particularly with input from external stakeholders. An
    implementation plan needs to be developed that could be used when
    CSR or NIH is considering changes to the peer review system
    (including, it was added by Dr. Hammond, SREA issues);

-   Scoring of individual criteria: A pilot should be considered in
    which each criterion receives a separate rating to see how that
    impacts on peer review;

-   Online meetings: Exploration of online meetings should continue. An
    evaluation should consider all impacts, including the impact on
    reviewer time and effort. Perhaps, as suggested earlier, the SBIR
    mechanism and fellowship applications would be good pilots.

-   ICs' different use of mechanisms: Resources need to be developed
    that help explain how different ICs may use mechanisms and treat new
    investigators differently;

-   SBIR: Plans for the SBIR oversight mechanisms should go forward,
    with consideration of the extra time burdens that the new
    requirements may place on SRAs;

-   LRP: An evaluation of this program should go forward. Perhaps the
    evaluation plan for the intramural program, which is further along,
    can be presented to PRAC for ideas about how a similar or expanded
    evaluation can be conducted for the extramural program.

Dr. Stanfield then asked for additional action items.

[Rating scale]{.underline}: Dr. Martinez suggested discussions about
whether the rating scale is being used scientifically. Dr. Ruiz Bravo
suggested pooling the data and taking it to the governance system as a
start. Dr. Stanfield asked if it was the sense of the group to convey to
the governance internally that PRAC feels that this is an issue that
should not be dropped and that NIH must be able to ensure that the way
grant applications are rated is rational. Dr. Pugh agreed, but said that
it was not clear that a better alternative exists. Facts are needed
first about how the system is behaving. Dr. Ruiz Bravo suggested
gathering the scientific information by which one creates a rating
scale, as well as facts about how the system is working now. Dr.
Martinez reminded the group that Dr. Johnson had said that the system
had biases. Dr. Pugh said that his interpretation was that Dr. Johnson
was describing a hypothetical situation. He said that it is important to
learn whether actual situations reflect this hypothetical one. Dr.
Stanfield suggested that Dr. Johnson's study could inform PRAC about
some of the information they feel they need. Dr. Karl Malik said that
Dr. Johnson will soon start collecting data and probably have
preliminary data in May 2006. Dr. Calhoun suggested also collecting data
on the patterns of voting on multidisciplinary versus single discipline
applications. Dr. Malik said this might be a second line of analysis.

[Minority involvement in clinical studies]{.underline}: Dr. Martinez
reminded Dr. Stanfield about communication PRAC members had received
about including minorities in clinical studies. He wanted to make sure
that this issue was not lost.

[D.V.Ms. in the LRP]{.underline}: Dr. Stanfield suggested that PRAC
members recommend to Dr. Ruiz Bravo that NIH do what it can to ensure
that D.V.Ms. are including in the LRP.

[Summary statement formats]{.underline}: Dr. Torok-Storb suggested that
summary statements include a cover sheet with a list of 10 queries that
can be checked 1 though 5. She thought this might help focus the summary
statements and make the information clearer to the applicants.

At the close of the discussion, Dr. Stanfield thanked the participants
and presenters and adjourned the meeting at 5:16 p.m.

We do hereby certify that, to the best of our knowledge, the foregoing
minutes of the second meeting of PRAC are accurate and complete. The
minutes will be considered at the third meeting of the Committee, and
any corrections or comments will be made at that meeting.

> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
>
> Karl Malik, Ph.D.
>
> Executive Secretary
>
> Peer Review Advisory Committee
>
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
>
> Brent Stanfield, Ph.D.
>
> Co-Chair
>
> Peer Review Advisory Committee
>
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
>
> Jeremy Berg, Ph.D.
>
> Co-Chair
>
> Peer Review Advisory Committee