Peer Review Advisory Committee Meeting

National Institutes of Health

U.S. Department of Health and Human Services

January 24--25, 2005

The first meeting of the Peer Review Advisory Committee (PRAC) convened
at 8:30 a.m. on Monday, January 24, 2005, at the Bethesda Marriott,
Bethesda, Maryland. The entire meeting was held in open session. Dr.
Brent Stanfield and Dr. Jeremy Berg presided as Co-Chairs.

## Members

Jeremy Berg, Ph.D., Co-Chair Edward Pugh, Jr., Ph.D.

Brent Stanfield, Ph.D., Co-Chair Louise Ramm, Ph.D.

Faye Calhoun, Ph.D. Anne Sassaman, Ph.D.

Robert Hammond, Ph.D. Beverly Torok-Storb, Ph.D.

Joe Martinez, Ph.D. Matt Winkler, Ph.D.

Craig McClain, M.D.

Dr. Karl Malik was the Executive Secretary for the meeting.

# Welcome and Opening Remarks

Dr. Stanfield welcomed members and participants to the inaugural meeting
of PRAC and thanked them for their attendance. He asked the PRAC members
to introduce themselves and briefly describe their backgrounds. He also
noted that PRAC has five vacancies and asked members to suggest
individuals or types of experience that could benefit PRAC's work.

# Charge to the Committee

Dr. Elias A. Zerhouni, Director of the National Institutes of Health
(NIH), summarized the history of peer review oversight at NIH to explain
how the PRAC came into being. Before 1996, the Division of Research
Grants (DRG) Advisory Committee advised the DRG Director on peer review
issues. Because the issues often extended across NIH, NIH Director Dr.
Harold Varmus created the Peer Review Oversight Group (PROG) in 1996 to
address review policy issues common to the entire NIH. PROG reported to
the Deputy Director for Extramural Research. DRG became the Center for
Scientific Review (CSR) in 1997, and the CSR Advisory Committee focused
on CSR-specific policy and administrative issues, though it often
encountered issues beyond its purview.

Dr. Zerhouni said that, when he became NIH Director, he found that there
had not been a review of how NIH governs itself in many years, even as
the organization had grown in complexity. A multitude of committees
existed. Through extensive discussions, a Steering Committee was created
about 18 months ago, consisting of nine NIH Institute and Center (IC)
Directors. This committee serves as the central place to make "corporate
decisions" that affect the entire NIH.

Dr. Zerhouni explained that the PRAC combines the goals of the CSR
Advisory Committee and PROG. The new advisory committee will focus on
peer review policy and operations in the context of the overall
extramural program. PRAC is charged with advising the NIH Director,
Deputy Director of Extramural Research, and CSR Director on matters
relating broadly to review procedures and policies for the evaluation of
scientific and technical merit of applications for grants and awards.
Dr. Zerhouni emphasized that, as a point of departure from the past,
PRAC can also advise IC Directors on the scope and manner of peer review
operations within their respective ICs. When advice goes to the Deputy
Director of Extramural Research, it also goes to the Extramural
Activities Working Group (EAWG) that the Deputy Director co-chairs and
that reports to the Steering Committee. This approach will help
harmonize NIH peer review.

He then summarized PRAC's structure and functions: (1) the CSR Director
and another NIH IC leader serve as co-chairs; (2) 16 people with
relevant backgrounds from within and outside NIH serve on PRAC, with
high-level NIH leadership serving both on PRAC and on EAWG; (3) terms of
service last up to 4 years, with meetings about 3 times per year; and
(4) PRAC can create subcommittees and working groups to consider
specific issues and take on other activities with the goal of fostering
harmonization across NIH.

Dr. Zerhouni said that the topic of peer review usually comes up within
30 seconds of any meeting he has with grantees. They frequently are
concerned about the harmonization, speed, and effectiveness of peer
review. He emphasized how important speed is to the community by noting
how when he was a vice dean of research, his institution and young
investigators bore a great burden because the review process is a long
one. The time from submission to notification takes 9 months, and only
20-25 percent of new applicants receive funding the first time they
apply. Those who do not receive grants may need to spend 4-5 months to
get new data and revise their application and then must wait an
additional 9 months for the review of their revised application. Dr.
Zerhouni noted that many grantees also are concerned that NIH may not be
recruiting the best reviewers, and the time reviewers must commit to the
process may prevent many qualified scientists from serving.

Dr. Zerhouni urged PRAC members to take up a number of issues:

-   The impact of IT on the peer review process,

-   The length of time for the review process,

-   The review process itself, particularly the challenges posed by a
    rapid increase in interdisciplinary research and the perception by
    some that the peer review process tends to be too conservative,

-   The need to come up with pilots with testable hypotheses to improve
    the review system,

-   The ability of NIH peer reviews to serve all research areas
    effectively,

-   The need to find ways to encourage highly qualified
    scientists---particularly clinical researchers---to be reviewers

-   The pressure to maintain quality as the number of applications
    increases.

Dr. Zerhouni concluded by telling PRAC members that he looks forward to
their innovation, as well as their ability to provide integration and
harmonization of peer review for the entire NIH.

Discussion

Dr. Edward Pugh said he particularly appreciated Dr. Zerhouni comments
on advancing an evidence-based process to test various improvements to
the peer review system. Dr. Zerhouni stressed that the philosophy
throughout NIH is not just to continue with things because they seem to
work or to make changes without evidence, but rather to experiment with
new approaches. Dr. Stanfield noted that some funding for these kinds of
evaluation studies is available through the Office of Science Policy.
Dr. Beverly Torok-Storb suggested that an information technology (IT)
person and a biostatistician might be needed as PRAC members.

Dr. Zerhouni said he was pleased how well the NIH's new advisory
committees helped to facilitate a change in the Federal rule that now
allows grants to multiple investigators, thereby addressing a need to
advance collaborative and interdisciplinary research.

# NIH Governance Structure and PRAC's Role in Facilitating Harmonization 

*Bird's Eye Overview of NIH*

Dr. Norka Ruiz Bravo, Deputy Director for Extramural Research, presented
an overview of NIH governance, management operations, and science.
Approximately 84 percent on the total NIH budget of \$28.8 billion is
spent outside the agency. NIH has played a major role in many medical
achievements, and it is naturally a complex organization, which has
policies and operations that need to evolve as science changes. She
noted that the new NIH Steering Committee that

Dr. Zerhouni discussed has five Working Groups, including the Extramural
Activities Working Group and two others of particular interest to PRAC:
IT, and Management and Budget Working Groups.

In explaining how NIH works, Dr. Ruiz Bravo emphasized that the peer
review process is the cornerstone of NIH. In terms of operations, NIH is
attempting to achieve administrative efficiencies in the entire grant
process. For example, the Division of Extramural Activity Support (DEAS)
now provides the support staff, organized in three hubs, to serve the
ICs. She acknowledged that implementing DEAS has been challenging and
she has received both positive and negative reactions from IC staff, but
that it was a necessary change. They are also working on identifying
best practices for grants management and consolidating training of
grants managers.

As science changes to a more systems approach, carried out across
disciplines in teams, this implies changes in NIH. However, it does not
mean that this is the only kind of science that NIH will be funding. Dr.
Ruiz Bravo also noted that changes cannot happen overnight, because of
policies, procedures and practices, as well as IT systems.

## Key Challenges

[Management and operations]{.underline}: NIH must continue to achieve
administrative efficiencies.

[Competitiveness of funding]{.underline}: The time of the doubling of
the NIH budget is over, and NIH is budgeted for about a 3 percent
increase in fiscal year 2005. About one-half of the applications for
renewal of research projects are funded.

[Difficulty of new investigators to get funding]{.underline}: The
average age at which a new investigator receives his or her first
independent research grant is rising, particularly as more new
investigators are entering the system than experienced investigators are
leaving it. New investigators continue to compete for about a 25 percent
share of the NIH "pie." She noted resources available for new
investigators (http://grants2.nih.gov/grants/new_investigators) and
invited PRAC members to e-mail her with other ideas.

[Evolution of public health challenges]{.underline}: NIH must help
address many public health challenges, including chronic conditions, an
aging population, health disparities, emerging diseases, and biodefense.
The NIH Roadmap, which establishes a framework of priorities that NIH
must address in order to optimize its research portfolio, is helping to
address these issues. Just as importantly, Dr. Ruiz Bravo noted that
working on the Roadmap also showed a way for the ICs to work together on
common issues.

[Development of knowledge-based management of science]{.underline}: NIH
can use new technology to transfer and share knowledge among people who
need it in such areas as disease coding, peer review, and portfolio
analysis. The ideal NIH decision support system would integrate data,
models, communication, documents, and knowledge.

## Discussion

Dr. Joe Martinez asked Dr. Ruiz Bravo about the role of NIH in the
training of doctoral and postdoctoral students. He observed that knowing
the number of trainees supported on an R01 grant would provide an idea
of people coming into the system, and that training could include
helping new investigators submit grants that have a better chance of
being funded.

Dr. Pugh noted the many pressures that could overwhelm the peer review
system, and he stressed the importance of keeping it honest, fair, and
free of conflict of interest. He also observed that meeting some of the
challenges outlined by Dr. Ruiz Bravo will be difficult. For instance,
bringing along more new investigators will be difficult with an
increased emphasis on system-level and multidisciplinary research. Dr.
Pugh then said he had concerns about speeding up the peer review process
by applying inappropriate business models and related notions of
efficiency. He concluded by saying that there are distinctive and
valuable aspects to the current system, and with adjustments, the NIH
peer review system could serve as a working model of good government.

Dr. Louise Ramm noted that some training grants are addressing how new
investigators fit into research teams. Dr. Anne Sassaman suggested
publicizing the Web site for new investigators, which she praised as a
great resource. She then suggested that PRAC look at the role of peer
review on new investigators entering the system. Dr. Sassaman also
recommended that PRAC members discuss at a future meeting how the system
does or does not respond to proposals for interdisciplinary team
research as well as how well the system meets the needs of new
individual investigators.

# CSR and the Office of Extramural Research (OER): Intersections and Divergences

*The Ties that Bind: The Linkage Between CSR and OER*

Dr. Anthony Demsey, Director, Office of Extramural Policy at the
National Institute of Biomedical Imaging and Bioengineering, presented a
historical overview of how CSR and OER have come to interact in dealing
with extramural research.

After reviewing the CSR and OER missions and relevant manual chapter on
review responsibility (Manual Chapter 4510), Dr. Demsey presented a
history of health research grant policy in the Federal Government,
beginning in the early 1900s when Congress legislated authority to the
Marine Health Service (MHS) to serve as a clearinghouse for scientific
information and coordinate the scientific work in industrial and
university laboratories.

Dr. Demsey noted that a one-room laboratory created within the MHS
hospital on Staten Island, New York, eventually grew into NIH.

By the end of World War II, Federal support of biomedicine was firmly
established, as was the concept of peer review. NIH assumed the
military's biomedical research portfolio as the country made the
transition to peacetime. CSR's predecessor, DRG, was established to
award and administer grants and other aid for biomedical and
health-related research. In 1948, the DRG Director encouraged the
formation of a central Office of Research Planning (ORP) outside of DRG.
Under this structure, ORP coordinated extramural policy and programs,
while DRG was responsible for administration and information service
activities for extramural grants.

In the 1950s, DRG was reorganized to focus on five functions: (1) audits
of grants;

\(2\) scientific evaluation of grant applications; (3) maintenance of
program statistics;

\(4\) preparation and distribution of grant information; and (5)
technical evaluation of program activities. The new National Institute
of General Medical Sciences (NIGMS) took responsibility for research
grant and training programs in non-categorical areas. The net effect was
that DRG shifted toward centralized service functions and away from
policy. Instead, the NIH Executive Committee on Extramural Affairs
implemented policy.

In the 1960s and 1970s, various attempts were made to reform the grants
process, and DRG's organizational location changed several times. The
Cancer Act of 1971 further favored non-DRG-reviewed funding mechanisms.
After 5 years of being under the Associate Director for Extramural
Research and Training, DRG again reported directly to the NIH Director
in 1974. This change and DRG's participation in the Grants Peer Review
Study Team strengthened DRG's role in extramural policy-making.

In the 1980s, NIH Director James Wyngaarden worked with Dr. William
Raub, Deputy Director for Extramural Research, to establish the Review
Policy Committee as the focal point for trans-NIH peer review issues. In
1989, Dr. Jerome Green, then DRG Director, formed the DRG Advisory
Committee, which is a predecessor to the CSR Advisory Committee and
PRAC.

In the 1990s, NIH Director Varmus asked Dr. Marvin Cassman, then
Director of NIGMS, to chair a committee to look at review issues. The
Cassman Committee recommended moving the Information Systems branch out
of DRG and into OER. This committee also recommended forming the Peer
Review Oversight Group. Though some members favored moving the DRG back
under the Deputy Director for Extramural Research, this recommendation
was not implemented.

## Discussion

Dr. Sassaman asked how some of this history could influence thinking in
the future. Dr. Demsey recommended balancing new thinking by revisiting
some of the past solutions to see if anything can be learned. In
discussing the responsibilities of OER versus CSR today, Dr. Ruiz Bravo
and Dr. Stanfield both said OER looks at trans-NIH issues, but not in
isolation. Input is needed from CSR and others. Dr. Stanfield expressed
the hope that CSR could pilot a number of improvements that may have
broader application across NIH, and he hoped that PRAC members would
provide both CSR and OER advice for moving in that direction.

Dr. Pugh said that the nation's early leaders had a set of core values
that they struggled to maintain while ensuring that the nation could
function. Just as enunciating the core values then was important, he
urged PRAC members to articulate the core values of the peer review
system. Suggested changes could then be looked at against this
"touchstone" of core values. Dr. Stanfield agreed that this would be a
good idea for PRAC members to undertake, and he later asked that a list
of these core values be drafted by Dr. Alan Willard, Scientific Review
Branch Chief of the National Institute of Neurological Disorders and
Stroke (NINDS) and chair of the NIH Review Policy Committee.

# CSR and Institute/Center Review

***History***

Dr. Anne Clark, Associate Director, CSR Division of Receipt and Referral
(DRR) presented a chronology of CSR and IC review. In 2004, CSR reviewed
47,000 applications in 24 Integrated Review Groups (IRGs), and 23 of the
ICs reviewed a total of 21,000 applications. She posed three questions
in her presentation:

1.  How did the NIH peer review system get to where it is now?

2.  When did its central features appear?

3.  What events contributed to the current distribution between CSR and
    IC review?

Dr. Clark acknowledged her sources for her presentation, particularly
Richard Mandel's *A Half Century of Peer Review, 1946--1996*. She showed
a timeline to present some of the key features of the system of peer
review of extramural grants, noting five developed in the 1950s remain
in place today: (1) selection of nongovernmental scientists as
reviewers, (2) separation of program and review functions, (3) recusal
of reviewers for conflicts of interest, (3) assignment of referees or
assigned reviewers to lead the discussion, and (5) priority ranking of
proposals (at that time, on a 1 to 5 scale). A number of the Institutes
began to conduct some of their own reviews in the early 1960s in part
because of the large increase in the number of grant applications (from
about 3,000 to 8,000 annually in the late 1950s), and the nature of some
grants. Dr. Clark acknowledged the friction that sometimes resulted
between DRG and the Institutes because of this situation.

Peer review encountered somewhat of a crisis in the late 1960s and
1970s, with a preference by Congress for mission-oriented research with
very targeted goals over investigator-initiated research. The Nixon
administration's Office of Management and Budget called for the
abolition of the study section function, but the proposal was ultimately
derailed as Watergate intensified. Dr. Clark pointed out that these
efforts show how fragile the peer review system can be. Fortunately, the
Grants Peer Review Study Team, chaired by Dr. Ruth Kirschstein in the
1970s, strengthened the peer review process through (1) a mechanism for
investigator appeals,

\(2\) provision of the summary statements to applicants, (3) codified
procedures, and

\(4\) amendments to the "sunshine acts" to ensure that peer review was
consistent with legislation. Centralized review was rejected in favor of
a flexible combination of DRG and Institute review functions, as exists
today. In the 1980s, the Review Policy Committee, which had IC and DRG
members, was established and NIH Manual chapters that relate to the
specifics of how peer review is conducted were produced.

Dr. Clark then explained that her experiences as a Scientific Review
Administrator (SRA) at DRG and the National Heart, Lung, and Blood
Institute were essentially the same, as are the underlying principles
and procedures. She also noted that scrutiny of the peer review system
tends to increase as the number of applications increases and the
success rate drops, as is the case now.

***IC Review Today and Future Considerations***

Dr. Willard said that the similarities between the way reviews are done
within the ICs and CSR are more important than the differences.

[Who reviews what?]{.underline} The ICs review almost all applications
for program project (P01s) and most large multi-center clinical trial
applications. The ICs also tend to review solicited applications and
applications for career development awards, institutional training
grants, loan repayments, and conference grants, as well as R&D contract
proposals. Dr. Willard noted that review decisions boil down to three
questions: What is being proposed? Why is it important? Can this group
of investigators with this set of resources do it? The latter two
questions may lead to the desire or need for a specific IC review
committee. Issues of coherence and management come into play, which may
call for IC experience. An IC review committee might be in a better
position to evaluate the importance of looking at a particular disorder.
In the case of program projects and centers, a review panel needs to
have experienced people who know what it takes to make a successful
center or program project work. Similarly, proposed multi-center
clinical trials raise additional review issues perhaps most
appropriately considered in an IC, such as coordination among the
centers in terms of patient recruitment and retention, data collection,
and safety and ethics.

Career development grants (the K awards) may also raise IC-relevant
issues, such as how to balance the training of individual researchers
with overall needs, particularly in areas with personnel shortages. Dr.
Willard cited pediatric neurology as an example. Some Institutes review
applications in this area with a focus on the scientific subdiscipline
of the applicants, while others focus more on the career development
aspect.

In terms of initiatives, challenges for review for both CSR and the
Institutes include making sure that the reviewers understand and accept
the goals of the initiative and that they can fairly judge research that
does not test hypotheses or mechanisms. Some sets of reviewers may be
less willing to look at translational research, which they may think is
not as exciting as testing cutting-edge hypotheses or exploring
fundamental biological mechanisms.

[IC and CSR challenges]{.underline}: Dr. Willard noted some generic
review challenges faced by the ICs and CSR.

-   Trading off speed against expertise,

-   Avoiding conflicts of interest, particularly in areas with few
    researchers,

-   Putting together review committees whose members have the
    appropriate level of expertise,

-   Identifying a diverse set of reviewers,

-   Dealing with the logistics of timing and travel, especially for site
    visits and interviews.

Even though the underlying principles govern both IC and CSR reviews,
ICs vary in how they handle their reviews, such as how they use standing
committees versus special emphasis panels; their use of site visits,
face-to-face meetings, and teleconferences; the level of interaction
between the review and program staff; and the extent of streamlining
that they do.

[A future wish list:]{.underline} Dr. Willard shared his wish list for
the future: (1) more collaborative reviews, (2) better technology for
virtual meetings, (3) fast-track administrative reviews instead of
amended applications to speed up turn-around time, and (4) more
effective use of reviewer reserves who can come in on an as-needed
basis.

In closing, Dr. Willard reiterated that ICs often review applications
that have more of a specific niche in terms of their goals, needs, or
review criteria. No matter where a grant is reviewed, it will be
reviewed with fairness and expertise.

## Discussion

Dr. Pugh suggested putting Dr. Clark's timeline on a Web site as a way
to show how the system has evolved. He also reiterated the need to keep
the peer review process transparent and fair.

Dr. Ramm asked about the impact of the increased number of applications
on SRA workloads. Dr. Stanfield said that the goal is to have one SRA
responsible for about 80 R01 equivalents for each round. By this
standard, about 20 additional SRAs are needed. Dr. Ramm pointed out that
CSR sometimes reviews applications for the National Center for Research
Resources, when it does not have the necessary expertise to review
certain grant applications, such as those related to the shared
instrumentation program or biomedical technology centers.

Dr. Craig McClain praised the idea of using reviewer reserves. In
fiscally tight times, people can only spend so much time away from their
departments. Dr. Willard agreed, adding that reserve reviewers help
avoid overworking the same people, although many of the reserves are in
fact participating more frequently than in the past. Dr. Pugh praised
NIH for fighting to keep SRAs at a Ph.D. level, supporting the core
principle of competence and professionalism in the peer review process.
Dr. Sassaman expressed concern about SRA burnout in Institutes and CSR.

Dr. Robert Hammond suggested SRAs could save time for reviewers of
applications submitted in response to a Request for Applications by
conducting a conference call with reviewers before they start writing
reviews and point out the criteria they need to consider. He also noted
that, though the ICs review a smaller number of the applications than
CSR, about half of the dollars granted by NIH go to grants reviewed by
the Institutes. He said that, whether it is a CSR or an Institute
review, the purpose is to make the most informed funding decisions
soundly and fairly.

# NIH Governance Structure: Relevance to Peer Review

Dr. Stanfield stated that fulfilling the NIH mission depends on the
ability of the NIH components to develop and implement effective
extramural activities, which, in turn, require some central
coordination. The Steering Committee Dr. Zerhouni described earlier
addresses corporate issues other than setting scientific priorities,
while standing and ad hoc working groups provide input and support the
Steering Committee's functions. As discussed earlier in the meeting, the
EAWG has purview over peer review issues at NIH and oversees CSR's
budget request to the Management and Budget Working Group.

He then discussed two examples to illustrate how decisions are made on
trans-NIH issues and on funding for central services.

[Information, Planning, Analysis, and Coordination II (IMPAC
II)]{.underline}: This computer system is part of eRA, an internal NIH
computer system that processes grant applications from when they first
come into the door to when an award notice is sent. OER manages eRA, but
since it serves all NIH Institutes, it gets its policy direction and
budget from the NIH Steering Committee.

Dr. Stanfield noted that the Review Users Group is a cross-IC user group
that focuses on eRA's IMPAC II Peer Review and Internet-Assisted Review
(IAR) modules.

[CSR]{.underline}: CSR itself, as another central function of NIH, also
has a complex governance structure. Responsibility for its governance
rests primarily with the EAWG. Funding comes from a central service
budget to which the ICs contribute, with the amount decided by the NIH
Steering Committee. The EAWG makes recommendations to the Management and
Budget Working Group, which looks at all the central service budgets
and, in turn, makes a recommendation to the Steering Committee.

Because CSR and most of the 24 funding ICs perform peer review, PRAC was
established to facilitate harmonization across NIH peer review
components within the NIH governance structure. For this reason, some
members of the EAWG serve on PRAC, and it may advise the ICs, in
addition to CSR, about peer review. Similarly, if the Steering Committee
wants to see a policy enacted related to peer review, it would make a
recommendation to Dr. Ruiz Bravo, who would make sure that it conforms
to applicable Government policies and laws and then would enunciate the
policy to the ICs and CSR.

## Discussion

Dr. Martinez asked whether PRAC could also provide advice to the NIH
Director. Dr. Stanfield said that it could do this directly or ask that
Dr. Ruiz Bravo relay advice to the NIH Director.

Dr. Sassaman asked Dr. Stanfield how he thought the new governance
system was working.

Dr. Stanfield said he thought the elements were in place and with some
adjustments things should run smoothly. Dr. Ruiz Bravo added that
advisory groups are providing feedback, and internal committees within
the EAWG are helping to develop policies.

# 

# Referral Philosophy and Practice

Dr. Suzanne Fisher, Director of DRR, and Dr. Jean Noronha, Referral
Officer of the National Institute of Mental Health (NIMH), talked about
referral issues from their different perspectives.

***CSR Perspective***

Dr. Fisher said that, though she often speaks about the logistics of the
workflow involved in application receipt, she planned to focus on the
decisions behind referrals. The referral function began in about 1946,
when someone had to decide which of the then 10 study sections would
review an application. This year, DRR, as the central receipt point for
all NIH competing grant applications, will make referrals for an
expected 80,000 grant applications.

CSR makes assignments to Institutes and to review locations within CSR
based on referral guidelines, regardless of different budgets and
paylines. Dr. Fisher said that DRR serves as an honest broker,
particularly for applicants unfamiliar with the system. DRR makes
information on its practices available to all potential applicants to
ensure equitable treatment. The office also serves as a gatekeeper and
as an information source for applicants and NIH staff.

*Realities of Referral*

Dr. Fisher noted some of the complications or realities of the referral
process.

[IC organization]{.underline}: The way that ICs are organized makes
referrals complex. They have different missions, budgets, priorities,
and ways of achieving their goals. Some were organized by disease and
others by stage of life. While almost every major topic involves
overlapping or shared interests among several or more ICs, the NIH
grants and computer system requires one primary IC assignment but as
many as 23 dual assignments are possible on any given application.

[Grant mechanisms]{.underline}: The ICs use the range of different grant
mechanisms in different ways because of their budgets and missions as
well as the different visions of their Directors and staff. This creates
tension between the desire for consistency across NIH and for
flexibility to meet specific IC needs. Dr. Fisher cited the R03 small
grant as an example. By looking at the program announcements of the 20
ICs that use this mechanism, one can see that the requirements range
greatly in terms of the amounts and duration of funding, page limits for
the research plan, receipt dates, and other factors. Great variations
also exist in how the ICs use the R21 grant mechanism. Though she said
she did not think uniformity should happen for its own sake, she
questioned the degree of variation, which makes it difficult on
applicants and NIH staff.

*CSR or IC?*

Dr. Fisher next summarized how DRR decides whether an application should
go to an IC or stay in CSR for review. CSR reviews applications for
commonly used mechanisms: the R01, F32, and small business grant
mechanisms. ICs review applications for IC-specific grant mechanisms:
P01, centers, and training applications. Many assignments are based on
precedent or negotiation, which are captured in DRR's 42-page referral
guide that lists all the different grant mechanisms and where
applications are to be assigned for review. The R03 and the R21 have the
greatest variation in how different ICs handle them. The "odd ducks," so
called because they sometimes go to CSR and sometimes to an IC, include
the Ks, P01, R24, R25, and T35 applications. (S10s are odd because they
are very IC-specific, but all are reviewed in CSR.)

Dr. Fisher presented a table that depicted where applications were
reviewed over the past decade in CSR, an IC, or elsewhere in the
Department of Health and Human Services. In 2004, of a total 71,956
applications, CSR reviewed 47,397, the ICs reviewed 21,369, and other
agencies in the department reviewed 3,190. Even with steadily increasing
total numbers over the past few years, the ratio has remained about the
same. As noted earlier in the meeting, though CSR reviews a larger
overall number, the dollar amount between CSR and IC applications is
roughly equal. Various efforts to simplify the system have not worked,
such as consolidating the various career award mechanisms or dividing
applications based on whether they cover mission-related or basic
research.

*DRR Operations*

Dr. Fisher explained how DRR and its 7 full-time staff and 16 part-time
Referral Officers work. Staff must have broad scientific scope and be
able to quickly analyze information and make decisions. The full-time
staff members do what they call a "breakout" to decide if an application
will go to an IC or stay in CSR and to assign it to a Referral Officer
if it stays within CSR. If it stays within CSR and goes to an IRG, that
IRG will then decide whether a study section or special emphasis panel
will review it. Institutes have their own but similar ways to make
decisions.

Each Referral Officer is now responsible for about 1,000 applications
per round. Dr. Fisher noted that her office tries to accommodate
applicant requests to assign an application to a specific study section
or IC. When CSR cannot fulfill such a request, the applicant is notified
of the reason behind this decision. Some applications are more difficult
to assign than others because they may relate to a number of different
IRGs or ICs.

Factors that enter into each decision include (1) determining whether
the application is assignable in its present form or needs fixing (for
example, in terms of format or budget presentation); (2) the focus of
the application and the expertise needed to evaluate it; and (3) the
grant mechanism. DRR makes primary and dual assignments, and the
Institutes can add or take themselves off dual assignments when
appropriate. DRR also receives and processes applications for other
parts of the Department of Health and Human Services, particularly the
Agency for Healthcare Research and Quality and the National Institute of
Occupational Safety and Health. There has been some discussion about
taking on referrals for more parts of the Department or for the entire
Department.

*eRA*

Dr. Fisher outlined some of the changes coming with electronic research
administration (eRA), which will help with many routine functions and
perhaps afford new approaches to making application assignments.
However, complex variations in application requirements will limit the
ability of an electronic submission system to check the completeness and
accuracy of applications. She also noted the need to develop tracking
and reporting systems so that applications do not get lost in
cyberspace, as well as the need to ensure equity in the transition from
paper to electronic applications.

Dr. Fisher concluded by saying that both the good news and the bad news
is that NIH provides a wealth of funding opportunities for investigators
at many stages in their career and across the entire spectrum of
biomedical and behavioral research. The variations sometimes make it
challenging for applicants and NIH staff to select the best location for
the IC assignment and review location of an application.

***IC Referral***

Dr. Noronha talked about referrals from an IC perspective, focusing on
NIMH, where she serves as a Referral Officer. NIMH has about \$1.3
billion in extramural funding and has had about 1,500 applications per
review cycle, making it a medium- to large-sized IC.

The main job of her office is to track the applications that come to
NIMH. She assigns a program class code to each application to track it
as it goes to a branch within the Institute and to a project officer
within that branch. The Referral Office also makes sure that an
application is appropriate for NIMH review, particularly in light of the
Institute's recent reorganization. NIMH has two standing study sections;
other applications are reviewed in special emphasis panels. She, the
review branch chief, and the director of extramural activities also
assess the workload of the SRAs based on the applications that come in
and where they need to be assigned.

When it appears that NIMH is not the best place to review an
application, the Referral Office will look for a more suitable review
venue within NIH. Reasons to change an assignment include (1) a better
match with another IC based on the details of the research, (2) another
IC had given the applicant pre-submission advice, or (3) the focus of a
renewal application has changed so that it is outside the scope of NIMH
activities. The Referral Office is the central point for requesting and
keeping track of these changes. Dr. Noronha also noted that NIMH is
updating its referral guidelines, which will give CSR more up-to-date
information about how the scope of the Institute has changed. Her office
also makes sure that such items as Program Announcement number and other
data are accurate on an application.

Dr. Noronha summarized the NIMH's referral challenges: (1) the
increasing workload;

\(2\) communications with investigators to ensure that their
applications are relevant to the NIMH mission, are matched to the right
funding mechanism, and are following NIMH policies;

\(3\) communications with other ICs; and (4) the transition to
electronic submissions.

## Discussion

Dr. Pugh observed that conflicts sometimes arise between CSR and the ICs
in terms of values, processes, or practices. For example, the ICs might
have flexibility of funding as a core value. Communication helps
understand the differences, as happened with the study section
reorganization. Dr. Sassaman praised DRR's cooperation and its role as a
neutral party sensitive to concerns both inside and outside of NIH.

Dr. McClain asked about dual IC assignments. The system allows only one
IC to have the primary assignment, but Dr. Fisher said that dual
assignments may benefit the applicant. While only slightly helpful in
increasing an investigator's chances of funding, dual assignments let
Institutes know about a wider range of investigators and applications.
Dr. Matt Winkler asked how Behavioral Science Track for Rapid Transition
(B/START) reviews are expedited.

Dr. Noronha explained the process for the next review round:
applications received on

February 1 will undergo an expedited review process with a turnaround
time for funding as short as 4 months. The process involves mail ballots
from peer reviewers and production of a summary statement-like document.
The process does not, however, include council review, since the awards
are small (similar to small grants). When Dr. Martinez asked why all
grants could not be considered in this manner, she replied that it is a
complex and time-consuming process.

Dr. Hammond asked about staff coordination after referrals are made to
study sections and ICs so that the ICs have sufficient time to provide
information. Dr. Fisher said that the IMPAC II system allows an SRA to
learn about the assignment and talk to the relevant program directors
within the ICs. Dr. Ramm asked about splitting up primary and dual
assignments differently in cases of overlap because of different funding
levels. Dr. Stanfield noted that has occurred with NINDS and NIMH, when
the overlap between the two is so complete. The ICs must decide among
themselves in these circumstances.

Dr. Torok-Storb asked about when applicants request assignments based on
which Institute might have more funding. Dr. Fisher said that referrals
by her office are made without reference to IC paylines. Dr. Stanfield
added that the time lag between when an application is submitted and
when an IC makes its funding decision could mean that previous payline
information is no longer relevant. Dr. Ruiz Bravo added that some
Institutes follow paylines more rigorously than others. Dr. Pugh
suggested that NIH create a Web page for applicants that could include
information that discourages them from "chasing last year's paylines."
Dr. Stanfield agreed in the benefits of such information.

Dr. Berg commented how different ICs review applications by focusing on
the R21 mechanism. He said that NIGMS intends to use the R21 for
high-risk, short-term projects, but reviews often manifest different
interpretations of how to use it: some reviewers give these applications
low scores because they do not have preliminary data, or the
applications with preliminary data receive higher scores---but the
application is not in the spirit of the announcement. Dr. Pugh observed
that changing 20 years of looking at hypothesis-driven research is not
going to happen overnight.

# Recent and Forthcoming Changes in Peer Review

Dr. Stanfield explained that the next series of presentations look at
electronic and personnel changes in managing the peer review process.

## 

***Impact of Electronic Tools on Review Management***

CSR IT Liaison, Dr. Thomas Tatham, talked about how technology can help
NIH deal with the large increase in applications in the face of a
one-third decrease in extramural support staff and non-proportional
increase in SRAs. He said that technology is helping to relieve the
increased burden and also improve quality and efficiency. Dr. Tatham
then highlighted some new tools: application CDs, IAR, a new "person
search" capability in the Query View Report System (QVR), and
SRA-developed management tools.

[CDs]{.underline}: CSR now offers reviewers applications on CDs, rather
than boxloads of paper applications. The CDs reduce cost and workload,
provide a search function, and contain additional information relevant
to the review (such as guidelines and program announcements). Reviewers
bring laptops to meetings. Most reviewers have responded favorably,
although about 5 to 10 percent of reviewers still prefer paper, and some
do not have a laptop to bring to a panel meeting. For these cases, CSR
now has a laptop loan program. CSR can also make an alternate CD in
cases where a reviewer would have a conflict of interest if he or she
saw a competitor's application on the CD that includes all the
applications for a given round. Dr. Tatham noted that the increase in
electronically submitted applications, which are presented at a much
higher-resolution than scanned paper applications, could have an impact
on how many fit on a CD.

[IAR]{.underline}: Almost all CSR peer review meetings now use IAR,
which is a component of the eRA Commons that interfaces directly with
IMPAC II and allows reviewers to submit their critiques and preliminary
scores electronically before a meeting. After the submission deadline,
reviewers can also access each other's critiques. The SRAs find this
tool extremely useful to improve efficiency by focusing attention on the
applications most in need of discussion. It also enhances effectiveness
by allowing reviewers to review or check facts in other reviewers'
arguments before the start of the meeting. In the near term, the system
will include a meeting folder, where an SRA can post late-arriving
materials and other information. Dr. Tatham said he hopes that IAR can
eventually serve as a single point of access for all materials
associated with a review meeting and as a central service for shipping
materials to reviewers. He said such a system will work best when most
applications are submitted electronically.

["Person search]{.underline}": A new component of the QVR system is
helping in the search for appropriate study section reviewers. In
addition to the existing search mode for applications, a capability
introduced within the past 6 months allows users to do a "person search"
to identify potential reviewers. The database includes all people who
have applied for a grant and/or served as a reviewer or in another NIH
capacity within the past 12 years. Dr. Tatham noted some of the person
search limitations, which are mostly due to limitations of the IMPAC II
database itself: it cannot provide gender or ethnic information, it does
not include people who have not had NIH contact, and it cannot identify
potential reviewers from other communities. The system also does not
capture such intangible, but essential, reviewer characteristics such as
fairness, reputation, or willingness to serve. In the future, he hoped
that more comprehensive information beyond the IMPAC II database could
be included, such as information from other specialized databases.

[SRA-developed tools]{.underline}: Dr. Tatham then described some of the
tools and systems developed by SRAs to help them in the review process.
Most are implemented in Excel, Access, or Filemaker Pro and often rely
on IMPAC II data. He showed several examples, such as one tool that
helps balance reviewer workloads and another that compiles pieces of
information about an application from disparate sources onto one screen.
These innovations serve as a test bed for wider use.

[Under evaluation]{.underline}: What does the future hold? Current
technologies under evaluation include dual displays, so that
applications and databases can be viewed simultaneously; speech
recognition; wireless Internet access for meetings; and improved video
conferencing.

## Discussion

Dr. Pugh noted that journals have dealt with the issue of file size for
electronic submissions that CSR might find useful. He also asked about
the proprietary nature of the information on CDs in terms of the
potential for plagiarism. Dr. Stanfield said that, now that the initial
hesitancy about using CDs has largely been overcome, it might be time to
revisit this issue, though NIH must ultimately rely on reviewer
integrity. Dr. Ruiz Bravo said there was some discussion with the NIH
lawyers about the wording of a possible warning on a CD, but that they
could be consulted on the issue again. Dr. Pugh reiterated that
innovation may be hampered if people are inhibited from submitting
information out of fear that it would fall in the wrong hands. Dr.
Winkler suggested active policing and publicizing any examples to
discourage plagiarism. Dr. Stanfield said that an early knowledge
management pilot turned up several essentially identical applications
that manual administrative review did not uncover.

Dr. Torok-Storb asked about investigators and reviewers who may not have
access to technology. Dr. Stanfield replied this situation has been
discussed in the context of the small business community, as well as
fellowship applicants who are not physically located at the university
from which they are applying.

Dr. Tatham said that anyone with access to IMPAC II data---about 5,000
individuals, including program officers and those involved in grants
management and review---can use QVR's person search. He also noted some
IMPAC II drawbacks, such as the fact that individual abstracts cannot be
automatically extracted from applications but instead must be "copy and
pasted" into the summary statement.

***eRA Update***

Ms. Jennifer Flach, Task Manager, eRA Program, presented an eRA update
on behalf of

Dr. Israel Lederhendler, Director of OER's Office of Electronic Research
and Reports Management, which runs the eRA Program. He is currently
helping NIH coordinate its e-Gov activities. She presented a series of
diagrams that showed how applications can be submitted electronically
and what happens to them afterwards. NIH handles applications as scanned
paper or through one of six service providers, and it will soon receive
applications through Grants.gov (a Government-wide system). Her office
is also working with research institutions to enable them to submit
electronic applications directly. Many business processes are involved
in the receipt, referral, and review of NIH grant applications.

Scanned images of paper applications look fuzzy, which is a drawback. As
an alternative, the service providers submit data in XML and attach PDF
files. Grants.gov uses an electronic, Government-wide Form 424. She said
that some applicants who submit to more than one agency say they prefer
Form 424 as a more standard format. She also said that some principal
investigators and ICs have been concerned that reviewers may judge
differently applications that look different (electronic versus paper).
She said that CSR has been doing a lot of outreach to explain electronic
submissions to applicants.

Four pilots of the Electronic Competitive Grant Application Process
(eCGAP) have taken place since October 2003: over 30 research
institutions have submitted about 100 applications electronically, with
at least 45 going all the way through the review process. These
institutions are now piloting electronic resubmissions. Electronic
submission currently is available for some applications (e.g., R01s,
R03, and R21s with modular budgets) by any research institution; a Pilot
for electronic submission of full-budget applications will take place
for February/March 2005 receipt. eRA will be expanding the types of
applications that can be submitted electronically, as well as getting
feedback, in the future. She also noted that the eRA exchange could go
beyond submissions to other business processes, such as electronic
notice of grant awards. They are also looking to expand current efforts
to integrate with Grants.gov, including use of the 424 Research and
Related Forms along with capturing additional NIH-specific information.
The submittal system used with service providers validates applications
against NIH business rules during processing, which Grants.gov cannot
do.

Ms. Flach invited PRAC members to view the "Submitting Grants
Electronically" section of the eRA Web site at
http://era.nih.gov/Projectmgmt/SBIR/sbir_grants.htm for more information
and identified the service providers and research institutions that are
participating in the pilots. She concluded by posing three questions for
PRAC members:

-   What role can it play in electronic submissions?

-   How does its role relate to other NIH governance bodies?

-   What advisory role can it play in NIH's approach to Departmental and
    Federal drivers?

## Discussion

Dr. Stanfield invited PRAC members to access the NIH Commons and use its
demo facility.

Dr. McClain asked about technical problems if applications are all
submitted electronically right at the deadline. Ms. Flach explained the
process by which submission systems request a "ticket" that informs NIH
that it has an application that is ready for submittal, which helps
avoid system overload. There also is a 3-day period between submittal
and when an applicant goes to the NIH Commons to take a look at the
final application and verify it, a process that replaces the signature
on a paper application. She said there has been some discussion about
moving back the deadline for electronic submittal as an incentive for
applicants to use it.

At this point, there are no reliable estimates of how many applications
will be submitted electronically for the next review round, but Ms.
Flach said that big numbers are not expected.

Dr. Torok-Storb noted that her institution requires institutional
sign-off 2 weeks before an investigator submits an application and
wondered how the system handles such situations.

Ms. Flach said that a number of situations have been raised, such as
when an applicant cannot be available for the verification 3 days later,
and she said that the Commons Working Group is discussing how to resolve
these types of situations.

***The Promise of Knowledge Management***

Dr. Stanfield said that his presentation was "getting to science
fiction," as knowledge management (KM) depends on a more widespread
electronic submittal system. He noted that scanned applications, CDs,
and IAR represent the present in terms of paperless reviews, but that
scanning has quality and efficiency shortcomings. The ability, as of
February 1, 2005, to submit R01, R03, and R21 applications that have
modular budgets and no subcontracts opens up new possibilities.

[KM and referral]{.underline}: Electronic receipt could facilitate the
referral process, in which the computer system could suggest which study
section an application might go to for review, with Referral Officers
involved in complicated cases. In addition, initial information could be
automatically captured (instead of entered manually) into the IMPAC II
database and scanning could be eliminated. Changes like these could
compress the receipt and referral process from 10 weeks down to perhaps
a few.

[KM and review]{.underline}: KM could enable computer assignment of
applications to reviewers to review, while also considering potential
conflicts of interest and reviewers' areas of expertise, load balance,
and other factors. KM might identify gaps in reviewer expertise for
certain applications and suggest potential reviewers based on
publication histories or other information.

[Future uses]{.underline}: Other future KM applications might be used to
ensure integrity by detecting plagiarism or scientific overlap between
different applications, balance NIH's research portfolio, and help
identify emerging areas of science. CSR is now developing an interactive
area on its Web site in which applicants could enter sections of their
applications (such as the abstract or specific aims) for suggestions of
which study sections might be the best fit.

Dr. Stanfield stressed that KM will not replace SRAs, but rather provide
tools to allow them to focus on the scientific aspects of their work.

## Discussion

Dr. Torok-Storb asked whether it would be helpful if applicants
submitted an "intention to submit" about a month before the deadline to
give SRAs some idea of how many applications to expect. Dr. Stanfield
said that such notifications have not proved predictive in the past, and
SRAs might not find even accurate data useful, since they are usually
busy throughout the different stages of the review cycle. He noted that
shortening the cycle will have implications, but will also hold
opportunities.

***Division of Extramural Activities Support (DEAS)***

Dr. Anita Miller Sostek, Director of the Division of Clinical and
Population-based Studies discussed the new system of NIH staff support
that resulted from the Government-wide A-76 initiative. CSR's four
review divisions are now receiving support services through DEAS, which
is a division of OER. CSR is the largest DEAS customer.

Under the A-76 initiative, agencies examine their operations to
determine what is inherently a Government function and what might be
performed by the Government or by the private sector, whichever would
make it a Most Efficient Organization (MEO). In the competition that led
to the establishment of DEAS, NIH and one outside contractor submitted
bids to provide NIH support for its application review, grants
management, and program activities. The external bid was deemed
technically unacceptable. DEAS started operation in October 2004,
centralizing services in three "hubs." CSR gets support from Hub C. The
new system reduces personnel slots from the current 909 to 613 over 5
years, lowers the average pay grade from a GS-7 to a GS-5, and
encourages technological advances as work-saving features.

Dr. Sostek noted that workload is distributed because review, grants
management, and program activities at most ICs occur at different times
of the year. Since CSR only conducts reviews, it has a different work
cycle than a typical IC. To request help, an SRA or IRG staff member
enters a work request through a system called ECARES. At present, every
individual task is a separate request, which generates a lot of e-mail
traffic. The system was designed to keep assignments clear and to assure
that requests are promptly and properly assigned and completed.
Ultimately, ECARES data can inform workload balancing and contribute to
more efficient use of support staff. A recent Customer Feedback Program
will help determine needed fine-tuning.

Dr. Sostek said that, as of January 2005, the transfer of staff and
tasks beyond one primary organizational unit is not yet occurring, and
DEAS has vacancies at several levels, including the Hub C manager. An
NIH-wide Employee Activity Support Advisory Committee is working to
smooth out some of the problems that have arisen with this new system.
She noted that a training coordinator was recently hired and a
communication coordinator will also be hired. The Division of Quality
Assurance will start looking at whether customers are getting what they
need.

CSR has set up liaisons to DEAS to bridge the gaps between what the
organization can deliver and what needs to be done. Each IRG has an
administrative assistant on contract who is permitted to spend half-time
on "in scope" activities until DEAS staff is trained to handle them.
Each review division also has an administrative staff contact. The
division administrative assistants will work to ensure compliance with
DEAS policies and procedures, educate CSR staff as needed, and interact
with DEAS on various issues. The CSR Executive Officer, Mr. David
Whitmer, deals directly with DEAS on Center-wide issues.

## Discussion

Dr. Pugh asked how novel this model is as an alternative to outsourcing.
Dr. Stanfield noted that the A-76 process does not require outsourcing,
but rather competitive sourcing. Dr. Sostek agreed, saying that, in most
cases, the Government ends up winning the bid with the idea of
developing a more efficient organization. She said it was made very
clear that doing business as usual, even with no competing bid, would
not be considered acceptable. Dr. Sassaman noted that the process was
not only totally new to NIH but that the competition and creation of
DEAS occurred in only 6 months.

Dr. Winkler said he had heard trepidation at earlier meetings, but that
the fears did not seem realized. Dr. Sostek acknowledged the
trepidation, noting that there was concern that Federal employees would
lose their jobs. Another concern is that SRAs will end up doing more
support tasks. Dr. Stanfield said that no CSR employee lost their job
due to this reorganization.

Dr. Torok-Storb expressed concern that the new support structure might
affect the peer review system. Dr. Pugh said that it seemed to galvanize
the organization, and he praised people for coming up with a solution
rather than accepting that contract workers would carry out the support
function. Dr. Ruiz Bravo observed that the A-76 process was a real test
of NIH's 27 ICs working together. She said that many of CSR's problems
with the system are mirrored in other ICs, but that they will work
through the problems. She said it was too early to see what affect DEAS
would have on peer review. Dr. Pugh said that the information must go to
the Director and others if it appears that the system is not doing what
it was intended to do.

Dr. Ruiz Bravo said that it was initially believed that a lot of
employees would leave NIH unable or uninterested in finding a DEAS job.
That fear did not materialize. Those at a higher grade than the job they
are performing will be replaced with a person at the lower grade when
they leave. The average grade went down based on an analysis of the
tasks that DEAS employees will perform.

Dr. Torok-Storb asked whether these changes will affect retaining
quality staff. Dr. Ruiz Bravo said she hopes that employees at Grades 4,
5, and 6 would be motivated to find better jobs over time. She noted
that the human resource effort to implement these changes was "just
short of spectacular." Even those who stayed had to be hired into the
new organization. Dr. McClain commented on how few people outside of NIH
know what was going on. Dr. Stanfield said that, in this first round of
reviews, the timeliness of summary statement production was not
impacted, and perhaps this explained the lack of outside reaction. Dr.
Sassaman said there was some concern as a result of the crunch in
reviews and that she has asked for patience from her council and others.
Dr. Stanfield said that the workflow in the three functional areas will
need to be monitored, as there are perhaps not the peaks and valleys
originally envisioned in planning work flow. Dr. Ramm said that the
recent meeting of the National Center for Research Resources Advisory
Council went smoothly with DEAS support.

*The meeting was adjourned until the next morning, January 25, at 9
a.m.*

# CSR Study Section Reorganization

Dr. Donald Schneider, Director, CSR Division of Molecular and Cellular
Mechanisms, gave an update on the reorganization of study section
boundaries. He began with a brief history, explaining that study
sections were formed in response to increased workloads and on a
case-by-case basis. From 1945 to 1998, the number grew from about 10 to
more than 100. In the late 90s, the merger of the Alcohol, Drug Abuse
and Mental Health Administration into NIH meant new IRGs would be needed
to cover these new areas. Participation by NIH staff and the scientific
community contributed to the creation of six new IRGs. Around that time,
Congress called for the creation of an AIDS IRG. The scientific
community expressed concerns about the scientific boundaries of existing
study sections, and Dr. Ellie Ehrenfeld, then CSR Director, appointed a
blue-ribbon panel, the Panel on Scientific Boundaries for Review (PSBR),
to consider the need to reorganize the rest of CSR's study sections.

*Phase I*

In Phase I, PSBR laid out general principles on peer review philosophy
and developed a blueprint that called for 24 IRGs and a systemic
reorganization of those that had not been recently formed. PSBR began in
1998 and presented its work for acceptance by the CSR Advisory Committee
in January 2000. The principles articulated included the following:

-   There should be at least one home for review of all science relevant
    to health-related research, which implies that NIH could serve all
    segments of the health research community;

-   Each IRG's research topics should be cohesive or scientifically
    related, with small clusters of study sections related by science;

-   The organization should be flexible enough to adjust to rapid
    changes and provide homes for emerging fields.

PSBR also set goals for the review process: (1) set high standards, (2)
advance health-related science, (3) encourage innovation and
risk-taking, (4) show fairness and clarity, and (5) be monitored
continuously. Reorganizing the IRGs with an organ/disease focus help
cluster basic scientists, clinicians, and "those in between" working on
similar topics in order to foster further advances and encourage broader
applications. PSBR also recommended that several clusters be devoted to
areas of basic science that do not focus on a disease or organ.

Recognizing that no distribution system can achieve perfect clustering,
PSBR said that two homes for an application would be good; Dr. Schneider
cited the example of prokaryotic molecular genetics, an area which could
be covered by both the Genes, Genomes and Genetics IRG and the
Infectious Diseases and Microbiology IRG. Conversely, the panel did not
want the clusters to be organized so that they resulted in "orphan
applications" that did not really fit anyplace.

The Phase I report went beyond study section structure to say that
cultural norms needed changing, too. Dr. Schneider explained some of the
changes that the panel had in mind. Reviewers should not be "too
expert," but be active researchers able to judge the scientific merit of
broad areas. Study sections should not protect a discipline or field.
Different kinds of research must be judged on their potential to impact
biomedical and behavioral research, whether it is hypothesis-driven,
exploratory, or methods development research. Dr. Schneider said that
the panel was most concerned that the review process tended to
over-emphasize preliminary data, thus discriminating against bold new
ideas. The panel also encouraged the expansion of applicant
participation in study section referrals.

## Phase II

With these recommendations, the Phase II process began in early 2000,
focusing on one IRG or cluster at a time. First, a steering committee of
NIH program and review staff would form and invite leading scientists in
the topic area to help design the study sections. The resulting Study
Sections Boundary Teams each included about 30 participants, about 6 of
whom were NIH staff members. In intense, 3-day meetings, these teams
took application abstracts and determined where they would best be
reviewed. A set of draft guidelines resulted from the meetings and were
posted on the Web for 90 days for public comment. If necessary, changes
to the guidelines were made, and then the CSR Advisory Committee
considered the guidelines for each proposed IRG. CSR intentionally
allowed 1 year from the time of Committee approval to implementation of
the new study sections to allow time to staff up and to give the
scientific community time to learn more about where their applications
would be going.

*Assessment*

Though he said it is premature to determine the success of the
reorganization, Dr. Schneider shared some encouraging data. Ideally, a
study section should have a workload of between 60 and 80 applications
per cycle, and this seems to be the case with the new IRGs. Dr.
Schneider then looked more closely at different types of study sections:
five that are more clinical, five that deal with basic research, and
five that were less changed from the past than others. He noted that the
less-changed study sections seemed to average the most number of
applications per cycle (92), but that all are in a good range. To see
whether basic science has shifted to the disease and organ IRGs, as PSBR
had recommended, Dr. Schneider shared some data on type II competitive
renewal applications which show that some of this hoped-for shifting is
occurring. The scoring of shifted applications seems to be relatively
steady from the initial application to the renewal. He did note one
anomaly: clinical applications coming in for renewal in the disease and
organ clusters do not, on average, seem to fare as well as those in
basic science.

Dr. Schneider said that, because of the important work performed by the
IRGs in helping NIH identify the most promising research, it is
necessary to assess them and make adjustments when necessary. Planning
for the assessment is being finalized. Assessments of each IRG will be
on a 5-year cycle, starting with the neuroscience IRGs. CSR will report
the results to PRAC.

Even as the reorganization took place, changes were occurring in terms
of workload and science itself. Fixes to deal with these changes include
forming ad hoc or special emphasis panels on a temporary basis, or
involving researchers and NIH staff to form study sections for more
permanent situations. For example, the epidemiology area has seen much
growth, causing an increase from two study sections in 2001 to five in
2005. In a new paradigm, the Health of the Population (HOP) IRG formed
an Epidemiology Working Group that first met via an interactive Web site
and then face-to-face to craft an overall design for the five
epidemiology study sections.

Dr. Schneider closed his talk by summarizing its main points:

-   Multiple needs drove the need for study section reorganization,

-   Implementation of the last study sections is on schedule for
    February 2005,

-   Workloads are largely reasonable as the over-design of the study
    sections and the increased number of applications balance,

-   Early results show a shift of applications from basic to
    disease-organ IRGs, with encouraging initial scoring results,

-   Regular assessment is scheduled, and

-   A process for timely adjustments to changes by creations of new
    study sections is in place.

*Discussion*

[New process]{.underline}: Dr. Stanfield invited Dr. Robert Weller,
Chief, CSR Health of the Population IRG, to explain the process used to
create the new epidemiology study sections. Dr. Weller said review load
issues first caused an increase from two to three study sections, but
each still got about 100 applications per cycle. The working group
recommended five sections:

\(1\) epidemiology of cancer; (2) aging and neurology issues; (3)
infectious diseases, reproductive health, asthma, and pulmonary
conditions; (4) cardiovascular issues; and (5) kidney, obesity, and
diabetes conditions. Dr. Torok-Storb asked how epidemiological tools
would be handled in terms of study section assignment. Dr. Weller
replied that such applications would go to HOP's Biostatistical Methods
in the Research Design Study Section if the primary thrust of the
application is on the design or methods development.

[Community reaction]{.underline}: Dr. McClain observed that certain
areas become "hot topics" and generate increasing numbers of
applications, such as the area of obesity. He then warned against
overloading reviewers. He also expressed concern about scoring, stating
that as applications shifted, the relevance of clinical applications may
be getting lost. More generally, Dr. Sassaman asked about the reaction
of different communities to the reorganization. Dr. Schneider replied
that a handful have felt they are disadvantaged under the new system,
and their concerns are being discussed in the context of the PSBR
blueprint. Dr. Sassaman noted that getting and sharing data is very
useful in seeing whether a perception is real, and thanked CSR staff for
their efforts.

Dr. Berg observed that the reorganization coincided with the ending of
the budget doubling, and some frustrated investigators may be confused
about the cause of their difficulty in getting funded. The success rate
is being driven down because a larger number of applications are
competing for a limited amount of money. Dr. Torok-Storb agreed with the
difficulty and said that her institution, like others, is more
understanding in terms of the stability of positions if an application
scores well but is not funded.

[Workload]{.underline}: Dr. Faye Calhoun said that the number of
applications and as well as their complexity will determine what
constitutes a reasonable workload for a study section. She recommended
that NIH should not wait too long in responding when a study section has
a large number of applications per round. Dr. Stanfield said that the
SRA and IRG Chief serve as an early warning system to see if a spike in
applications represents a blip or a trend.

# Review of Special Topics: Clinical Research

Dr. Theodore Kotchen, CSR Advisor for Review of Clinical Research,
discussed his charge for the past 2 ½ years: to determine if grant
applications for clinical research are disadvantaged in the NIH peer
review process. He noted that clinical investigators frequently express
this concern, and his work has been to see if this perception is correct
or not. He also functions as a liaison to clinical societies and has
spoken at a number of their meetings on such topics as the study section
reorganization.

[Overview of clinical versus non-clinical research]{.underline}: Dr.
Kotchen explained that he defines clinical research as applications in
which the investigator checks the "human subjects" box. Looking at the
R01 applications for the May/October 2002 Councils, a total of 7,227
clinical applications received a median priority score of 254.0, and
23.9 percent of them were funded. In contrast, 10,209 non-clinical
applications received a median score of 244.0, and 28.1 percent of them
were funded. This includes applications reviewed both in CSR and IC
study sections.

[Area of clinical research]{.underline}: Working with others, he
categorized R01 clinical applications received in the October 2002
Council into seven different areas to see if one or more area seemed
particularly disadvantaged: (1) mechanisms of human disease, (2)
clinical trials and other clinical interventions, (3) development of new
technologies, (4) epidemiologic studies, (5) behavioral studies, (6)
health services research, and (7) utilization of human tissue. He
recognized that some applications could fall into more than one area and
also noted that the first two areas account for about 50 percent of all
clinical applications. These two areas---mechanisms of disease and
clinical trials and interventions---received significantly lower scores
and a lower percentage of funding than did non-clinical research. One
area (human tissue) received similar median scores and funding
percentages as non-clinical research. The other four areas were lower
than non-clinical research as well, but not to a statistically
significant degree, perhaps because the sample size was smaller.

[Changes over time]{.underline}: The next question Dr. Kotchen
investigated was whether differences in peer review outcomes for
clinical and non-clinical applications have persisted over time. To
answer this question, he compared R01 applications reviewed in 1994 by
DRG (CSR's predecessor) with those reviewed in 2004. Triaged (unscored)
applications increased from about 6 to 10 percent in 1994 to about 40
percent in 2004. The median scores in 2004 were worse, but this result
reflects the fact that triaged applications were given an arbitrary
score of 501. In both time periods, the clinical applications received
less favorable median scores than non-clinical applications.

*Why do clinical applications not fare as well?*

Dr. Kotchen shared data that address some possible explanations that
relate to the review process and to the applications themselves.

["Density" of clinical applications in study sections]{.underline}: Does
it make a difference if a study section is considering many ("high
density") or fewer (low density) clinical applications among the total
number reviewed in a cycle? In 1994, it made a difference: clinical
applications reviewed in low-density study sections received less
favorable median priority scores. However, in 2004, the density issue
did not affect median scores. A conscious decision was made to avoid
assigning clinical applications to study sections that had few other
clinical applications to consider. But in 2004, while the difference in
funding rates between clinical and non-clinical applications persisted,
the density was no longer a factor that could explain the differences.

[Funding amounts requested]{.underline}: Although study sections are not
supposed to consider the amount of funding requested in their
recommendations, clinical applications tend to be more expensive, which,
it was conjectured, may influence reviewers. In fact, the data show that
both clinical and non-clinical applications requesting more support
(\$250,000 and above) have actually fared somewhat better than others.
Dr. Kotchen said that more experienced investigators might tend to be
those who apply for the larger grants. In any event, he noted, the
amount requested does not account for the discrepancy between clinical
and non-clinical applications.

[Composition of the study section]{.underline}: The next possible
explanation examined was whether study section members with clinical
research experience evaluate R01 clinical applications differently than
those without that experience. To determine this, he and Dr. Sostek
reviewed 900 CVs of reviewers to see how many did and did not have
clinical research experience, using publication and grant funding
history as their criteria. Clinical research experience as they defined
it did not affect the overall scoring---again, those with and those
without clinical research experience scored clinical applications less
favorably than non-clinical applications in the October 2004 Council.

[Review criteria]{.underline}: While acknowledging that an explanation
may be found in the review criteria, Dr. Kotchen said that he did not
know of a way to evaluate the impact of the recent refinement of the
review criteria to better accommodate the unique contributions of
clinical applications.

[Application factors]{.underline}: Finding no factor in the review of
clinical applications that would explain the different review outcomes
of clinical versus non-clinical applications, Dr. Kotchen raised the
possibility that elements of the clinical applications themselves may
contribute to the discrepancy. For example, in looking at the May and
October 2002 Council rounds, applications involving human subjects that
raised concerns about safety or privacy received significantly less
favorable scores and were funded a lower rate than clinical applications
that did not raise safety or privacy concerns. The message then, which
is important to get out to investigators, is that clinical investigators
need to pay attention to that part of the application.

Dr. Kotchen also noted that M.D.s submitted about 26 percent of the
total applications between 1997 and 2003, but were funded at a slightly
higher percentage---about 28 percent. The year 2003 saw a higher
percentage both in terms of applications submitted by M.D.s and
applications funded, but he did not have data to show whether this was a
one-year blip or the start of a trend.

[CSR initiatives]{.underline}: Dr. Kotchen highlighted what CSR has done
or is considering doing in regards to the review of clinical
applications. CSR tracks review outcomes, by study section, for clinical
and basic applications. It developed supplementary guidelines for the
preparation of clinical applications, posted on the CSR Web site, and
created at least two new clinical study sections (clinical oncology and
clinical cardiovascular sciences). CSR is now considering strategies to
increase the pool of clinical investigators to serve on study sections,
perhaps by requiring less of a time commitment.

In addition, CSR continues to interact with scientific societies on this
and other peer review issues. Dr. Kotchen says that the message in his
presentations at meetings of different societies is that, though the
peer review system is not perfect and it must be continually monitored,
clinical investigators need to think about what they can do to improve
their competitive posture for NIH funding.

## Discussion

Dr. Pugh reiterated how important it is to disseminate this information,
noting that the perception of fairness is critical to the system. He
also asked whether the revised grant application, which asks about
impact, could be used in some way so that those in translational
medicine have a way to talk about the impact of their research on
clinical practice. He asked whether investigators proposing a clinical
trial seem to have appropriate statistical competence in terms of study
design. Related to that, he then asked whether study sections have the
expertise to evaluate not only the medical discipline, but also the
statistical design of these applications. Dr. Kotchen agreed that
statistical considerations for clinical projects were often their
Achilles heel and therefore warrant attention. Dr. Stanfield added that
committees that do not see a lot of clinical applications have the
biggest challenge in this regard. Dr. Calhoun noted that methodology in
some types of clinical research is still evolving, and this situation
could result in poorer scores. The peer review communities to review
some areas of clinical research must be built up.

Dr. Ramm asked about how those who receive K training awards fare in
receiving R01s.

Dr. Kotchen said that CSR had recently begun to examine this issue. Dr.
Ramm suggested looking at some of the older training programs, where
there would be data over the years.

Dr. Martinez suggested communicating the message that M.D.s have a
higher success rate than Ph.D.s. Dr. Torok-Storb added a caveat that an
M.D. is not necessarily synonymous with clinical research, and a Ph.D.
is not necessarily synonymous with basic research.

In closing this session, Dr. Ruiz Bravo introduced Dr. Sally Rockey,
OER's new Deputy Director, who came to NIH from the U.S. Department of
Agriculture.

# Identification of Trans-NIH Issues in Need of Discussion at Future Meetings and Priority Setting

Dr. Stanfield initiated a discussion with PRAC members about the
Committee's priorities to help develop the agenda for future meetings.
He suggested possible topics: the use of new technologies, the balance
between the need for and the cost of change, ways to encourage
appropriate researchers to serve on NIH review panels, the locus of
review of an application (CSR or IC), and resource management to best
handle increasing application numbers in the face of flat or declining
resources.

[Core values]{.underline}: Dr. Pugh suggested future discussions on the
main principles or core values of peer review as a way of framing PRAC
discussions on current practices, challenges, and the future of NIH peer
review. As changes to the peer review process are made, he noted, they
must be assessed against whether or not they are serving the core
values. The PSBR process could serve as a good model as it effectively
represented the interests of all stakeholders. Dr. Stanfield said that
Dr. Willard, chair of the Review Policy Committee (RPC), would take a
draft list of values that Dr. Pugh developed as a starting point and ask
the RPC to discuss it further. The issue can be brought up at the next
PRAC meeting.

[Small Business Innovation Research (SBIR)]{.underline}: Dr. Winkler
asked whether there were any issues specific to the SBIR community that
need addressing. Dr. Ruiz Bravo said that, while she was not aware of
any outstanding issues, her office would ask and report back to PRAC
members if need be. Dr. Stanfield said that the business and academic
research communities are not as separate as in the past. While this is a
good thing, he said it may have an impact on peer review that needs to
be considered.

Dr. McClain said that getting the appropriate expertise in a study
section could be an issue, given the breadth of the science considered.
He noted that he sits on an SBIR review group as a gastroenterologist,
and the person next to him may be an engineer. Dr. Stanfield noted that
it is good that SBIR review panels are not 4-year chartered committees,
so they have more flexibility to recruit members on an ad hoc basis to
review the varying range of applications that are submitted each round.
SBIR panels, however, do not have the quality control feedback loop that
chartered committees have.

[Resources]{.underline}: Dr. Sassaman expressed concern about resource
management and different ways of doing reviews. Dr. Stanfield said that
a look at the cost of reviewing applications within NIH, as well as in
other parts of the Government and entities such as the Howard Hughes
Medical Institute, would be useful.

[Reviewer behavior]{.underline}: Dr. Sassaman brought up the topic of
bad reviewer behavior and how to deal with these situations as a topic
for future discussion. Dr. Pugh stressed that a core value of the peer
review system is fairness. The system is very vulnerable---a single
instance flagged by the media could have devastating effects. Being
forearmed about potential conflicts of interest is important. Assessing
conflicts of interest is difficult, because sometimes the person with
the best scientific competence to review an application happens to be a
competitor. Dr. Berg noted that the more complicated mechanisms
exacerbate the issue; as an example, he cited a grant application
involving 18 institutions. Finding reviewers without a direct conflict
of interest is difficult.

[SRA Involvement]{.underline}: Dr. Torok-Storb said that SRAs could
provide PRAC members useful input as they evaluate NIH peer review. Dr.
Stanfield noted that SRAs contributed at previous CSR Advisory Committee
meetings, and he assumes this will happen at future PRAC meetings.

Dr. Sassaman suggested that future PRAC meetings include IC review
chiefs as well.

[Other issues]{.underline}: Dr. Calhoun suggested revisiting the
historical core value of separation between program and review, and
discussing the role of peer review to provide advice to a program.

Dr. Martinez suggested that PRAC members look at metrics or other
validation in reviewing applications. He also shared that the
Specialized Neuroscience Research Program, of which he is a member, is
helping to develop R01 investigators in minority institutions. They do
not feel they are being treated fairly, which leads to a broader issue
for PRAC members to look at---why minorities and minority schools may
not be succeeding at the same rate as non-minorities?

Dr. Torok-Storb said that other groups also feel disadvantaged in
receiving NIH funding, such as researchers who work in large animal
models.

# 

# 2005 Biennial Advisory Council Report on Inclusion of Women and Minorities in Clinical Research

Dr. Walter Schaffer, Senior Scientific Advisor for Extramural Research
at OER,--- reported on NIH compliance with Section 131 of the Public
Health Service Act, which established a requirement for NIH to include
women and minorities in clinical studies and prepare guidelines related
to their inclusion. Each IC Advisory Council must prepare a biennial
report on how it has complied with the requirement.

To ensure compliance, it is necessary to train staff and to have clear
guidance on policies and procedures to brief reviewers. In the next few
weeks, Dr. Schaffer will report to PRAC about these activities. He
invited PRAC members to contact him with any concerns and noted that the
information goes to the Office of Research on Women's Health.

# Closing Comments

In closing the meeting, Dr. Stanfield reminded PRAC members to suggest
individuals or types of expertise needed on the PRAC. He asked members
to check the dates of the next scheduled PRAC meetings, thanked them for
their participation, and wished them a safe trip home.

With no other business to address, PRAC members adjourned their meeting.

We do hereby certify that, to the best of our knowledge, the foregoing
minutes of the first meeting of PRAC are accurate and complete. The
minutes will be considered at the second meeting of the Committee, and
any corrections or comments will be made at that meeting.

> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
>
> Karl Malik, Ph.D.
>
> Executive Secretary
>
> Peer Review Advisory Committee
>
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
>
> Brent Stanfield, Ph.D.
>
> Co-Chair
>
> Peer Review Advisory Committee
>
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
>
> Jeremy Berg, M.D.
>
> Co-Chair
>
> Peer Review Advisory Committee