Center for Scientific Review Advisory Committee Meeting

National Institutes of Health

U.S. Department of Health and Human Services

May 17, 2004

The Center for Scientific Review (CSR) convened the 35th meeting of its
Advisory Committee at 8:32 a.m. on Monday, May 17, 2004, at the Bethesda
Marriott, Bethesda, Maryland. The entire meeting was held in open
session. Dr. Michael Leon presided as Chair.

## Members

Michael Leon, Ph.D., Chair David Soybel, M.D.

Edward Pugh, Jr., Ph.D. David Williams, Ph.D., M.P.H.

Lucia Rothman-Denes, Ph.D. Matt Winkler, Ph.D.

## Ad Hoc Advisors

Norman Braveman, Ph.D.

Robert Hammond, Ph.D.

Dr. Karl Malik was the Executive Secretary for the meeting.

**Welcome and Opening Remarks**

Dr. Leon welcomed members and participants to the CSR Advisory Committee
(CSRAC) meeting. He then asked members to consider the minutes from
CSRAC\'s January meeting. After they were approved, Dr. Leon asked CSR's
Acting Director, Dr. Brent Stanfield, to present his update.

## CSR Update

*New* *Outreach* *Efforts*

Dr. Stanfield described how CSR developed a new logo and a new exhibit
booth to enhance its outreach efforts. The exhibit will be taken to a
number of major scientific meetings where there will be CSR personnel to
staff it appropriately. He then noted that CSR recently published the
first CSR Annual Report For Fiscal Year 2003, and he encouraged CSRAC
members to offer suggestions for its improvement.

*Personnel Changes*

Over the previous 3 months, CSR hired 10 new Scientific Review
Administrators (SRAs) while only 2 SRAs left CSR. CSR now has 189 SRAs,
including 22 Integrated Review Group (IRG) Chiefs.

[CSR]{.underline} [Review Internship Program]{.underline}: CSR currently
has 17 review interns, and it plans to offer positions to 7 candidates
in July 2004. To date, a total of 26 scientists have participated in
this program during the 3.5 years it has been in place.

[CSR Division of Receipt and Referral]{.underline}: Dr. Kalman Salata
was recently named Deputy Director of this division, and Dr. Anne Clark
was named Associate Director. She will also serve as CSR\'s Research
Integrity Officer. In addition, CSR has two new Referral Officers: Dr.
Weija Ni and

Dr. Charles Rafferty.

[Promotions]{.underline}: Dr. Richard Panniers was named Chief of the
CSR Genetic Sciences IRG.

Dr. Dharam Dhindsa was made Deputy Chief of the Surgical Sciences,
Biomedical Imaging, and Bioengineering IRG; and Dr. Ann Hardy was named
Deputy Chief of the Health of the Population IRG.

[New Coordinators]{.underline}: Six coordinating functions are now being
performed by SRAs:

\(1\) IT Coordinator (4) IMPACT II Coordinator

Dr. Thomas Tatham Dr. Everett Sinnett

CSR Biobehavioral Mechanisms of CSR Respiratory Integrative Biology and

Emotion, Stress and Health Study Section Translational Research Study
Section

\(2\) Internship Program Coordinator (5) IT Training Coordinator

Dr. Noni Byrnes Dr. George Chacko

CSR Bioanalytical Engineering and CSR Biodata Management and Analysis

Chemistry Study Section Study Section

\(3\) Review Training Coordinator (6) Review Policy Web Coordinator

Dr. Robert Freund Dr. Jean Sipe

> CSR Virology B Study Section CSR Musculoskeletal Tissue Engineering

Study Section

*Internet-Assisted Review (IAR)*

Dr. Stanfield noted that all CSR study sections will be using IAR in the
upcoming June 2004 review round. This system allows reviewers to submit
critiques and preliminary scores to a secure Web site. After all
critiques are posted, reviewers and their SRAs can then view the
critiques to better prepare for their meetings. He explained that the
IAR is now integrated with the IMPACT II database, which the National
Institutes of Health (NIH) uses to process grants. As a result, certain
data need only be entered once, and the IAR can quickly create
preliminary summary statements.

*Workload Increases*

Dr. Stanfield said that CSR\'s workload continues to increase. Between
fiscal year (FY) 1998 and FY 2003, the number of applications reviewed
by CSR increased 55 percent. In FY 2002 and FY 2003, the number of
applications reviewed by CSR increased 22 and 23 percent respectively.
Currently, CSR appears to be facing a 10 percent increase. He explained
that CSR\'s needs have outpaced its resources. On average, an SRA can be
expected to handle about 80 R01 grant applications per review round.
Using this metric, CSR should have 225 SRAs instead of 189. New hiring
ceilings as well as tighter budgets have led to some uncertainty over
how CSR will be able to keep pace with its growing workload. Dr.
Stanfield has been working with NIH staff to address the situation. He
added that NIH has received 748 applications in response to different
requests for applications (RFAs) related to the NIH Roadmap initiative.
CSR is reviewing 251 or 36 percent of these applications.

# Other Issues

[Congressional Reauthorization]{.underline}: Dr. Stanfield noted that
Congress has two separate processes for appropriating funds and
authorizing appropriations. NIH has not been reauthorized in over

10 years, and the Chair of the House Energy and Commerce Committee,
Representative Joe Barton, has taken an interest in doing this in FY
2004. An authorization committee has met with a number of NIH Institute
and Center (IC) Directors and their staff.

[Extramural Activities Support]{.underline}: Mr. David Whitmer, CSR
Executive Officer, is now heading the NIH Extramural Activities Support
transition team, which is seeking to implement a new "most efficient
organization" that will provide services in support of the NIH grants
program. A new Division of Extramural Activities Support has been
created, and a director, managers and supervisory staff have been hired.
The target date for implementation is July 28, 2004.

# CSRAC Members

Dr. Stanfield announced the attendance of a new ad hoc CSRAC member: Dr.
Norman Braveman, who is the Assistant to the Director at the National
Institute of Dental and Craniofacial Research. He was recruited as part
of a new practice to provide CSRAC members greater insights into the NIH
program and grants management. Dr. Stanfield then noted that

Drs. Lucia Rothman-Denes and Susan Berget were completing their terms of
service to CSRAC. He personally thanked Dr. Rothman-Denes for her valued
contributions over the years, and similarly expressed his appreciation
of Dr. Berget, who was unable to attend this meeting.

# Search for a New CSR Director

Dr. Edward Pugh updated CSRAC members on the search for a new CSR
Director. He explained that he is serving on the search committee, which
is chaired by Dr. Paul Sieving, Director of the National Eye Institute.
This committee also includes representatives of other ICs and the
external scientific community. Forty-three applications were received
and reviewed. Eight viable candidates were identified for further
consideration. Dr. Pugh said that interviews should be completed within
the next month, and a new Director may be appointed by July 1, 2004.

**Shortening the Review Cycle**

Dr. Elliot Postow, Director, CSR Division of Biologic Basis of Disease,
noted that shortening the period between application submission and
grant award has been a goal of both present and former NIH Directors. He
then presented timelines for the current application receipt, referral,
and review processes. Each of the three current review cycles run for
about 6 months―from the receipt of the application to the release of the
review summary statement.

The receipt and referral process takes about two months. Staff (1) open
about 500 boxes a day; (2) affix control numbers, check for reference
letters when required, and disassemble appendices; (3) assign
applications to an IRG and an IC, check claimed application status and
check the possible funding mechanism; and (4) check data quality, affix
application number label, and forward to an IRG and the scanning room.
Referral Officers perform many verifications and checks in addition to
referring applications to the appropriate IRG and IC.

In focusing on the review process, Dr. Postow listed the multiple tasks
performed by the SRAs: (1) reviewing how well applications are matched
to their review venue, (2) assessing applications for administrative
problems, (3) recruiting temporary reviewers, (4) assigning applications
to reviewers, (5) overseeing logistical arrangements, (6) communicating
with reviewers,

\(7\) conducting study section meetings, (8) checking scores and codes,
(9) preparing summary statements, (10) organizing and conducting special
emphasis panels (SEPs) to review applications that require separate
reviews, (11) addressing appeals and attending IC Council meetings, (12)
developing nomination slates for their study sections, and (13)
maintaining currency with the development of science in their fields.

Before discussing how the review cycle might be shortened, Dr. Postow
noted two conditions previously suggested by CSRAC members: (1)
applicants should be able to submit amended applications for the next
receipt date, and (2) reviewers must continue to have 6 weeks to prepare
their critiques. He added a third condition: one council meeting will
need to be in September so that NIH can fund grants before the fiscal
year closes.

# Requirements and Challenges 

Dr. Postow explained how difficult it would be to shorten the cycle and
meet these conditions, noting that CSR would have to create a cycle that
is shorter than the current accelerated schedule for reviewing AIDS
applications. These applications have a receipt date that is 3 months
after the receipt date for regular applications, but reviewers typically
have only 4 weeks to prepare their critiques, and summary statements are
not regularly released 4 weeks prior to the next receipt date. A review
cycle that met the conditions presented would require CSR to move
applications immediately from receipt to the IRGs. SRAs would only have
3 weeks to perform their administrative reviews, consider the
appropriateness of the study section assignments, and begin to recruit
temporary reviewers for each batch of applications that come to them
(regular applications, amended applications, etc.). All study section
meetings would need to be held within a 2-week period, and summary
statement production would also need to be limited to a

2-week period.

In discussing how the review cycle could be compressed in this manner,
Dr. Postow said it would be impossible unless CSR received more than 90
percent of its applications electronically. He then provided a long list
of changes that would need to occur: (1) all grant mechanisms reviewed
principally by CSR would have the same receipt date; (2) all grant
mechanisms principally reviewed by the ICs would be submitted earlier;
(3) deadlines would be the day received not the day postmarked, with all
new applications due the same day and all amended applications due 4
weeks later; (4) late applications, corrections, late material, and
updates would not be permitted; (5) applications deficient in any
significant way would be returned to the applicant for correction and
submission at the next receipt date; (6) applications would be assigned
directly to study sections and ICs by a knowledge management system; (7)
reviewer critiques would not be edited by the SRAs; (8) summaries of
discussion would only be written by SRAs if they saw a compelling need
for them; and (9) all summary statements would be released at the same
time.

# Pros and Cons

Dr. Postow continued his analysis by discussing how such a shortened
review cycle might impact different stakeholders:

[Applicants]{.underline} would receive awards sooner after submitting
their applications and, in many cases, be able to submit an amended
application for the next receipt date. They would not, however, be able
to submit corrections, updates or additional material, and they would
receive less useful summary statements. In addition, their sponsored
research office staff would have less time to complete their tasks.

[NIH]{.underline} would not have to deal with late applications,
corrections, late material or updates, and the ICs would receive all
summary statements at the same time. NIH might, however, have difficulty
securing enough meeting rooms and providing support services during peak
periods of activity. NIH would face increased limitations on its ability
to: (1) set receipt dates for RFAs issued by the ICs, (2) be "user
friendly," and (3) provide advance notice of SEP meetings in the
*Federal Register*. In addition, NIH would also likely face added
difficulties resulting from a decreased quality of summary statements
and an increase in the number of "less than optimal" reviewers. NIH also
would have to increase its staff when it is already short staffed and
facing limited budgets.

[SRAs]{.underline} would be freed from dealing with corrections,
updates, etc. and would not have to spend as much time preparing summary
statements. They would, however, have less time to recruit temporary
reviewers and complete summary statements. And they would find it more
difficult to schedule SEPs. In addition, the SRAs would likely have a
reduced feeling of professional accomplishment and find greater stress
in the rigid schedule that would not easily accommodate different work
styles. Minor emergencies could become catastrophes. CSR would become
less of a family friendly workplace, and the SRAs would have reduced
opportunities to attend scientific meetings or participate in NIH
committees or training sessions.

[IC Program Officers]{.underline} would be able to make awards sooner
after receipt of the applications, but they would receive all of their
summary statements in a short period of time. They would also likely
face pressure from applicants to reveal information about study section
discussions before the release of summary statements, and they would
find it more difficult to attend all study sections reviewing
applications in their portfolio.

Dr. Postow concluded by saying that compressing the receipt, referral,
and review process as proposed could only be done if at least 90 percent
of the applications were submitted in electronic format and if a
knowledge base management system were used to refer applications to the
IRG and IC. The costs throughout the peer review system, however, would
be great. Nonetheless, a more modest reduction in the period between
receipt and award could be possible.

# Discussion of the Challenges

Dr. Robert Hammond said that the challenge is not just to shorten the
review cycle but to maintain the quality of CSR reviews. He suggested
that CSR examine staffing patterns and consider having midlevel
employees assist SRAs in certain tasks, such as identifying temporary
reviewers and identifying potential conflicts of interests. Dr. Hammond
continued by saying he was concerned about doing away with the resume
and summary of discussion. Some applicants may need a good review of
discussion to revise their applications appropriately.

Dr. Leon said he was concerned about the quality of digital images
resulting from scanned applications, and suggested the problem could be
solved if applicants were asked to submit their applications and
appendices in pdf format. He also said he wondered about the validity of
the claim often made that unedited summary statements were unacceptable.
Dr. Leon proposed pilot studies to assess the value of edited vs.
nonedited summary statements and the usefulness of having applicants
submit pdf files. He also proposed pilot testing knowledge management
systems for referring applications and identifying ad hoc reviewers. He
concluded his remarks by noting how difficult it was to consider
shortening the review cycle when CSR is struggling to deal with dramatic
increases in its workload. He recommended that CSR advance the use of
practices and technologies which could ease the current burden and
shorten the review process in the future.

Dr. David Soybel said that reviewers should have an opportunity after
their meetings to revise their summary statements, and they should be
released as soon as possible after the meetings. However, he said that
he thought it important to edit pejorative comments or other problematic
things in summary statements that could spark an increase in appeals. He
also echoed

earlier comments about the importance of maintaining the quality of
reviews as the number of submissions increases.

Dr. Pugh said that it was clear that shortening the review process could
only be possible with electronic submissions. Dr. Stanfield noted that
NIH has been conducting a number of pilot tests to this end, and NIH
intends to allow anyone with a simple R01 application to submit it
electronically for the February 2005 receipt date.

Dr. Braveman said that he was once an SRA and suggested that it might be
difficult to recruit and retain SRAs if some of the proposed changes
were enacted. SRAs use their scientific expertise to produce the final
summary statements, feeling they are providing a valuable service. He
encouraged CSR to look at other timesaving activities. Dr. Rothman-Denes
suggested that time could be saved if reviewers addressed inflammatory
comments they found in each others\' critiques after their meetings. Dr.
Anita Sostek Miller, Director, CSR Division of Clinical and
Population-Based Studies, explained that SRAs do this to avoid pitting
reviewers against each other. She added that it was also critical to
edit reviewer critiques for nonstandard English to reduce the chances
that applicants will be able to identify the reviewer.

Dr. Stanfield said that these discussions were useful, but he emphasized
that changing the review cycle would affect NIH as well as the
extramural community and that CSR could not act unilaterally. The NIH
Office of Extramural Research (OER) will have the ultimate
responsibility for setting new policies. He also said that using
knowledge management software to assign applications would require
modifying the grant management computer system, which is overseen by
OER. Dr. Stanfield explained that CSR is convening an internal committee
to consider adding some knowledge management software to its Web site to
help principal investigators identify study sections for their
applications. Dr. Don Schneider, Director, CSR Division of Molecular and
Cellular Mechanisms, said that there was work to do, since the current
knowledge management system being studied is only right about 30 percent
of the time. Dr. Stanfield added that such systems are being considered
as tools to help referral officers and SRAs perform their jobs more
efficiently and effectively.

# 

**OER Priorities and Plans**

The Deputy Director for Extramural Research, Dr. Norka Ruiz Bravo,
provided an overview of OER. She explained that OER is the nexus for the
NIH extramural research program because it coordinates the development
of policies and the management of operations related to this program.
One of the biggest challenges for OER is balancing its efforts in
regards to its many stakeholders: the 27 ICs, other agencies, the
extramural community, interest groups, Congress, the White House, the
Department of Health and Human Services (DHHS), et al. Dr. Ruiz Bravo
continued by discussing five specific areas of concern.

1.  *The NIH Budget―The Doubling Is Over*

Congress has completed the doubling of the NIH budget. Dr. Ruiz Bravo
explained that NIH received a 3.0 percent increase in FY 2003, and the
proposed increase for FY 2005 is 2.6 percent. She added, however, that
the NIH budget represents 40 percent of the DHHS discretionary funds,
and NIH expects to receive 89 percent of the increase in these funds in
FY 2005. NIH will fund approximately 10,135 new and competing research
project grants (RPGs) in FY 2004 and approximately 10,393 RPGs in FY
2005. The success rate for grant applications is estimated to be the
same for FY 2004 and 2005: 27 percent, which is just slightly less than
the 30-31 percent success rate for FY 2003.

*Evolving Public Health Challenges*

NIH is increasing its focus on five evolving public health challenges:
(1) the growing importance of chronic vs. acute health conditions; (2)
the aging population; (3) the health disparities in the United States;
(4) the emerging diseases, such as SARS; and (5) the need for
biodefense. The new NIH Roadmap initiative will seek to address many of
these challenges, focusing on three themes―New Pathways to Discovery;
Research Teams of the Future; and Reengineering the Clinical Research
Enterprise. The Roadmap initiative presents its own challenges to NIH.
It will be a challenge to fund the Roadmap appropriately, but it will
only represent a small .9 percent of the NIH budget from

FY 2004 to FY 2009. Dr. Ruiz Bravo added that funding for new
investigators also represented a challenge, and she wants to form a
group to address their needs so that NIH research can continue to
advance in the future.

2.  *Evolving Research Approaches*

NIH also is working to adjust to evolving research approaches: (1) the
emergence of more research on complex systems vs. single molecules, (2)
the growing emphasis on multiple vs. single investigators to conduct
this research, and (3) policy and operational changes at NIH and the
institutions it funds. Dr. Ruiz Bravo cited the policy and operations
challenges related to the Roadmap initiative, which has called for
innovative ways for soliciting and reviewing applications. The new
nanomedicine initiative is also calling for flexible research funding
authority that must still involve peer reviews.

3.  *Evolving Research Workforce*

Three evolving factors were cited as affecting the development of the
research workforce:

\(1\) evolving demographics―an increase in the average age at which
researchers receive their first faculty appointments at U.S. medical
schools has increased the need to do more to get these researchers into
the system earlier; (2) evolving needs for expertise―an expansion of
multidisciplinary research has increased needs for
quantitatively-trained biologists, physicists, chemists, and engineers;
(3) evolving needs for benefits and status for graduate and postdoctoral
researchers―uncompetitive benefits and low status makes life difficult
for these individuals, and more needs to be done to raise their benefits
and status. To address these needs, OER is working with the NIH Budget
Office, the National Postdoctoral Association, postdoctoral offices at
research institutions, and the Federation of American Societies for
Experimental Biology.

4.  *Evolving Policy and Operational Challenges*

In addition to addressing key scientific challenges, NIH needs to
develop new policies and review, revise, and update existing ones. In
addition, NIH needs to achieve operational efficiencies. Dr. Ruiz Bravo
described how, to this end, OER has become involved in the Government's
competitive sourcing initiative (A-76), and OER staff will soon expand
as it implements "the most efficient organization" for providing grant
management support for the ICs. OER also is involved in an initiative
for administrative restructuring and consolidation.

In addition, OER is working to advance electronic research
administration (eRA). Major goals of this effort include: implementing
paperless grant processing, reducing time from submission to award, and
developing knowledge management tools for referral and assignment of
applications to reviewers and study sections and for coding and
analyzing information for scientific decision making. Dr. Ruiz Bravo
added that OER will seek to make these tools available to interested
parties outside NIH with appropriate privacy and confidentiality
constraints. The current eRA computer system (NIH Commons) tracks the
status of summary statements and scores and allows the submission of
financial status reports from grantees. Ninety-seven percent of the NIH
grantee institutions are registered Commons users, but only a limited
number of their principal investigators use the system. OER is working
to enable this system to accept electronic submission of modular R01
grant applications by February 2005. Three modes of submission will be
developed. Applicants may download, complete and submit applications via
Grants.gov, or a service provider funded via an NIH small business
grant, or a submission system developed by their institution.

After completing her presentation, Dr. Ruiz Bravo encouraged CSRAC
members to ask questions.

Dr. Pugh focused on OER efforts to increase postdoctoral salaries,
saying that it would be important to coordinate efforts with other
agencies, since many universities tend to use NIH pay scales for
postdoctoral students in non-NIH fields. Increases could have a downward
impact on these fields. Dr. Ruiz Bravo said that OER has been discussing
this issue with the National Science and Technology Council within the
White House. Dr. Soybel noted the current fluctuations in funding and
other factors and the fact that large numbers of well-trained and
experienced researchers are finding it increasingly difficult to stay
funded. He asked if this was considered a problem. Dr. Ruiz Bravo
explained that scientific research was a "very Darwinian" enterprise and
there will always be a limited number of academic positions. She
continued by saying that there are a number of alternative career paths
for these individuals and more could be done to inform postdoctoral
researchers about these opportunities. Dr. Soybel suggested that status
could be given to previously successful researchers who later serve as
co-investigators to help others advance their research projects.

Dr. Leon said he was concerned about communication between OER and CSR
and asked

Dr. Ruiz Bravo if OER could attend CSRAC meetings on an ongoing basis.
She said that this was not a problem and noted that she and Dr.
Stanfield meet together weekly with the Extramural Program Management
Committee.

**Clinical Research Community Outreach Activities**

Dr. Theodore Kotchen noted that he has been CSR's Advisor for the Review
of Clinical Research for a year and a half. During this period, he has
worked to examine the practices for reviewing clinical research grant
applications and to serve as a liaison with the external research
community.

He then described how he worked with CSR staff to compare the review
outcomes for clinical vs. non-clinical research applications during two
review rounds in FY 2002. In analyzing R01 grant applications, he found
that those involving human subjects received less favorable scores and
were less likely to be funded than those that did not involve human
subjects. Though the differences were modest, they were statistically
significant. When subgroups of clinical research applications were
analyzed, those devoted to the mechanisms of disease and to clinical
trials received significantly less favorable scores and were less likely
to be funded. However, no differences were observed between the scoring
of applications submitted by M.Ds. vs. non-M.Ds. or for M.Ds. vs.
M.D./Ph.Ds. The results of these studies were published in *JAMA*
earlier this year.

Clinical research applications that raised human subjects concerns
during review had less favorable outcomes than those that did not raise
such concerns. Dr. Kotchen said that the reasons for this difference
were unclear. It is possible that human subjects research is
scientifically weaker or more difficult to conduct, or applicants for
this research are less skilled in preparing grant applications. He
explained he reviewed a large number of clinical research applications,
and it was apparent that inadequate grant preparation is an important
factor.

Dr. Kotchen said that it is also possible that the review criteria may
be inappropriate for assessing clinical research.

The "density" of clinical research applications reviewed in individual
study sections has been cited as another possible factor. CSR has worked
to cluster clinical research application in its newly reorganized study
sections. He noted, however, that recent data collected did not show a
relationship between the density of clinical applications and their
review scores.

Two other factors cited are the greater cost of clinical research and
the composition of the review group. Reviewers may be hesitant to give
more favorable scores for more expensive research, and the appropriate
peers for clinical research applications may not be serving on study
sections. Dr. Kotchen explained that the data analyzed to date shows
that applications for more expensive research tend to receive more
favorable scores, possibly due to the fact that more senior
investigators submit more costly applications. CSR is currently
examining data related to the composition of study sections and review
outcome.

*Outreach Activities*

Dr. Kotchen then focused on his role as liaison to the external
community. He has met with various groups of scientists to discuss CSR's
reorganization and its efforts to address concerns about the review of
clinical research applications: (1) leadership of the Endocrine Sister
Societies; (2) the Mayo Clinic; (3) the K30 Program Directors
Association; (4) the National Association of Pediatric,
Gastroenterology/Hepatology and Nutrition; (4) Emergency Medicine MDs;
and (5) the Academic Health Centers Clinical Research Forum. He said
that he has plans to speak to members of the American GI Association and
to faculty at the University of Arkansas and the University of
Pennsylvania.

Through these meetings, Dr. Kotchen has observed that many scientists in
the community are surprised to learn that clinical research applications
fare better than they imagined when reviewed. He said that the clinical
research community tends to think that not enough of their peers serve
on study sections. Dr. Kotchen has encouraged clinical researchers to
submit lists of those in their community who could serve on CSR study
sections. He continued by noting community concerns about the time
commitment for study section service, and he suggested that flexible
terms of service might help address this problem. Members of the
clinical research community have varying opinions on the desirability of
integrated vs. dedicated clinical study sections. A number of members
from the community have expressed concerns about the practice of
factoring safety and privacy concerns into the scoring process.

Dr. Kotchen then listed current CSR efforts related to clinical
research: (1) supplementary guidelines for preparing clinical
applications have been developed; (2) review outcomes for clinical and
basic applications are being tracked and data is being supplied to study
section members; (3) trends in how reviewers with and without clinical
research experiences score clinical research applications are being
studied; (4) strategies for increasing the pool of clinical research
reviewers are being developed; (5) a collaboration with the National
Center for Research Resources is being considered to reach inexperienced
clinical investigators; and

\(6\) interactions with clinical societies are ongoing.

Dr. Kotchen concluded his presentation by encouraging CSRAC members to
share their thoughts on these issues.

Dr. Soybel said that the clinical researchers at his institution are not
faring as well in reviews as the data presented suggests, and he
encouraged NIH to develop a prospective database to collect better data
for evaluating clinical research reviews. He continued by saying that he
was interested in the clinical research guidelines that were developed.
Dr. Soybel then suggested that CSR conduct mock reviews to assess how
well reviewers with different backgrounds review clinical research
applications. The resulting data could be useful in training reviewers.

Dr. Kotchen said despite the limitations of currently available data
defining and identifying clinical applications, consistent and
significant trends were observed.

Dr. Pugh emphasized how important it was that Dr. Kotchen\'s efforts
have opened a dialogue with a community that had become increasingly
disenfranchised. Dr. Pugh then asked how the NIH Roadmap initiative may
help advance the clinical research enterprise. He added that addressing
the outstanding concerns was going to be a big job and additional
coordination and effort is needed. Dr. David Williams agreed on the
importance of opening a dialogue with the community. He also agreed that
the data presented was imperfect, but he said that it was suggestive and
could be useful in refining research into the reviews of clinical
research applications. Dr. Williams then asked Dr. Kotchen to comment on
the *JAMA* editorial that emphasized the role market forces play in the
strength of the clinical research enterprise.

Dr. Kotchen said that there were indeed many stakeholders and larger
issues involved, explaining that his focus has been limited to the
review of clinical research applications and what may be done to fine
tune the review process.

Dr. Braveman said that he thought it might be worthwhile to develop a
classification scheme for clinical research to improve assessment of
clinical applications, and he endorsed efforts to assess the positive
and negative aspects of these applications and develop guidelines for
clinical researchers.

**NIH Director\'s Pioneer Award Program**

Dr. Ellie Ehrenfeld, CSR\'s former Director, provided an update on the
NIH Director\'s Award Program. She noted how this program grew out of
efforts initiated by CSRAC members, who voiced concerns by the research
community that the NIH peer review process was too risk averse and not
effective enough in identifying truly innovative, high-impact research
proposals that may lead to conceptual and technological breakthroughs.
Dr. Elias Zerhouni, NIH Director, heard these same concerns when he
sought advice from the research community, and he asked

Dr. Ehrenfeld and Dr. Stephen Straus, Director of the National Center
for Complementary and Alternative Medicine to spearhead the development
of programs within his Roadmap initiative to advance high-risk research.
A group of outside experts subsequently proposed three programs: (1) the
Exceptional Projects Program to fund individual projects with
exceptional promise;

\(2\) the Grand Challenges Program to fund broad, complex, and costly
research programs of interest to multiple ICs; and (3) the NIH
Director\'s Pioneer Award to fund researchers with exceptional promise.
Dr. Ehrenfeld noted that most of the outside experts who endorsed these
programs still regarded the peer review process as an effective one that
works well the vast majority of the time.

NIH decided to begin by initiating the NIH Director\'s Pioneer Award
program in FY 2004.

Dr. Ehrenfeld said that it will (1) support individuals at all career
levels who have exceptional promise, (2) give them the freedom to set
their own research agenda, (3) enlist elite panels of outside experts to
conduct interviews and assess presentations, and (4) provide awardees
\$500,000 per year for 5 years.

*New Application and Review Processes*

Dr. Ehrenfeld then described the application and review processes. In
the first phase of the application process, nominations may be submitted
by mentors, colleagues, institutions, or by the individuals themselves.
Nomination packages include a letter and the nominee\'s curriculum vita,
each no more than two pages in length. These nominations will be
evaluated by NIH staff for eligibility. Outside experts then identify
promising candidates who will be invited to formally apply. In the
second phase of the application process, applicants will be asked to
submit a

3-5 page description of a major challenge in a research area where they
believe they can make seminal contributions. They also will be asked to
submit three letters of support and evidence of their most significant
achievement, such as an article, patent, or device. Outside experts will
assess these submissions, looking for evidence of exceptional
creativity, an interest in tackling important problems relevant to the
NIH mission, as well as evidence that the applicant has the ability to
boldly and intelligently pursue high-risk research. The most promising
candidates will be interviewed by a panel of outside experts, and a
secondary review will be conducted by the NIH Director's Advisory
Committee.

In discussing award criteria, Dr. Ehrenfeld said that (1) awardees will
be expected to commit the major portion of their efforts to their
high-risk research, (2) awards will not be made to simply support or
expand research projects already funded, (3) awards may be made to
individuals in nonbiological research areas if they are seeking to
explore research with biomedical relevance, and (4) awardees will be
required to submit annual progress reports and attend an annual meeting
with their colleagues.

NIH solicited nominations for these awards in January, and the response
was greater than expected: over 1,300 nominations were received, and the
quality of these nominations appears to be high. Dr. Ehrenfeld explained
that the initial review by outside experts will be completed shortly.
NIH expects to invite approximately 200 nominees to apply and 20-25
applicants for interviews. Awards will be announced by the end of
September 2004. Plans call for NIH to make 5-10 awards each year for the
next 5 years.

Dr. Ehrenfeld emphasized that this program was an experiment and a work
in progress, and evaluating the program will be difficult, since
significant outcomes will not likely begin to surface for about 5 years.
NIH, however, plans to conduct a blue-ribbon evaluation in 2008. She
then encouraged CSRAC members to comment and ask questions.

Dr. Leon said the fact that there were 1,300 nominations for these
awards suggests that the program would be addressing a very real need.
Dr. Matt Winkler asked if the nominees are researchers who were not
previously supported by the NIH. Dr. Ehrenfeld said that many of them
were indeed researchers who had not applied to NIH before. She added
that most of them were nonetheless successful researchers and appear to
show the kind of promise NIH wants to encourage. NIH expects to use
available data to better assess the character of the nominees and the
awardees as the program advances.

Dr. Pugh suggested that the large number of nominees may be due to the
fact that the nomination process is not very difficult. Dr. Hammond
asked how NIH might deal with a situation where it had more worthy
nominations than it could fund. Dr. Ehrenfeld said that some
applications have been referred to the ICs when they did not seem
appropriate for the Director's Awards program. She continued by saying
that it is hoped that the program will achieve enough prestige that
awardees and finalists would receive greater recognition and thus
opportunities.

Dr. Schneider asked Dr. Ehrenfeld how this program might impact the peer
review system if it is successful. She said that it might (1) help shift
the conservative culture of study sections so that they better recognize
and value innovative research applications, and (2) encourage
researchers to submit more innovative applications. Dr. Ehrenfeld said
more needed to be done to shift this culture so that the peer review
process can accommodate the rapidly changing nature of science. She
noted that there were two other proposed programs for funding high-risk
research in the NIH

Roadmap initiative. Financial and other considerations will determine
when and if these programs are implemented, but everything NIH does to
encourage applicants and reviewers to advance innovative research will
be helpful.

# 

# **Reorganization Implementation Update**

Dr. Schneider provided a brief overview of the reorganization process.
From the founding of the NIH grants program until 1998, study sections
were formed in response to workload increases on a case-by-case basis.
In 1998, the Alcohol, Drug Abuse, and Mental Health Administration was
merged into NIH, and in the following 2 years, NIH systematically
reorganized its study sections in the areas of neuroscience, AIDS, and
behavioral and social sciences. In 2000, CSRAC members endorsed the
recommendations of the Panel on Scientific Boundaries for Review (PSBR)
to systematically reorganize the remaining study sections. Steering
Committees of NIH staff for each of the 17 IRGs to be reorganized were
established to select members of Study Section Boundaries Teams that
would develop guidelines for these new IRGs and their respective study
sections. Each team included about 24 members of the external scientific
community and 6 NIH staff members. After a 90-day period for public
comment, these guidelines were presented to CSRAC members for a final
review.

In May 2002, CSRAC members endorsed the first draft guidelines, which
were for the Hematology IRG. Study Section Boundaries Teams recommended
and CSRAC members agreed that Molecular Approaches to Gene Function and
Fundamental Genetics and Population Biology be combined into a single
Genes, Genomes and Genetics IRG with six study sections all together due
to the number of applications. In January 2004, CSRAC members endorsed
the last IRG guidelines, which were for the Biological Chemistry and
Macromolecular Biophysics IRG and the Cell Biology IRG. Dr. Schneider
continued by explaining that nine new IRGs with 50 study sections have
met one to four times; three new IRGs with 18 study sections will meet
in June/July 2004; and the four remaining new IRGs are expected to hold
their first meetings in October 2004 or February 2005.

## *Implementation Issues*

[Workloads]{.underline}: Based on recommendations of previous working
group reports, the reorganization had a goal of designing study sections
that would handle 60-80 applications per round. The design teams
nonetheless over-designed their study sections, creating more than
expected.

Dr. Schneider explained that this result was fortuitous, given the
dramatic increase in applications submitted to NIH. The first meetings
of 68 study sections have averaged 71 applications. Twelve of these
study sections had fewer than 50 applications, and 15 study sections now
have more than 90 applications. The Hematology IRG has seen a 5 percent
increase in applications per cycle over its first four cycles. IRGs that
have met for three cycles have seen 5-15 percent increases per cycle:
Biology of Development and Aging (15 percent); Musculoskeletal, Oral and
Skin Sciences (5 percent); and Oncological Sciences (10 percent).

[Gaps and Extended Overlaps]{.underline}: A major gap developed in the
review of applications involving research related to DNA repair. The
study section that used to review these applications ended in October
2003, and the new Genetics Study Section that is designed to review them
will not be implemented until January 2005. In the meantime, these
applications have had to be reviewed by a special emphasis panel (SEP).
A problematic overlap has resulted for the review of applications
related to research on proteases, particularly those involved in
clotting and metastases. The study section where they have been reviewed
will not end until January 2005, and several of the protease
applications are not going to the new IRGs where they are supposed to be
reviewed: Hematology and Oncological Sciences.

[Communication and Identifying Appropriate Study Sections]{.underline}:
The ranges of research covered by the new study sections are different
from those covered by the old study sections. As a result, it is often
difficult for applicants to know where their application should go. To
address this issue, CSR has (1) posted the new IRG guidelines on its Web
site, (2) placed a search engine on its Web site to help applicants
identify appropriate study sections with keyword searches,

\(3\) initiated efforts to develop a knowledge management tool for its
Web site for matching applications to related study sections, and (4)
posted provisional rosters prior to the first meetings of new study
sections. In addition, CSR has worked to reach out to the community and
present updates on the reorganization process by holding meetings with
representatives of 24 societies and providing reorganization
presentations at 21 scientific meetings and 10 IC council meetings.

[Assessment of IRGs and Study Sections]{.underline}: CSR has previously
used working groups to review its study sections and used surveys to
assess the reorganization of the neuroscience study sections. Working
group evaluations of the new IRGs are planned, beginning with the
Hematology IRG in 2006 after that IRG is reviewing its full complement
of applications. In the meantime, CSR expects to tweak the structure of
its study sections to address emerging needs and may conduct focused
assessments to evaluate the reviews of specific kinds of applications,
such as clinical or technology research applications.

*Implementation Status Summary and Discussion*

In summarizing his comments, Dr. Schneider said that (1) the
implementation is on schedule for completion in February 2005, (2) the
new study sections are balanced in terms of accommodating CSR's
increased workload, (3) the stepwise implementations are creating
interim complexities, (4) needed communication activities are ongoing,
and (5) assessments are being planned.

Dr. Leon said he was glad that this enormous project continues to go
well and impressed that the PSBR principles seem to have been
conscientiously adhered to both in design and execution. He then asked
how CSR will know how effective the reorganization has gone in the
future. One measure proposed by Dr. Schneider was the degree to which
community members voice complaints. Dr. Braveman said that CSR should
consider assessing the success of the reorganization by considering the
input of its three main stakeholders: (1) reviewers,

\(2\) applicants, and (3) the ICs, including program and grants
management staff and IC council members. Dr. Stanfield noted that CSR
conducted a three-prong assessment when it evaluated the effectiveness
of the neuroscience IRG reorganization. Working groups evaluated the
neuroscience study sections, and surveys assessed the views of IC
program staff and applicants, including those who submitted applications
both before and after the reorganization. He said that creating new
working groups to assess the new study sections would be the first step
in assessing them, and he encouraged CSRAC members to provide
suggestions on how to improve these assessments. Dr. Stanfield added
that future assessments might examine the intersections of the different
IRGs and include more input from IC program staff.

Dr. Pugh asked what was known about how well the ICs regard the
reorganization.

Dr. Stanfield noted that IC staff was involved in identifying the
boundaries teams that developed the guidelines for the new IRGs, but
explained that the only relevant data CSR has collected was from the
satisfaction survey given to neuroscience program staff after their
study sections were reorganized. Program staff were somewhat less
satisfied than the applicants and were less likely to complete the
survey. Dr. Braveman suggested that many of the nonresponders may have
not bothered to reply because they were satisfied with the
reorganization and saw no need to provide input. Dr. Hammond said that
staff at his IC thought the reorganization process had worked well.

**Modifying Study Section Boundaries to Accommodate Workload Changes**

Dr. Sostek explained that the workload of a large number of
CSR-chartered study sections has increased to the point that some are
reviewing 120-140 applications a round. In the past, such large study
sections occurred rarely, and the procedures for portioning study
sections were fairly complex. Working groups of the various stakeholders
performed much of the work as part of their 5-year reviews of individual
IRGs. A working group included the study section chair, reviewers, IC
and CSR staff, and representatives of the relevant research communities.
IC staff and outside representatives sorted applications from three
review rounds and developed different partitioning plans, which were
discussed in face-to-face meetings. A report with recommendations was
then developed and submitted to the CSR Director.

## *Basic Principles for Modifying Study Sections*

Dr. Sostek explained that CSR needs a new more streamlined process to
modify its study sections. To this end, she summarized the basic
principles that a new process should satisfy:

\(1\) workloads need to be evaluated overtime to ensure that partition
is necessary; (2) extensive input must be sought from a broad base of
stakeholders―the study section chair and members, IC staff, and
community representatives; and (3) opportunities must be provided for
stakeholders to interact in order to inform the process. She added that
it might not be necessary for the IC staff and outside experts to
perform time-consuming application sorts.

A model for a new way to partition study sections could be the way the
Community-Level Health Promotion Study Section was recently partitioned.
Dr. Sostek explained that this study section received about 130
applications each round for the last 2 years. Based on their experience,
the SRA and IRG Chief sorted the applications to determine the numbers
in various subcategories in order to propose a plan for two balanced
study sections. The IRG Chief solicited input on ways to cluster the
areas identified. As a result of this process, a plan was developed to
divide the applications into a new study section focused on community
interventions and another focused on community influences. After
consultations with key reviewers, external advisors and IC staff, review
groups covering these areas were then piloted. Reviewers were assigned
to the two groups based on their expertise, and when reviewers could
serve on either one, CSR solicited their preferences. Members of the
design panel were encouraged to attend at least one of the study section
meetings. After the pilot, the IRG Chief again gathered opinions and
suggestions and encouraged stakeholders to participate in an electronic
discussion group. A summary of their comments was then provided to all
participants, and all suggestions were given serious consideration in
refining the guidelines for the proposed study sections.

Dr. Sostek noted that this process worked well and few problems arose.
The guidelines for these new study sections will be made available to
the public, and they have been submitted to NIH Committee Management for
approval. She noted that a face-to-face meeting of stakeholders might
have been necessary if substantial issues had arisen.

Dr. Sostek concluded her presentation by emphasizing that (1) a new
approach to partitioning study sections is necessary because this work
often cannot wait for regular working group reviews to occur, (2) the
approach she described could be adapted to different situations, and

\(3\) any approach taken must involve extensive discussion with all
stakeholders.

Dr. Hammond said that the proposed process seems to be very thoughtful
and inclusive, as well as rapid. He continued by asking how application
percentiles are calculated when a study section is divided. Dr. Sostek
said that CSR generally percentiles scores against those given at the
first meeting of a reconfigured study section when there is a shift in
30 percent or more of the applications it reviews. Dr. Rothman-Denes
asked how long the process would take. Dr. Sostek said that the process
would take about a year, but that informal discussions typically begin
when a study section is first assigned 100 or more applications.

# 

# 

# 

# 

# 

# 

# 

# 

## General Discussion

Focusing on the roles of the SRAs, Dr. Rothman-Denes said that she
thought that recruiting superb reviewers was much more important than
editing critiques and producing summary statements. Dr. Williams echoed
this sentiment, suggesting that CSR study the degree to which this work
is needed. Dr. Stanfield noted that scientific editors could be hired to
do some of this work, and Dr. Hammond said that NIDDK has used
contractors to perform this task and it has been pleased with the
practice. In addition, NIDDK has used retired SRAs to review
applications for conflicts of interests, allowing its SRAs to focus on
other issues.

Dr. Hammond then focused on the issue of speeding the review cycle,
suggesting that applicants who fall just short of the payline may
benefit more from a quick turnaround than those with streamlined
applications. He continued by proposing that reviewers be given
check-off boxes to note how well an application addressed the five
standard review criteria. Such information could be useful when no
resume is provided. Dr. Leon asked Dr. Stanfield if CSR could test the
effects of providing raw review data to applicants in lieu of polished
summary statements. Dr. Stanfield said that it would be necessary to
think through such a concept.

Dr. Rothman-Denes suggested adding a box to the 398 application form for
applicants to list collaborators, so that SRAs would not have to search
through letters of collaboration inside the grant application. Dr.
Stanfield said that revising the 398 in this way would only have value
if applicants filled in such a box appropriately.

Dr. Leon again raised the issue of having applicants submit applications
via CDs. Dr. Stanfield said that this was a proposal that would also
have to be thought through, and OER would have to be engaged. Dr. Leon
then asked for more information on using knowledge management systems.
Dr. Stanfield said that NIH has set a priority on using this technology
to code scientific projects, but no decision has been made about what
application would be next. He said that CSR is lobbying for NIH to
develop such systems for referring applications and also making review
assignments. Dr. Tom Tatham, CSR's Information Technology Coordinator,
said that a committee of SRAs, referral personnel, IC program officers,
and IT professionals has developed a series of recommendations on how to
proceed with implementing a system to help applicants self-refer their
applications to CSR study sections.

# **NIH Ethics Panel**

The Director of the NIH Ethics Office, Holli Beckerman Jaffe, J.D.,
described how concerns raised by Congress and the press about the
outside activities of NIH scientists led Dr. Zerhouni to create an NIH
Ethics Advisory Committee to review the outside activities of various
NIH scientists and provide recommendations on whether or not specific
activities should be approved. In addition, he formed a Blue Ribbon
Panel under the Advisory Council of the NIH Director. This panel met
several times, starting in January 2004. It was chaired by Mr. Norman
Augustine of Lockheed Martin Corporation and Dr. Bruce Alberts of the
National Academy of Sciences, and it produced a series of
recommendations to strengthen the NIH Ethics Program. Some of these
recommendations affirmed existing policies, some called for changes in
NIH policies, and others called for changes in either Government
regulation or statute. These recommendations were endorsed by the
Advisory Committee of the NIH Director, who says that he is committed to
implementing all the recommendations.

# Highlights of the Ethics Recommendations

-   Top administrators and those responsible for funding decisions and
    peer review should be banned from consulting for biotechnology and
    pharmaceutical companies, and NIH staff should continue to be banned
    as they are by law from participating in any matter where they have
    an outside financial interest.

-   Lower level administrators and scientists should be allowed to be
    paid consultants for academic institutions or
    biotechnology/pharmacology companies, but these individuals should
    not work as a consultant for more than 400 hours per year. In
    addition, these individuals should not be paid an amount more than
    50 percent of their NIH salary, and they should not be paid with any
    form of equity. Hour limits would not apply to writing or editing
    textbooks, or journal/review articles and would not apply to
    healthcare practitioners who would be able to receive up to 100
    percent of their NIH salary. No limits, however, should be made on
    patent or book royalties.

-   NIH should carefully track and assess all requests for approval to
    be sure that no conflicts of interest result when outside activities
    are approved.

-   Supervisors should be well trained to engage in more extensive
    dialogues with employees seeking approval for outside activities.

-   NIH scientists who are currently prevented by regulation from being
    paid to teach, speak, or write about their NIH research should be
    allowed to do so if the work has been published in the public
    domain. NIH scientists also should be able to accept awards and
    related payment for meritorious public service.

-   Top-level administrators or scientists employed as special experts
    (Title 42 employees) should be required to submit public financial
    disclosure forms like other civil servants. NIH, through DHHS,
    sought and received permission to require such employees to file the
    public report.

-   Supervisors and or subordinates should be informed on a need-to-know
    basis when an individual is reclused from participating in an NIH
    matter due to a conflict of interest with an outside activity.

Dr. Pugh said it was good for CSRAC members to learn as much as they can
about the activities of the NIH Ethics Office so that they can begin to
see how they apply to regulations related to peer review. He said that
more clarity is needed in articulating what constitutes a conflict of
interest for reviewers and staff and what actions or inactions are
forbidden. Dr. Stanfield agreed, explaining that SRA candidates have
many different ongoing activities than they used to have. Though few of
CSR's SRAs now have the kinds of consulting arrangements covered by the
Blue Ribbon Panel recommendations, the situation could change. Further
discussions are therefore needed.

Dr. Postow asked if the Panel addressed temporary Federal employees,
such as council members and reviewers. Ms. Jaffe said that these special
Government employees were not discussed and the recommendations proposed
do not apply to them. She then explained that the Blue Ribbon Panel
repeatedly said NIH should not limit outside activities so severely that
it is unable to recruit and retain the best and the brightest. Dr.
Soybel suggested that NIH should perhaps pay these individuals more so
they do not need to seek outside activities to supplement their income.
Ms. Jaffe said that this was one of the Panel's recommendations. She
also explained that a member of Congress recently agreed, noting his
colleagues used to receive honoraria for speaking engagements, but they
subsequently decided to ban the practice and give themselves pay raises.

Dr. Winkler suggested that attention should be given to how conflicts of
interests in study sections are addressed. CSR should publicize existing
regulations and make sure they are being enforced. Dr. Pugh agreed and
proposed that this topic be considered at a future CSRAC meeting.

When there were no more comments from CSRAC members, Dr. Leon adjourned
the meeting at 4:04 pm.

# 

# 

# 

-   

# 

# 

-   
-   
-   

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# We do hereby certify that, to the best of our knowledge, the foregoing minutes of the 35th meeting of CSRAC are accurate and complete. The minutes will be considered at the 36th meeting of the Advisory Committee, and any corrections or comments will be made at that meeting. 

> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
>
> Karl Malik, Ph.D.
>
> Executive Secretary
>
> Center for Scientific Review Advisory Committee
>
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
>
> Brent Stanfield, Ph.D.
>
> Acting Director
>
> Center for Scientific Review
>
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
>
> Michael Leon, Ph.D.
>
> Chair
>
> Center for Scientific Review Advisory Committee