Center for Scientific Review Advisory Committee Meeting

National Institutes of Health

U.S. Department of Health and Human Services

September 22-23, 2003

The Center for Scientific Review (CSR) convened the 33rd meeting of its
Advisory Committee at 8:30 a.m. on Monday, September 22, 2003, at the
Chevy Chase Holiday Inn in Chevy Chase, Maryland. The entire meeting was
held in open session. Dr. Michael Leon presided as Chair.

## Members

Michael Leon, Ph.D., Chair Lucia Rothman-Denes, Ph.D.

Susan Berget, Ph.D (via phone) David Soybel, M.D.

James Hildreth, Ph.D., M.D. David Williams, Ph.D., M.P.H.

Edward Pugh, Jr., Ph.D. Matt Winkler, Ph.D.

## Ad Hoc Advisor

Craig McClain, M.D.

Dr. Brent Stanfield was the Executive Secretary for the meeting.

**Welcome and Opening Remarks**

Dr. Leon welcomed members and participants to the CSR Advisory Committee
(CSRAC) meeting. It was noted that CSR moved the meeting to a hotel
because the Rockledge 2 building lost power in the wake of Hurricane
Isabel. Dr. Leon then asked the members to consider the minutes from the
May 12-13, 2003 CSRAC meeting. After the minutes were approved,

Dr. Leon asked Dr. Ellie Ehrenfeld, CSR Director, to present her update.

**CSR Update**

*Personnel Changes*

Dr. Ehrenfeld noted that she had recently decided to step down as CSR\'s
Director. She said she was proud of what CSR accomplished during her
tenure, and she thanked members of her staff and CSRAC for their
contributions. Dr. Ehrenfeld then said that Dr. Brent Stanfield, CSR\'s
Deputy Director, would soon be appointed Acting Director of CSR. She
added that CSR recently recruited a new Executive Officer, Mr. Dave
Whitmer, who will also serve as CSR\'s Director of Management Services.
He comes from the National Heart, Lung and Blood Institute, where he was
a Supervisory Management Analyst.

*Budget Developments*

The U.S. Senate passed a bill calling for the National Institutes of
Health (NIH) to receive \$27.98 billion, or about \$1 billion more than
it did in fiscal year (FY) 2002. Dr. Ehrenfeld said that the Senate
budget represents about a \$319 million increase over the budget
previously passed by the U.S. House of Representatives. The House budget
mirrors the Administration\'s proposed budget, which calls for NIH to
receive a 2.5 percent increase over FY 2003. This budget would, however,
provide a 7.2 percent increase for research grants due to the fact that
the FY 2003 NIH budget included a number of one-time funding items. Dr.
Ehrenfeld said that NIH would probably operate under a continuing
resolution well into October as Congress resolves its differences. The
Administration and the NIH Institutes and Centers (ICs) have agreed to
CSR\'s request for an extra \$3 million in FY 2004 to fund the resources
needed to accommodate CSR\'s increased workload.

*CSR Workload*

Dr. Ehrenfeld said that CSR has been receiving and reviewing record
numbers of applications. Between FY 2002 and FY 2003, the number of
applications CSR reviewed increased 23 percent. CSR is performing a more
rigorous analysis of the incoming applications to determine where they
are coming from and to see if it would be possible to predict future
trends. She said that CSR hired 58 new review staff members: 38 new
Scientific Review Administrators (SRAs),

11 interns, and 9 previously retired SRAs, who were hired on a temporary
basis.

*CSR Review Internship Program*

CSR\'s Review Internship Program has proven very useful in the
management of the increased workload at CSR. Additionally, the program
has been stimulating for both mentors and trainees. Dr. Ehrenfeld said
that CSR received 45 intern applications for its July receipt date and
20 of the applicants will be invited for an interview. CSR hopes to hire
eight of them.

*Issues Raised at the Last CSRAC Meeting*

[A-76 Initiative]{.underline}: Dr. Ehrenfeld returned to previous
discussions of the Government\'s A-76 initiative, which calls for
agencies to consider outsourcing work if it can be done more
economically by contractors and if it is not "inherently governmental."
She said that NIH recently held a competitive bidding process for grant
administration services, including those provided by CSR\'s Grants
Technical Assistants. Results of the competition are expected soon. Dr.
Ehrenfeld continued by explaining that the work of 13 additional CSR
staff members in the areas of committee management support and
information technology would be studied in

FY 2004.

[New Investigators]{.underline}: At the last CSRAC meeting, it was noted
that there was some round-to-round variation in how well new
investigator applications scored in relation to how well new
applications of experienced investigators scored, but new investigator
applications generally fared equally well. Dr. Ehrenfeld said that CSR
has since examined amended applications that were originally unscored to
see if new investigators were more adversely affected than experienced
investigators. No differences were observed between these groups of
applications. She continued by focusing on the kinds of funding
mechanisms utilized by new investigators. Between FY 2000 and FY 2003,
the proportion of R01 applications submitted by new investigators
increased from 25 percent to 35 percent. However, new R01 applications
as a proportion of all applications received by NIH during this same
time period declined from 66 to 60 percent, while the number of new R01
applications submitted by experienced investigators declined from 76
percent to 71 percent. Dr. Ehrenfeld explained that the percentage of
R21 applications submitted by both new and experienced investigators
went up during the study period. Dr. Leon asked about the success rates
for R21 applications. Dr. Ehrenfeld said that CSR could provide this
information and that it was working to better understand where these
applications are coming from.

[Chartering of Small Business Innovation Research (SBIR) Study
Sections]{.underline}: CSR has formed a working group to examine ways to
address concerns about reviewer diversity and continuity in unchartered
SBIR study sections. This group suggested that some of the guidelines
developed for chartered study sections should be applied to the
unchartered SBIR study sections. The group specifically proposed a
system to monitor reviewers and how long they have served and to set
targets for the inclusion of minorities and women reviewers. The
appropriate Division Directors and Integrated Review Group (IRG) Chiefs
would be provided with this information to allow such needs to be
addressed.

[Mock Study Section Video]{.underline}: The production company producing
the mock study section video for CSR has made extensive changes to the
video script in accord with comments made and discussed at the last
CSRAC meeting. CSR hopes the final video will be completed soon, and
when it is, CSR will send copies of it to CSRAC members.

[Review of Fellowship Applications]{.underline}: CSR intends to develop
an improved Web site for information on applying for NIH fellowships.

[Reorganization Evaluations]{.underline}: The completed evaluation of
the reorganized neuroscience study sections has been posted on CSR\'s
Web site. CSR has begun efforts to conduct an evaluation of the
reorganized behavioral and social sciences study sections.

**Scoring Scales**

Dr. Stanfield introduced Dr. Connie Atwell, who chaired a 1996 Committee
on Improving Peer Review and who was a member of its Rating of Grant
Applications Subcommittee. He explained that she works for the National
Institute of Neurological Disorders and Stroke, where she is now the
Director of its Division of Extramural Research.

Dr. Atwell said that her committee worked to identify ways to improve
how applications are evaluated, focusing on ways to improve the
quantification of scores. Her committee suggested a number of changes to
criteria for reviewing grant applications, but only one change was made,
i.e., the addition of \"innovation\" as a review criterion. Her
committee also recommended that reviewers write critiques with sections
for each of the review criteria and that reviewers give individual
scores for each of the review criteria instead of one integrated score
for the application. NIH adopted the practice of asking reviewers to
write something for each review criterion, but it does not ask them to
provide scores for each criterion.

Dr. Atwell continued by describing three recommendations related to the
rating scale: (1) use larger scale values to indicate desired
characteristics; (2) use a number of possible scores (0-7 or 0-10) that
are more attuned to the human ability to make reliable judgments; and
(3) anchor rating scales with descriptors (\"outstanding,\" etc.) only
at the ends of the rating scale. In addition, Dr. Atwell said her
committee recommended that percentile scores be based on each
reviewer\'s scoring pattern instead of the study section\'s scoring
pattern, and that percentiles be calculated for each review criterion.
Her committee also recommended that scores be provided for the three
criteria, but if a single score is given, it should be calculated from
the averaged scores for the three criteria using a common algorithm. She
noted that none of these recommendations was adopted.

Dr. David Williams asked Dr. Atwell what she thought the key priorities
would be now. She suggested having reviewers give scores for each of the
review criteria and having them use more reliable scoring scales. Dr.
Williams then asked about the pitfalls her group experienced so that
CSRAC could learn from them. Dr. Stanfield said that one of the big
issues raised by the Peer Review Oversight Group when these
recommendations were discussed was the concern that reviewers would lose
control if they no longer gave a global score to each application.

Dr. Atwell said that a single score still could be developed from scores
given to individual criteria. She said that the stumbling block was the
belief that reviewers are capable of calculating and intuiting
assessments with a level of precision that is not justified by the
psychological literature.

After considering these issues, Dr. Edward Pugh said he thought it was
important to discuss what NIH wants its reviewers to do, what they are
indeed doing, and how best to assess reviewer behavior and better train
them for their jobs. CSR needs to better understand not just scoring
behavior but the kinds of things reviewers emphasize in their critiques
and how they present them. Dr. Pugh then questioned the viability of
internally rescaling scores, explaining that doing so may not be
productive. He said that the fate of an application is usually in the
hands of three or four reviewers, and it often takes only one of them to
scuttle it. He added that the prescores may reflect a better rating of a
grant application, since individual scores tend to congeal at review
meetings. Encouraging discussion would be helpful, but averaging scores
that are not \"independent\" may not provide better information.

Dr. David Soybel said scoring applications by individual review criteria
may be useful so long as the ICs have a transparent mechanism to
integrate the scores so that the final funding decisions are seen as
fair. Dr. Pugh agreed and then said he was concerned about how reviewers
sometimes try to refine scores so that they can in effect decide whether
a grant application will be above or below the payline. He suggested
that a cadre of senior reviewers trusted by the scientific community
could be useful in assessing study section behavior.

Dr. Williams asked Dr. Atwell if her committee addressed how different
criteria should be weighted. She said her committee performed some
simulations using different algorithms and found that they would not
make a big difference when assessing large groups of applications. The
committee, however, recommended that a score computed from individual
criterion scores would be a more accurate score than a single overall
score. Dr. Leon suggested that CSRAC members discuss this issue at their
next meeting.

**NIH Director\'s Update**

The NIH Director, Dr. Elias Zerhouni, noted that Dr. Ehrenfeld would be
stepping down as CSR\'s Director and that Dr. Stanfield would become
CSR\'s Acting Director. He praised CSR for its successes in managing its
increasingly complex responsibilities, and he thanked Dr. Ehrenfeld for
her leadership, particularly for her efforts to advance the
identification of innovative research.

Dr. Zerhouni explained how extraordinarily fast science is moving and
the degree to which fields are merging. To get a better handle on how
NIH can best meet its evolving challenges, he worked with the NIH IC
Directors to convene 15 working groups, which included experts from
inside and outside NIH. These working groups focused on five needs: (1)
developing new pathways of discovery to stimulate scientific teams to
address increasingly complex challenges, (2) building new models and
understanding of biological pathways, (3) advancing multidisciplinary
research, (4) improving translational research efforts, and (5)
reengineering the clinical research enterprise. Dr. Zerhouni then
described how he and the IC Directors developed the multiyear NIH
Roadmap initiatives to begin to address these needs. He briefly
discussed the high-risk research initiative to fund innovative research.
NIH would experiment with funding creative researchers and then with
funding creative research projects.

Dr. Leon asked about the naming of a new Director for the NIH Office of
Extramural Research (OER). Dr. Zerhouni said that he had first formed a
small group to examine OER\'s structure, and a new OER Director soon
would be named. Dr. Leon also asked about modular budgets, and Dr.
Zerhouni said that they simplify NIH management tasks and help keep its
administrative costs low. Dr. Soybel asked him to comment on the future
fate of clinicians who are principal investigators.

Dr. Zerhouni noted how increases in scientific complexities and
regulatory burdens have made it more difficult to do clinical research.
He said that clinical researchers need to be nurtured with specialized
training, and they need access to the appropriate physical and
intellectual tools.

Dr. James Hildreth asked about the representation of minorities in the
higher levels of NIH management. Dr. Zerhouni noted that NIH was just
named to receive a national diversity award, but he acknowledged the
importance of continuing to recruit minorities into leadership roles.

Dr. Craig McClain returned to the issue of clinical researchers, asking
if NIH had a plan to increase the number of General Clinical Research
Centers. Dr. Zerhouni said that NIH would take a look at all clinical
research efforts to see what can be done to advance this important area
of research. Dr. Williams then asked Dr. Zerhouni to comment on the role
of behavioral and social sciences in his initiatives. Dr. Zerhouni said
that they permeate almost every other NIH activity, and he expects they
will evolve with new NIH activities. He concluded his remarks by
thanking CSRAC members for their efforts.

**Streamlining the Review of Grant Applications**

Dr. Richard Panniers, SRA, CSR Physiological Chemistry Study Section,
explained that some study sections are now using prescores submitted via
the Internet-Assisted Peer Review System to identify those grant
applications that would fall in the lower half for streamlining before
the review meeting. He discussed the possibility of adopting such a
practice for all CSR study sections. Dr. Panniers then showed a sample
set of prescores and highlighted issues that should be considered: (1) a
third of applications received scores that varied significantly among
different reviewers; (2) not all reviewers met the deadline to submit
prescores, although compliance averaged about 70-85 percent when it was
evaluated about a year ago; (3) SRAs ask reviewers to submit
streamlining recommendations in letterform (\"UN\" for unscored);
numerical scores would be necessary if an averaging method were adopted
to identify streamlining candidates; (4) temporary members of the study
section may not be aware of the score scale used by each study section
before the meeting; and (5) reviewers do not use the less favorable end
of the scoring scale and they may need extra guidance to do this.

Dr. Panniers then discussed two potential methods for using prescores to
generate an initial list of applications to be streamlined: (1) average
the prescores and make the lower half the initial streamline list, and
(2) enhance the current procedure and use prescores to help identify
applications with two nominations for streamlining. He recommended that
applications with significantly varying prescores be flagged to call
attention to applications where there may be one or more options showing
enthusiasm much greater than a streamlining range. In addition, he
suggested that reviewers should still be given the opportunity to add or
remove applications from the list, and that final streamlining decisions
must be unanimous. Dr. Panniers concluded his comments saying that study
sections should be very conservative about streamlining and that
individual study sections should be allowed to decide the most
appropriate streamlining methods for the spread of the prescores seen at
any given meeting.

Drs. Pugh and Hildreth expressed an interest in identifying ways to come
up with scores that better represent reviewers' individual assessments
before they are averaged and become congealed. Dr. Hildreth asked if CSR
was trying to determine a set percentage of applications that its study
sections should streamline. Dr. Panniers said that the study sections
typically set their own cut-off points for streamlining depending on
recommendations of reviewers, and applications that receive
significantly discordant prescores are always discussed. He noted that
streamlining decisions are more informed now that reviewers can read
each other\'s critiques via the Internet-Assisted Peer Review System.
Dr. Leon added that if any reviewer thinks that an application proposed
from streamlining has high merit, that application is discussed.

Dr. McClain said that sometimes this practice is a problem because
valuable time can be wasted when a single reviewer wants to discuss an
application. Dr. Leon suggested that a number of applications are
discussed because reviewers do not want to unscore them. He added by
saying that he thought it was unfair for different study sections to
have different cut-off points for streamlining. He suggested that it
would be better for CSR to have reviewers score all applications and
have a set cut-off point for streamlining. Dr. Soybel said that
applicants should know the degree to which the study section thought
their proposals were good and feasible as well as how fixable they are
in light of concerns raised by the study section. He said that the
practice of unscoring streamlined applications was thus problematic.
Since this concern was the topic of the next presentation, CSRAC members
agreed to table their discussions.

## Scoring Streamlined Applications

Dr. Anita Miller Sostek, Director, CSR Division of Clinical and
Population-Based Studies, provided an overview of the practice of
streamlining reviews of applications that fall in the lower half of
those considered by study sections. She then focused on the benefits of
this practice:

\(1\) study sections have more time to have in-depth discussion of those
applications with high merit, (2) study section meetings are shorter and
costs are lower, and (3) applicants still receive summary statements
containing reviewer critiques, which are usually unaltered. Dr. Sostek
continued by discussing the disadvantages of streamlining: (1) there is
likely a stigma to having one\'s application streamlined, (2) reviewers
may be pressured to reach consensus on applications proposed for
streamlining, (3) new investigators may not know how to deal with a
streamlined application, and (4) scores for all applications are not
necessarily submitted, which hinders consistency in scoring these
applications. Reviewers with conflicting reviews are often advised of
the situation and encouraged to read each other's critiques and revise
theirs if appropriate, although they are not obliged to do this.

Two options for scoring streamlined applications were proposed by Dr.
Sostek as possible ways to diminish the negative effects of
streamlining. If the assigned reviewers recommended scores, the rating
of streamlined applications could be made more informative by providing
the average or the range of those scores. If one of these options were
implemented, reviewers might be more conscientious about revising
critiques and scores after their meetings, and applicants might get a
better idea of reviewer assessments. Dr. Sostek then focused on
disadvantages, saying that both options would increase review burdens.
Providing a range of prescores would not be affected by discussion of
the streamlined applications or by sharing their critiques on the Web.
Broad score ranges and mixed scoring patterns might lead to an increase
in applicant appeals. Outliers may be particularly confusing. Providing
score ranges to streamlined applicants could lead others to demand that
this information be provided for \"discussed\" applications as well. Dr.
Sostek mentioned another option: continue providing "UN" letter scores
but adding additional letters to better signify the range of reviewer
sentiments.

Dr. Williams said he strongly favored providing numeric scores to all
applications as a means of reducing the stigma associated with
streamlining, and he said that he thought it would be easiest to average
scores since applications that receive scores that vary significantly
are typically discussed. Dr. Williams said that many streamlined
applications possess great potential, and he reiterated his preference
for seeing all applications scored. Dr. Soybel said he agreed, adding
that it was reasonable not to discuss all applications, but that it was
unreasonable not to score all applications. Dr. Leon focused on another
benefit to scoring all applications. Doing so could allow the ICs more
flexibility in funding innovative proposals and those submitted by new
investigators. After Dr. Stanfield explained that the ICs may fund
unscored applications,

Dr. Leon suggested that Advisory Councils would likely be more
accommodating if these applications were scored.

Dr. Ehrenfeld asked why Dr. Williams favored assigning averaged scores
vs. the range of scores to streamlined applications. He said that a
range of scores could lead applicants to contest their reviews. Dr.
Ehrenfeld agreed that this could be a problem, but that she generally
found it better to reveal information instead of hiding it. Dr.
Stanfield noted that it would likely take a long time to have the NIH
computer system generate score ranges but that it can provide averaged
scores now. Drs. Leon and Pugh agreed that it would be best to provide
averaged scores. Before the CSRAC members voted on the issue, it was
noted that these averaged scores from just the assigned reviewers would
not be the same as averaged scores from all the members of the study
section after the applications were fully discussed. Dr. Ehrenfeld
suggested that this fact could be mentioned in an administrative note
included on the streamlined applications\' summary statement.

Dr. Matt Winkler returned to the issue of scoring each of the review
criteria. Drs. Pugh and Leon agreed that CSRAC members should discuss
this issue at a future meeting. Dr. Leon moved to endorse assigning
averaged prescores to streamlined applications. CSRAC members discussed
whether there should be a preset level at which streamlining would occur
or whether study sections should be left to set their own proportion of
streamlined proposals. Dr. Ehrenfeld suggested that CSR staff draft a
best practices document on this point and provide it to CSRAC members
for discussion. Dr. Leon then moved that CSRAC members endorse the
general concept of assigning streamlined applications scores that are
calculated from prescores. His motion was passed unanimously.

## 

## Update on the Regulatory Checklist

Dr. Cheryl Corsaro, SRA, CSR Mammalian Genetics Study Section, said that
a regulatory checklist was developed to help reviewers deal with the
increased burden of their responsibilities for assessing whether
investigators appropriately address (1) issues regarding protection of
human subjects in research; (2) policies on inclusion of women,
minorities and children in research; and (3) policies regarding the use
of vertebrate animals in research. She noted that she made a
presentation on this topic a year earlier when CSRAC members discussed a
number of possibilities for helping reviewers deal with regulatory
issues. At that time, members decided against recruiting temporary
reviewers whose primary responsibility would be to address regulatory
issues and endorsed the idea of developing a regulatory checklist to
help regular reviewers deal with these responsibilities. Dr. Corsaro
explained that a checklist was developed and tested during the summer in
two study sections. The checklist or worksheet consists of five sections
that cover (1) protection of human subjects from research risks, (2)
inclusion of women, (3) inclusion of minorities, (4) inclusion of
children, and (5) use of vertebrate animals.

Dr. Corsaro explained that a few reviewers in the pilot actually filled
out the worksheet and the SRAs had to transfer information to their
critiques. She emphasized that the worksheet was intended to help
reviewers complete their reviews and not to be an extra form for them to
fill out. She explained that the worksheet would be further piloted by
eight study sections in the fall review round to get a better idea as to
how useful it might be.

Dr. Soybel said that he thought the worksheet would be very helpful to
reviewers. He said that it would be important to be sure it was clear to
reviewers that they need to address these regulatory issues in their
written critiques. He asked why the worksheet uses the terms \"inclusion
of women\" and not \"inclusion of genders.\" Dr. Corsaro said that the
language comes from an NIH policy document, and that problems of gender
inclusion and age inclusion have historically been related to the
exclusion of women and children. Dr. Soybel then noted that the
worksheet asks reviewers to check the justification for the numbers of
animals to be studied but not the numbers of humans to be studied. Dr.
McClain said that this issue was covered by other areas reviewers are
asked to consider. A question arose about the need for data safety and
monitoring plans in all human subjects research. Dr. Corsaro said that
CSR would investigate the issue and report back to the committee. Dr.
McClain then asked why the worksheet prompts reviewers to write about
how the plan for the inclusion of women and minorities is acceptable,
rather than make comments only if it is unacceptable. Dr. Sostek
explained that the General Accounting Office has asked that it be clear
that reviewers addressed these issues. Dr. Leon concluded this
discussion, saying that CSRAC members were looking forward to the
results of the second pilot test of the worksheet.

# Study Section Reorganization

Dr. Elliot Postow, Director, CSR Division of Biologic Basis of Disease,
provided an overview of the four basic principles developed by the Panel
on Scientific Boundaries for Review (PSBR) to guide CSR's
reorganization: (1) study section boundaries should not be too broad or
too narrow; (2) there should be sufficient overlap between other study
sections inside and outside their IRGs; (3) reviews should be provided
in the context of biological questions; and (4) clinical applications
should be clustered when possible.

### Infectious Diseases and Microbiology IRG

Dr. Postow explained that the Study Section Boundary Team that drafted
guidelines for the Infectious Diseases and Microbiology IRG proposed
eight study sections: (1) Microbial Cell and Molecular Biology; (2)
Bacterial Pathogenesis; (3) Pathogenic Eukaryotes; (4) Basic Virology;
(5) Virology; (6) Enviropathogenesis, Clinical Research and
Epidemiology; (7) Drug Discovery and Mechanisms of Antimicrobial
Resistance; and (8) Vaccine Development and Immunology of Infectious
Diseases. In addition, the boundaries team proposed a Special Emphasis
Panel (SEP) on Vector Biology.

In reviewing Web comments submitted by the scientific community, CSR
received varying comments on how well the proposed study sections would
support translational research. There was support for the Pathogenic
Eukaryotes Study Section. Many thought that the two proposed virology
study sections artificially divided the field. Dr. Postow continued by
discussing the overlaps that existed with other IRGs. The Immunology IRG
contains a vaccine study section and also covers innate immunity, and
the Genes, Genomes and Genetics IRG also covers prokaryotic genetics. In
addition, the area of eukaryotic biology is covered by a number of other
IRG study sections. Dr. Postow explained how these and other issues were
resolved in telephone consultations with ad hoc advisors: The two
proposed virology study sections were converted into two mirror study
sections. The review of vaccine applications will be centralized in the
Immunology IRG. Concerns about innate immunity applications were
resolved by a decision to refer those tied to pathogenicity to the
Infectious Diseases and Microbiology IRG and to refer those of a more
general nature to the Immunology IRG. Concerns about the prokaryotic
genetics applications were resolved by a decision to refer basic science
applications to the Genetics IRG and the others to the appropriate organ
system or disease study section. Dr. Postow explained that the Bacterial
Pathogenesis Study Section was split into two study sections after CSR
realized it would receive too many applications following a second mock
sort of applications. A reconfigured Bacterial Pathogenesis Study
Section will review those applications focused on the pathogen while a
new Host Interactions to Bacterial Pathogens Study Section will review
those focused on the host response to pathogens. Finally, the proposed
Vector Biology SEP would become a regular study section.

After these modifications, a total of nine study sections were proposed
for the Infectious Diseases and Microbiology IRG: (1) Prokaryotic Cell
and Molecular Biology, (2) Bacterial Pathogenesis, (3) Host Interactions
to Bacterial Pathogenes, (4) Pathogenic Eukayotes,

\(5\) Virology A, (6) Virology B, (7) Clinical Research and Field
Studies of Infectious Diseases,

\(8\) Drug Discovery and Mechanisms of Antimicrobial Resistance, and (9)
Vector Biology.

Dr. Lucia Rothman-Denes began the discussion saying that she thought the
scientific area covered by the Pathogenic Eukayotes Study Section was
large and that it would be interesting to see how well it works. She
said that the two bacterial pathogenesis study sections would likely
work well, and that the creation of two mirror virology study sections
was an interesting development. Dr. Rothman-Denes also noted that many
scientists in the community were excited about the new Drug Discovery
and Mechanisms of Antimicrobial Resistance Study Section.

CSRAC member Dr. Susan Berget participated in this discussion via
telephone. She asked if the proposed guidelines were flexible enough to
handle future demands. Dr. Postow said that CSR would adapt as
necessary. Dr. Berget then said she thought the proposed guidelines
looked good.

Dr. Ehrenfeld noted that these guidelines have undergone more
modifications than usual. She said they were understandable since the
infectious disease and microbiology portfolio and mindset has changed
enormously in the last year and a half since the boundaries team met.
Not only had the science changed, but more interest and money was being
devoted to biodefense research. CSRAC members then unanimously endorsed
the revised guidelines.

*Genes, Genomes and Genetics IRG*

Dr. Donald Schneider, Director, CSR Division of Molecular and Cellular
Mechanisms, explained that PSBR called for two gene-related IRGs, but
when the boundaries team looked at the applications assigned to them in
the mock sort, it decided to combine the two IRGs into one with six
study sections: Molecular Genetics A, B, and C---three mirror study
sections for research on the mechanisms and regulation of gene
expression; Genomics, Computational Biology and Technology; Genetic
Variation and Evolution; and Genetics of Health and Disease.

Dr. Schneider then summarized the comments submitted via the Web by the
scientific community. Considerable support for the redesign was evident,
especially for the Genetic Variation and Evolution Study Section. There
were concerns that (1) the Molecular Genetics A, B and C Study Sections
could slight prokaryotic genetics and RNA research; (2) the Genomics,
Computational Biology and Technology Study Section is too broad and
technology would crowd basic research; (3) the Genetic Variation and
Evolution Study Section might be improved if it had more of a focus on
concepts rather than on organisms; and (4) the Genetics of Health and
Disease Study Section may be too broad. To address these concerns, CSR
convened teleconferences with CSR/NIH staff and external experts,
including a few CSRAC members.

[Molecular Genetics Study Sections]{.underline}: Participants
unanimously recommended that prokaryotic genetics should have two
homes---one in the Genes, Genomes and Genetics IRG (the Molecular
Genetics A Study Section) and one in the Infectious Diseases and
Microbiology IRG. They also discussed whether to define the Molecular
Genetics Study Sections further. CSR staff subsequently examined the
applications sorted to these three study sections. Seventy percent of
them covered five topics: (1) DNA replication and cell cycle control,
(2) DNA repair,

\(3\) transcription mechanisms and regulation, (4) RNA processing and
stability, and (5) protein synthesis and translation control.
Participants suggested that all these topics be covered by each of the
three study sections. They also proposed that the remaining applications
be clustered in either one or two of the Molecular Genetics Study
Sections.

[Genomics, Computational Biology and Technology Study
Section]{.underline}: Participants unanimously recommended that the
Bioengineering Sciences and Technologies IRG should be the primary home
for pure technology development applications. The Genomics,
Computational Biology and Technology Study Section should consider
applications where technologies are applied to or service biological
questions. Participants, however, thought that there should be
overlapping boundaries and that having two homes for these applications
was inherently good.

[Genetic Variation and Evolution Study Section]{.underline}:
Participants unanimously recommended that this study section should be
implemented despite the fact that it did not receive many applications
in the second mock sort. They felt that it would meet a growing
scientific need for a home for applications with a focus on quantitative
and mathematical genetics. CSR staff subsequently examined applications
received for the January 2004 review round and found that this study
section would have a viable number of proposals to review.

[Genetics of Health and Disease Study Section]{.underline}: Participants
considered dividing this study section in two. One would focus on model
organisms and the other on health and disease. They, however,
unanimously agreed that these applications should be reviewed together.

Dr. Schneider noted that an interim SEP was currently covering the areas
of gene therapy and inborn errors. Participants suggested that the
Genetics of Health and Disease Study Section could cover these areas.
However, the Gene Therapy and Inborn Errors SEP should be continued,
particularly if the Genetics of Health and Disease Study Section
receives too many applications to review.

Dr. Berget said that she thought a good plan had been developed. She
said that she was not pleased to see the clustering of prokaryotic
genetics, because she liked the original approach of not dividing these
study sections by organism, but according to the biological questions.

Dr. Rothman-Denes noted that the drive to cluster this and other
research in the Molecular Genetics Study Sections came from a concern
that it would be difficult to have all the appropriate expertise in
three mirror study sections. Dr. Berget said that the Genomics,
Computational Biology and Technology Study Section would likely have
reviewers with two different perspectives on the science, but she said
that this was to be expected, given the subject matter. Over time, the
differences between these reviewers should diminish. Dr. Rothman-Denes
then said that there was a great deal of excitement in the community
about the Genetic Variation and Evolution Study Section. After this
discussion, CSRAC members voted unanimously to endorse the modified
guidelines for the Genes, Genomes and Genetics IRG.

**NIH Investment in High-Risk, Innovative Research**

Dr. Ehrenfeld described CSR\'s previous efforts to partner with some NIH
Institutes to develop new ways to encourage, identify, and fund
innovative research and how these efforts recently became a part of the
new NIH Roadmap initiative. She explained that all of the Roadmap
advisory groups made the following conclusion: the current NIH system of
peer review works very well most of the time in identifying and funding
the best biomedical research, but NIH also needs a completely different
approach targeted specifically to support innovative people and
high-risk projects that may produce high-impact conceptual and
technological breakthroughs.

Dr. Zerhouni created a working group to consider options for advancing
high-risk research. He asked that this group be chaired by Dr. Ehrenfeld
and Dr. Stephen Straus, Director of the National Center for
Complementary and Alternative Medicine. They assembled a group of
representatives from NIH ICs and a group of external experts. This group
met in June 2003 and recommended that NIH develop three programs to
advance high-risk research: (1) Exceptional Projects Program, which
would fund individual projects with exceptional promise; (2) Grand
Challenges Program, which would identify broad, new program areas of
interest to multiple ICs; and (3) Director\'s Innovator Award Program,
which would support individuals with exceptional promise.

Dr. Ehrenfeld explained that these programs were discussed at a
subsequent meeting of all IC Directors. They agreed to set aside FY 2004
funds to initiate the Director\'s Innovator Award Program. She explained
that the details had yet to be determined, but she provided an overview
of some of the possibilities being considered. The program could (1)
support individuals at all career levels with the greatest potential for
conducting high-risk, high-impact research, (2) be modeled after other
successful programs such as the MacArthur award program, (3) allow
awardees to set their own research agenda, (4) involve elite panels for
selecting and reviewing scientists based on interviews and
presentations, and (5) award about \$500,000 per year for each project
for 5 years.

Nominations would be made in a two-page description of the candidate\'s
creative abilities and potential for groundbreaking discovery; and
nominations could be made by mentors, colleagues, institutions, or the
candidates themselves. NIH staff would consult with outside experts to

\(1\) evaluate nominees, (2) invite candidates to submit a 3-5 page
essay giving evidence of potential for success along with a resume and
professional references, and (3) interview top candidates. Those
applications with the most promise would be reviewed a second time by an
advisory committee to the NIH Director. Awardees would meet and file
annual progress reports. After emphasizing the tentative nature of the
program details, Dr. Ehrenfeld discussed a proposed timeline, which
called for nominations to be solicited early in calendar year 2004, with
awards being made by the end of September 2004.

Dr. Ehrenfeld said that the Director\'s Innovator Award Program was an
experiment that will complement, rather than replace current IC research
efforts; spawn new NIH programs; and send a signal that NIH values
innovation. She said that there would be further discussions with

Dr. Zerhouni and IC representatives to help refine the plan. In
addition, the NIH Communications Director will help prepare and
implement a marketing strategy for the program.

Dr. Pugh said that the initiative would go a long way in addressing the
perception that the NIH grants process is too conservative. He said that
it would be important for the community to know that the new high-risk
awards are distributed fairly, and for NIH to conduct an evaluation to
see if it changes perceptions and to assess the individuals who receive
the awards.

Dr. Ehrenfeld said that the awards process will be transparent once the
first award round is completed, but she said it would be important for
NIH to produce a frequently-asked-questions document to educate the
community about the program.

Dr. Leon said he was enthusiastic about the proposed program and agreed
with Dr. Pugh in saying that it would help address concerns that the NIH
grants program is too conservative.

Dr. Ehrenfeld said that she hoped the selection process would be geared
to provide awards to scientists who need a primary source of funds
instead of providing awards to researchers who are already well funded.
Dr. Rothman-Denes asked if there was a way to measure how many times NIH
has failed to fund really innovative research proposals. Dr. Ehrenfeld
said that it would be difficult to do much more than gather anecdotes.

Dr. Soybel wondered if the results would differ if, instead of seeking
to identify and fund the most creative researchers, NIH randomly funded
a number of scientists from a pool of accomplished researchers. He said
that it might be interesting to conduct a study to see if the results
would differ. Dr. Ehrenfeld said that it would be difficult to justify
funding researchers randomly and conducting a human subjects experiment.
Dr. Soybel said that the important question would be how much the awards
change the direction of research and lead to something new and
interesting that only occurred because the researchers had the funds.

**Update on the Review of Clinical Research Applications**

Dr. Theodore Kotchen, CSR Special Advisor on Clinical Research, noted
that one of the reasons he was recruited was to help determine if there
is any validity to the perception that clinical grant applications are
not fairly reviewed. He then summarized data he presented to CSRAC
members at their last meeting, comparing review outcomes and funding
rates for clinical and non-clinical research applications considered for
the May and October 2002 councils. This data showed that these funding
cycles and review outcomes (1) did not differ for M.Ds. vs. non-M.Ds.,
(2) did not differ for M.Ds. vs. M.D./Ph.Ds.,and (3) were less favorable
for \"clinical\" applications. In addition, among \"clinical\"
applications, outcomes were less favorable for those with human subject
concerns.

Dr. Kotchen explained that \"clinical\" applications were defined as
those where applicants checked the box indicating that their research
would involve the use of human subjects. He noted that this definition
was problematic, since it included many other studies beyond clinical
trials. To get more useful data, CSR reviewed all the \"clinical\"
applications reviewed for the October 2002 councils and coded them
according to clinical research categories used by the Institute of
Medicine and others: (1) mechanisms of human disease, (2) clinical
trials and other clinical interventions, (3) development of new
technologies, (4) epidemiologic studies,

\(5\) behavioral studies, (6) health services research, and (7)
utilization of human tissue.

Dr. Kotchen explained that, among all these types of clinical
applications, those in the first two categories (mechanisms of disease
and clinical trials) received significantly less favorable scores than
those proposals in the other categories. He said that it would be
worthwhile to gather additional data to understand these findings
better. He then welcomed comments from CSRAC members.

Dr. Soybel said that this data confirmed his impression from serving on
a study section that applications for clinical trials research tended
not to fare as well as other applications. He suggested two possible
reasons for this difference: (1) the clinical research applicants are
not appropriately trained and (2) their applications are often reviewed
along with basic research applications. The reviewers in these
situations may not have the in-depth familiarity with the kinds of
clinical research that was being proposed. Dr. Kotchen added two more
possible reasons: (1) the review criteria may not be appropriate for
reviewing clinical projects, and

\(2\) many of the clinical research applications may just be poorly
prepared. Dr. McClain said that obtaining appropriate reviewers is a
major issue for assessing clinical applications. He noted that his
clinical research proposals often receive inconsistent reviews, and he
has observed reviewers who raise inappropriate human subjects concerns.

Dr. Kotchen said that he recently reviewed groups of applications that
scored very well and very poorly and developed an advisory document for
applicants to help them avoid problems in their applications. Dr. Leon
asked if a similar advisory document could be produced for reviewers.
Dr. Kotchen said that such a document was drafted, but CSR and NIH staff
did not think it would be useful.

Dr. Winkler suggested that NIH might want to consider simply adjusting
its payline for clinical research applications and funding more of them.
Dr. Ehrenfeld noted that the ICs already have the ability to do this.
Dr. Leon hoped that NIH would not want to fund poorly developed clinical
projects. Dr. McClain said that giving extra review points to clinical
applications would not solve the problems related to review
inconsistencies.

**Review of Fellowships in Behavioral Sciences**

Dr. Sostek reminded CSRAC members of their January 2003 discussions on
the review of fellowship applications in the behavioral and social
sciences. These applications are generally reviewed with R01
applications in regular study sections while most other fellowship
applications are reviewed in dedicated study sections. She said that
previously presented data showed that fellowship applications reviewed
in the regular behavioral and social sciences study sections received
somewhat more favorable scores. This trend persisted over three review
rounds for both predoctoral (F-31) and postdoctoral (F-32) applications.

In further assessing the situation, Dr. Sostek examined a recently
created fellowship review panel in one of the behavioral sciences IRGs,
which was formed for workload reasons. She explained that the scoring
pattern did not change. CSR then recalculated the percentiles that
fellowship applications received in regular behavioral and social
sciences study sections. Instead of calculating percentiles based on all
the fellowship and R01 applications reviewed in their respective study
sections, CSR calculated percentiles based on all the fellowship
applications reviewed in their respective IRGs. Dr. Sostek reported that
the percentiles calculated from the IRG base were more balanced than for
those calculated for each study section.

She then discussed three options for addressing the disparity of scores
received by fellowship applications reviewed in regular study sections
and dedicated study sections: (1) review all fellowship applications in
dedicated study sections even though it may be difficult for them to
handle the wide range of science covered in those IRGs, (2) use an IRG
base when calculating percentiles for fellowship applications reviewed
in regular study sections, and (3) continue efforts to encourage those
who review fellowship applications to spread their scores. Dr. Sostek
also suggested that CSR consider conducting some qualitative analysis of
the fellowship study sections to understand better how well the
fellowship study sections are functioning. She then noted that reviewers
in behavioral and social sciences had been known for being too hard on
fellowship applications and they were encouraged to be more supportive.

Dr. Leon asked her for her thoughts on the best options to pursue. Dr.
Sostek said that she did not think there was a large scientific need to
assign all behavioral and social sciences fellowship applications to
dedicated study sections. Reviewers in regular study sections can shift
how they apply review criteria, but these reviewers certainly should be
encouraged to spread their scores more appropriately.

Dr. Stanfield said that reviewers could become more skilled in spreading
scores if CSR follows recommendations to encourage them to score
streamlined applications. Dr. Williams endorsed the proposal to
encourage reviewers to spread their scores. Dr. Ehrenfeld said that the
data just presented should be shared with the concerned reviewers, and
they would likely correct themselves. Dr. Leon agreed that providing
this data and encouragement would be helpful.

Dr. Williams said that it might also be helpful to study a group of
fellowship applications that scored well and a group that scored poorly
to identify characteristics of these applications for instructive
purposes. Dr. Pugh agreed, saying that such an analysis would provide
valuable information to the community.

# Reviewer Training

Dr. Noni Byrnes, SRA, CSR Bioanalytical Engineering and Chemistry
Special Emphasis Panel, said that current efforts at CSR were focused on
what could be done to improve the format and quality of summary
statements. She explained that a number of problems with reviewer
critiques had been identified: (1) an inappropriate focus on minor
technical issues; (2) lack of candor regarding boring/uninteresting
ideas; (3) unclear evaluations of significance/impact;

\(4\) descriptive vs. evaluative comments; (5) lack of attention to
mechanisms, specific review criteria, and weighting of criteria; and (6)
insufficient detail and unclear language.

Dr. Byrnes said that CSR has a number of reviewer training efforts
underway. It has an existing Reviewer Training Committee, has piloted
reviewer-training workshops, and, as reported earlier, is producing a
mock study-section video. She noted that three draft documents produced
by the Reviewer Training Committee were distributed to CSRAC members:
(1) Recommended Practices for Training Reviewers, (2) Annotated Sample
Critique for R01 Applications, and

\(3\) Frequently Asked Questions, which were drafted by Dr. Cathleen
Cooper, SRA, CSR Experimental Immunology Study Section.

Dr. Byrnes said that CSR already had a number of good documents on the
Web, but the main hurdle was getting reviewers to read them. She
proposed a two-pronged approach for addressing this problem: (1)
reducing the volume of information and improving the documentation and
navigation features of instructional information, and (2) enhancing the
interaction between SRAs and their reviewers. She said that enhancing
these communications is key to training reviewers.

To address the first issue, she showed CSRAC members a prototype of
reviewer guidelines in the pdf format adapted from the FAQ document that
could be sent by e-mail or on CD to provide easy navigation and allow
reviewers to obtain answers to specific questions without going through
unnecessary material.

To address the second issue, she proposed a draft checklist of
recommended practices for SRAs to guide their interactions with their
reviewers:

Approximately 1 week after applications are mailed to reviewers, the
SRAs are asked to call and speak with them about four issues: (1)
conflict of interest, (2) appropriate assignment of applications to
reviewers, (3) the different grant mechanisms, and (4) what reviewers
should expect at the study section meeting. Some of these issues would
apply only to new reviewers, but others are relevant for all reviewers.

Approximately 3 weeks before the meeting, the SRAs are asked to be sure
their reviewers understand six things: (1) evaluating different
mechanisms; (2) evaluating the "significance" of the proposed project;
(3) evaluating the "approach" of the proposed project; (4) evaluating
revised, new investigator, and renewal applications; (5) using the
Internet-Assisted Peer Review System and knowing that the SRA is
available to preview critiques; and (6) developing preliminary scores
and using procedures for streamlining.

Within a day or two after the submission deadline for using the
Internet-Assisted Peer Review System, the SRAs are encouraged to (1)
assess the quality of preliminary critiques and contact reviewers if
concerns surface, (2) be sure that scores and words are correlated, and
(3) identify where reviewer opinions diverge. Often, a cursory look at
one or two critiques from each reviewer will give the SRA a good
indication of whether or not the reviewer understands what is expected.

Dr. Byrnes said that she hopes that these practices become part of a
consistent, coordinated, and multifaceted effort to improve the quality
of reviewer critiques throughout CSR. She concluded her comments by
inviting comments from CSRAC members.

Dr. Hildreth said that CSR had done a great job with this initiative and
noted how helpful it was when he was a new reviewer and his SRA took him
aside to tell him what he needed to know about the roles of reviewers
and SRAs. He also said that another factor in determining how well a
critique is written is how well a reviewer's abilities are matched to
the assigned applications. Dr. Winkler said he was enthusiastic about
the proposed plan to improve reviewer critiques.

Dr. Leon said that he liked the idea of having SRAs call their reviewers
to address these issues instead of sending them e-mails. SRAs can assess
what reviewers need to know as they speak with them and thus better
address problems. He noted that SRAs frequently say that they spend a
great deal of time editing inappropriate comments out of reviewer
critiques. He suggested that CSR produce a document to educate reviewers
about what kinds of comments are inappropriate. Dr. Leon then asked
CSRAC members to indicate their support for this effort, and the motion
was passed.

*Training Workshops*

Dr. Michael Micklin, Chief, CSR Risk, Prevention and Health Behavior
IRG, explained how CSR had joined with the NIH Office of Behavioral and
Social Sciences Research to pilot workshops in July 2002 and July 2003
to train new reviewers. NIH and CSR staff along with outside consultants
participated in these workshops. Both workshops included an overview of
the NIH peer review process, comments from experienced reviewers, a mock
review involving the participants, and group discussions. The 2002
workshop included a mock review with SRAs while the 2003 workshop
included a prescreening of CSR's mock study section video and a case
study of a successfully amended application.

An e-mail evaluation consisting largely of open-ended questions was sent
to all participants. The response rates were high, and the overall
assessments were very positive. Participants said they most appreciated
(1) the discussion of the NIH organization and peer review process, (2)
their participation in a mock review, and (3) the comments from
experienced peer reviewers and NIH staff. Other topics of interest
included (1) the logic and procedures of scoring, (2) the significance
of study section culture, (3) the division of labor between NIH program
and review, (4) differences between grant mechanisms, and (5)
appropriate and inappropriate reviewer behavior.

Dr. Micklin then focused on the training issues/challenges that came to
the surface in conducting the pilot workshops: (1) What is the optimal
unit for training sessions---the division, IRG, or study section? (2)
When is the optimal time for training sessions---before temporary
reviewer assignment or before member assignment? (3) What is the optimal
mode of training---an in-person workshop, Web, CD, video, e-mail, or
printed materials? (4) What types of reviewers need
training---first-time reviewers, new members, rejuvenated members,
senior scholars, newly appointed study section chairs, or reviewers
serving on multidisciplinary research study sections?

Dr. Hildreth said that he thought the workshops appeared to be a great
way to train reviewers. Dr. Ehrenfeld asked if the majority of new
reviewers would be willing to attend a day-and-a-half training workshop.
Dr. Hildreth said that the newer investigators would likely come, and he
wished such a workshop were available when he first became a reviewer.
Dr. Micklin noted that the workshops were offered in the summer and
about 40 percent of those invited attended.

Dr. Hildreth asked if it would be possible to time these workshops a day
or two before or after review meetings. Dr. Ehrenfeld suggested that it
might be better to start training earlier, since these reviewers would
have already written their critiques. Dr. Winkler proposed that CSR
consider holding these workshops in conjunction with major scientific
meetings. Dr. Pugh suggested that new reviewers should be required to
attend a training workshop. Applicants deserve to have reviewers who are
appropriately prepared for the task. He continued by saying that the
training proposals presented could significantly enhance CSR\'s
workforce after

3 or 4 years.

Dr. Soybel also suggested that CSR consider requiring anyone who
receives a R01 grant to attend an NIH workshop on the peer review
process. Dr. Hildreth said he had reservations about asking new grantees
to participate in a required workshop. They could likely learn a great
deal from the mock study section video. Dr. Williams said he thought
some form of required training would be useful, but he said he favored
leaving CSR staff with the flexibility for deciding what kind of
training is used for different types of reviewers. Dr. Stanfield said
that CSR needs this flexibility, since it would be an intense drain on
its staff and budget to convene the 50 workshops that would be needed to
train all its new reviewers each year. Dr. Ehrenfeld noted that many
potential reviewers may not have the time to come to Washington for a
workshop. It might, however, be more realistic for CSR to convene
workshops in association with major scientific and professional
meetings. She added that CSR is learning about the effectiveness of
different training tools, and CSR should be able to do a lot by
combining them.

Dr. Leon said that CSRAC members see the value of these training tools.
He emphasized the importance of doing as much as can be done, and said
he thought it best to leave it up to CSR to decide what it can afford to
do. Dr. Ehrenfeld said that she hopes CSR will advance these training
efforts in the coming year.

**Agenda for the Next Meeting**

In discussing topics for the next CSRAC meeting, Dr. Leon said that
CSRAC members would consider the organization of the last IRG in CSR\'s
reorganization. He then suggested that they consider ways to speed up
the review process, make things better for CSR staff and reviewers, and
improve the quality of the reviews.

With no other business to address, CSRAC adjourned the meeting at 10:10
a.m.

We do hereby certify that, to the best of our knowledge, the foregoing
minutes of the 33rd meeting of CSRAC are accurate and complete. The
minutes will be considered at the 34th meeting of the Advisory
Committee, and any corrections or comments will be made at that meeting.

> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
>
> Brent Stanfield, Ph.D.
>
> Executive Secretary
>
> Center for Scientific Review Advisory Committee
>
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
>
> Ellie Ehrenfeld, Ph.D.
>
> Director
>
> Center for Scientific Review
>
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
>
> Michael Leon, Ph.D.
>
> Chair
>
> Center for Scientific Review Advisory Committee