Center for Scientific Review Advisory Committee Meeting

National Institutes of Health

U.S. Department of Health and Human Services

September 30 - October 1, 2002

The Center for Scientific Review (CSR) convened the 30th meeting of its
Advisory Committee at 8:30 a.m. on Monday, September 30, 2002, in
Conference Room 9100, Rockledge II Building. The entire meeting was held
in open session. Dr. Karen Matthews presided as Chair.

## Members

Karen Matthews, Ph.D., Chair Susan Berget, Ph.D.

Michael Colvin, M.D. Edward Pugh, Jr., Ph.D.

Lucia Rothman-Denes, Ph.D. David Soybel, M.D.

David Williams, Ph.D., M.P.H.

## Ad Hoc Advisors

Michael Leon, Ph.D. Marvin Wickens, Ph.D.

Dr. Howard Schachman was the official observer, and Dr. Brent Stanfield
was the Executive Secretary for the meeting.

**Welcome and Opening Remarks**

Dr. Matthews welcomed members of the CSR Advisory Committee (CSRAC) and
asked them to consider the minutes from the May 20, 2002 CSRAC meeting.
Dr. Michael Leon moved that the minutes be approved, Dr. Michael Colvin
seconded the motion, and the Committee approved them without comment.
Dr. Matthews then asked Dr. Ellie Ehrenfeld, CSR Director, to present
her update.

**CSR Update**

*Developing a Roadmap for the National Institutes of Health (NIH)*

Dr. Ehrenfeld described how the new NIH Director, Dr. Elias Zerhouni, is
seeking to develop an NIH roadmap that lists the most important
opportunities and obstacles facing medical research. His goal is to help
NIH pursue a course with the greatest positive impact on public health.
In August, Dr. Zerhouni held five meetings with NIH and extramural
scientists. They discussed many opportunities and obstacles in five
areas: (1) clinical research, (2) interdisciplinary research, (3)
research resources, (4) critical infrastructure issues, and (5) new
directions for research. Dr. Ehrenfeld added that development of this
roadmap is ongoing and that

Dr. Zerhouni will be seeking additional input.

*Special Advisor for Clinical Research*

Dr. Ehrenfeld introduced CSR\'s new Special Advisor for Clinical
Research, Dr. Theodore Kotchen. He will help CSR address challenges it
faces in reviewing clinical research applications. Dr. Kotchen earned
his M.D. from Case Western Reserve and has since held appointments at
the Walter Reed Army Medical Center and medical schools at Georgetown
University, the University of Kentucky, and the University of West
Virginia, where he was chair of its department of medicine. As an NIH
grantee for many years, he has studied hypertension control both in the
laboratory and in the clinic, linking population-based studies with
physiologic and genetic studies. He currently is professor of medicine
and epidemiology and associate dean for clinical research at the Medical
College of Wisconsin. He will continue his efforts there as he devotes
25 percent of his time to CSR. Dr. Ehrenfeld said that he was deeply
committed to helping CSR ensure that the reviews of clinical research
applications are fair, rigorous, and of a high quality. He specifically
will serve as a liaison with the external clinical research communities
to transmit and gather information about CSR\'s reorganization; help
educate clinical investigators and CSR to improve clinical research
reviews; and assist CSR in assessing how well it is doing in this area.

Dr. Kotchen said that he is delighted to be at CSR and looks forward to
working with CSR and reaching out to the clinical research community. As
he becomes more involved and familiar with his assignment, he will
welcome the advice of CSRAC members.

*CSR\'s Internship Program*

In August 2001, CSR piloted an internship program for biomedical and
behavioral scientists interested in careers in science administration.
Six interns from the NIH intramural program were recruited and given 1-2
year, full-time appointments. They have received formal training, have
been mentored by experienced scientific review administrators (SRAs),
and have received hands-on experience overseeing the peer review
process. One intern has already been hired as a full-time SRA, and
others are pursuing full-time appointments either in CSR or elsewhere at
NIH. In August 2002, four additional interns were recruited from NIH
intramural research laboratories. Next year, the program will be
expanded to include researchers from outside NIH. Applications are now
being sought for positions that will start next August.

*Senior Reviewers*

Dr. Ehrenfeld reminded members of previous discussions on recruiting
more senior reviewers by providing them opportunities to serve without
having to make an unrealistic commitment. Plans for a pilot program were
developed by a CSR committee chaired by Dr. Ann Hardy, SRA, Social
Sciences, Nursing, Epidemiology and Methods 5 Study Section. Her
committee recommended that senior reviewers be offered reduced terms of
service and play different roles as needed. CSR is now selecting two
study sections in each of its review divisions to participate in the
pilot during the February or June 2003 review rounds. Dr. Ehrenfeld said
she will solicit nominations from appropriate professional
organizations, and the SRAs will solicit nominations from other sources,
including NIH program staff.

*New Investigators*

Many in the community have expressed concerns about how well new
investigators fare when their applications are reviewed. Dr. Ehrenfeld
noted that NIH eliminated the R29 mechanism \-- a first award for new
investigators \-- because it was limited in scope, time, and budget. New
investigators are now asked to self-identify when they submit a 398
application for an R01 grant. SRAs usually attach one-sheet instructions
to these applications that encourage reviewers to consider the stage the
applicant is in his/her career. Dr. Ehrenfeld cited data on the scores
of applications submitted by new investigators and established
investigators. There is a 20-point difference between the scores of R01
applications from new investigators and the scores of new R01
applications submitted by previously funded applicants. The significance
and appropriateness of this difference could be debated, but CSR is more
concerned about a large variation in how new investigator applications
fare across different study sections. Dr. Ehrenfeld said that the SRAs
will provide data to their study sections on how well their new
investigator applications have fared so that reviewers will be able to
see if they need to change how they review these applications. She added
that agreements must be reached on how NIH defines new investigators.
Currently, they are defined as anyone who has not been in the NIH
database before. As a result, researchers who have been funded by other
agencies for many years are counted as new investigators, and
researchers who may have received small grants for preliminary projects
(R21s or R03s) are not considered new investigators. Dr. Ehrenfeld
emphasized the importance of having accurate data so that the situation
can be assessed appropriately.

*Improvements in Review Technology*

All study sections that wish to use the Internet-Assisted Peer Review
(IAPR) system are now able to do so. For the October 2002 round, 147
study sections will use IAPR. Dr. Ehrenfeld noted that IAPR does not
necessarily save time but it improves the peer review process. Because
reviewers can see critiques before their meeting, they know where their
reviews are disparate and can thus better focus their review
discussions. She added that CSR is also continuing to expand its use of
CDs in the October round. A total of 115 study sections will receive CDs
instead of the large boxes of applications they used to receive.

*Closure of the Medical and Biochemistry Study Section*

Dr. Ehrenfeld announced the closure of the Medical and Biochemistry
(MEDB) Study Section. She explained that this study section has covered
broad areas of science: animal models for inborn errors of metabolism;
some diagnostic biochemistry; gene therapy and delivery systems for
treating metabolic diseases; molecular/biological approaches to
characterizing gene products associated with these diseases; and lipid
metabolism, related receptors and anchor proteins, and associated
signaling pathways. Because of MEDB\'s breadth, its external advisory
group recommended that its applications be reviewed differently. Since
the MEDB\'s SRA is retiring at the end of the year, CSR has decided to
move ahead and divide MEDB\'s applications into two clusters. The
Genetic Sciences Integrated Review Group (IRG) will form a special
emphasis panel (SEP) to review applications related to inborn errors,
and the Physiological Chemistry Study Section will review the
lipid-related applications. Dr. Ehrenfeld emphasized that MEDB will
continue to meet through February 2003, and advance notice of the
planned change will be given to both applicants and reviewers.

*Questions and Answers*

Dr. David Soybel focused on new investigators and asked how many of them
were previously granted K awards, which are intended to bring new
researchers into the community. These data could help NIH determine how
well the K awards are working and how fairly reviewers may be looking at
applications from new investigators. Dr. Ehrenfeld said she would make
sure that these data are collected and shared with him. Dr. Brent
Stanfield, CSR Deputy Director, noted that some NIH Institutes use the
R21 mechanism as a transition award for new investigators. These
Institutes are frustrated by the fact that NIH does not consider these
investigators new when they later apply for R01 grants.

Dr. Matthews asked how members of the current MEDB Study Section have
responded to CSR\'s plan to close it. Dr. Camilla Day, Chief, CSR
Genetic Sciences IRG, said she called all MEDB members and found they
thought the plan made sense. Dr. Ehrenfeld noted that these reviewers
were prepared for change, since an external committee had already
reviewed MEDB and recommended that its applications be reviewed
differently. Dr. Matthews then asked if CSR will issue an announcement
on this change. Dr. Ehrenfeld said that an implementation plan will be
posted on the CSR Web site.

**Solicitation and Review of Innovative Research Applications**

Dr. David Armstrong, Chief, CSR Brain Disorders and Clinical
Neuroscience IRG, named the members of his subcommittee, which is
working to develop a pilot for soliciting and reviewing innovation
grants: Dr. R.V. Srinivas, SRA, CSR AIDS and Related Research 1 Study
Section; Dr. Edward Pugh; and Dr. David Williams. Drs. Ehrenfeld and
Stanfield have also been active in this effort.

CSR is discussing the pilot with the National Cancer Institute (NCI) and
the National Institute of Allergy and Infectious Diseases (NIAID).
Discussions with the National Institute of Neurological Disorders and
Stroke (NINDS) are on hold pending the naming of a new Institute
Director. Dr. Armstrong focused on the more advanced partnership with
NIAID. He explained that NIAID proposes to commit \$5 million a year to
an innovation pilot in the area of microbicides. Everyone agrees that
the pilot should seek a balance between the creative, analytic, and
practical aspects. In addition, the critical point of review should be
the impact in the field. Proposed research methods also should be sound,
and the proposed analysis should have clearly defined and anticipated
decision points. Dr. Armstrong briefly discussed the application format
and administrative guidelines before focusing on the structure of the
proposed application and the review criteria: (1) significance \-- will
the research clarify the problem to be investigated? (2) impact \-- will
it lead to new technologies, diagnostics, or treatments, or will it
improve medical practice? (3) research methodology \-- will its
methodology be appropriate, reasonable, creative and practicable? and
(4) Administrative compliance \-- will it comply with established
policies and regulations? Scores could be given for the first three
criteria, although additional discussions would be necessary to
determine how they are weighted.

In discussing future directions, Dr. Armstrong said that NIH workgroups
need to further discuss the mechanics of this initiative. He said that
everyone agrees that NIH must show the extramural community that this
innovation grant mechanism is truly new and different. A workshop with
the community has been proposed to (1) develop working definitions of
"conceptual and technical innovation"; (2) make recommendations on how
proposals would be reviewed;

\(3\) determine if a rating scale can be developed for evaluating the
creative, analytic, and practical aspects of the proposals; (4) engage
those in the community who stand to benefit from this initiative; and
(5) identify advisors to evaluate the initiative as it goes forward.

Dr. Ehrenfeld said that Dr. Armstrong had presented working ideas.
Before going forward, NIH needs to seek legal counsel and additional
advice from many sources. She emphasized the importance of developing an
application and process that convinces both applicants and reviewers
that NIH is looking to do something different. New investigators in the
community are routinely told that NIH funds conservative science and
they should submit a conservative application and then do what they want
to do if they get funding. Dr. Ehrenfeld said she thought it was worth
trying the pilot to see if this situation could be overcome. After
stressing the importance of proceeding slowly, she asked CSRAC members
for their comments and suggestions.

Dr. Pugh doubted whether existing study sections could be encouraged to
use paradigm-shifting terminology to evaluate grants. He said the new
mechanism could be more effective. Dr. Pugh then raised four points: (1)
Valuable research could be stimulated by the proposed emphasis on
problem-oriented research instead of hypothesis-driven research. NIH
Institutes could advance desired research by listing problems they think
should be addressed. (2) Important research would be stimulated with
"impact" as one of the review criteria. (3) The value of ordinary
research may be reduced too much if "paradigm-shifting" terminology is
overemphasized.

\(4\) Two aspects should be considered when evaluating the success of
this initiative \-- the review process and the programs.

Dr. Marvin Wickens said he found the proposed mechanism exciting. He
then asked about the range of funding levels. Dr. Srinivas said that
NIAID was considering grants of \$200,000 a year for 2-3 years. Dr.
Wickens suggested that a wider range of funding be considered. Sometimes
only a small amount is needed to make important progress in a line of
research. Dr. Ehrenfeld noted that funding levels would be determined by
the Institutes, but she agreed that a wider range of funding levels
should be considered. Dr. Wickens then asked how the area of
microbicides was chosen. Dr. Srinivas explained that two of NIAID's
divisions have a lot of interest in microbicides. In addition, he said
that this area involves many complex scientific disciplines, which are
needed to adequately conduct a pilot. Dr. Wickens asked if this
mechanism is being designed for targeted research areas. Dr. Ehrenfeld
said that many Institutes could identify research areas that may benefit
more from this mechanism, but she could not say how targeted the
mechanism will become.

Dr. Williams said that he was very supportive of the proposed
initiative. He then asked what kind of turn-around time these new
applications will have. Dr. Ehrenfeld said that it could be

3 to 6 months, noting that the limiting factor would be the busy
schedules reviewers have.

Dr. Williams also expressed a concern about the kinds of reviewers
needed for this initiative, suggesting that the proposed workshop
include discussions on this topic. In making his final comment, he
turned to the review criteria, suggesting that new "interventions" be
added to the kinds of impact reviewers should consider.

Dr. Soybel suggested that the proposed mechanism would likely fund
researchers from the major laboratories. He mentioned the possibility of
using a dual-track approach to supporting innovation. One would focus on
the innovative ideas researchers propose and the other would focus on
promising individuals willing to take risks and change fields. Dr.
Soybel noted that the R21 mechanism that was designed to support
innovative research seems to have favored predictable and conservative
research at major laboratories.

Dr. Leon said he liked the proposed review criteria. He particularly
liked the emphasis placed on impact and suggested that these criteria
could be useful to assessing innovation in R01 applications. He
expressed some doubt as to whether the proposed mechanism would really
promote innovative grants. Dr. Leon cited a similar Institute initiative
that failed when he served on its council. He suggested that NIH look at
the successes achieved at other funding agencies. The Office of Naval
Research uses a peer review process, but it can fund innovative research
more easily because (1) funding decisions are made by one individual,
(2) large failure rates are tolerated, and (3) sustained funding is
provided when results show promise. Dr. Ehrenfeld said she agreed with
much of what Dr. Leon said. CSR, however, can only address the mechanism
for reviewing these applications.

Dr. Wickens suggested the review process should recognize that risks are
necessary to make important advances. He then asked when the pilot would
be initiated. Dr. Ehrenfeld said that some large administrative hurdles
had to be overcome first and that she might be able to answer this
question in a few months.

Dr. Matthews encouraged NIH to develop an "off-the-shelf" mechanism that
could be targeted to new areas deemed important for rapid development.
She also said she hoped that R01 reviews could be used to identify
innovative research. In addition, she suggested that innovation can be
seen in methods, new scales, and new lab procedures as well as in new,
paradigm-shifting ideas. In moving forward, CSR should work to better
define "innovation" or "creativity." CSR could help its reviewers better
recognize innovation by better defining these terms and recognizing that
innovation can occur in different forms. Dr. Matthews concluded by
saying that CSRAC members are very interested in seeing this initiative
advance. Change may be slow but it will be worth the effort.

**Involvement of Reviewers in Regulatory Issues**

Dr. Cheryl Corsaro, SRA, CSR Mammalian Genetics Study Section, explained
how reviewers who are recruited for their scientific expertise are now
asked to focus more on regulatory issues.

[The Protection of Human Subjects]{.underline}: Reviewers previously
were only asked to comment on the protection of human subjects when a
problem was noticed. Now, they are asked to comment on this issue in
their critiques and study section discussions whenever an application
involves human subjects. In addition, for clinical trials, they now must
also comment on the adequacy of data and safety-monitoring plans.

[Gender, Minorities, and Children]{.underline}: Reviewers are asked to
comment on how well these applications address the inclusion of women,
minorities, and children. If a Phase-III clinical trial is proposed,
reviewers now must assess the adequacy of the analysis plans for
producing significant results related to gender and minority status.

[Vertebrate Animals]{.underline}: Reviewers previously were only asked
to comment on the use of vertebrate animals if a problem was noticed.
They now are asked to comment on (1) the description of the proposed use
of these animals; (2) the justification for using them; (3) the
veterinary care; (4) the limitations on their discomfort, distress,
pain, and injury; and (5) the means by which they are euthanized.

Dr. Corsaro then discussed options for enabling reviewers to manage
these responsibilities. CSR currently provides documents to reviewers
that cover the many details. SRAs also usually send reviewers a memo
and/or e-mail reminding them of these responsibilities. Some SRAs
provide their reviewers a critique template in hard copy and/or
electronically that includes headers for each of these areas. In
addition, SRAs usually raise these responsibilities again when they
begin their study section meetings. Dr. Corsaro continued by focusing on
alternative approaches for helping reviewers deal with these issues.
SRAs could provide a more detailed template or checklist, although
reviewers could think NIH is putting too much emphasis on regulatory
issues vs. scientific issues. Another alternative would be to recruit
one to four \"regulatory consultants\" per study section to address
these issues. Dr. Corsaro explained that she successfully used such
consultants when she coordinated a review of applications for a Request
for Applications to improve informed consent in clinical research. She
also discussed the advantages of using this approach: (1) regular
reviewers would be relieved of these responsibilities, (2) a more
consistent approach to evaluating these areas could result, (3) these
regulatory consultants could be easier to recruit since they would not
have to write detailed scientific reviews. She then focused on the
disadvantages: (1) assigned reviewers may make little effort to consider
these regulatory issues, (2) it may be difficult to review these issues
out of the context of the science, and (3) study sections that review 80
to 90 applications involving human or other vertebrate subjects would
need five to six regulatory consultants. Dr. Corsaro presented a final
option and noted that

Dr. Colvin had earlier proposed this as a means of overcoming these
disadvantages. That option was to use both regulatory consultants and
regular reviewers to assess these issues. She concluded by asking CSRAC
members for their comments and suggestions.

Dr. Matthews asked if the proposed regulatory consultants would be
legal, considering that the primary reviewers are required to address
the regulatory issues in their critiques. Dr. Corsaro said she thought
the requirement was an NIH requirement and not a legal requirement.

Dr. Ehrenfeld said that CSR would examine the regulatory language
closely to be sure.

Dr. Colvin noted that most universities are keenly aware of these
regulations, and it would be best for reviewers to evaluate applications
for both their scientific content and regulatory compliance. Noting the
burden these regulations cause for reviewers \-- particularly those who
review clinical research proposals \-- he said he favored Dr. Corsaro\'s
proposals. He added that he thought it important to keep the proposed
evaluation template simple. Dr. Pugh said he thought a template or
checklist could be particularly helpful in evaluating regulatory issues
related to animal research; however, he said he was strongly opposed to
having individuals other than the scientific reviewers evaluate
applications for regulatory compliance. He noted that the review of
clinical research was more problematic, and he suggested that reviewers
could contact their SRA when they encounter a problem. The SRA could
then find someone to prepare an analysis before the meeting. Dr. Leon
said he agreed with Dr. Pugh. He added that he thought it would be
difficult to recruit regulatory consultants to serve a 4-year term.
Agreeing that there was still a need for guidance, he suggested that
someone at NIH be made available to field questions about the
regulations. Dr. Colvin agreed that there was a greater need for
guidance on clinical research issues, and suggested that recruiting the
proposed consultants would not be too difficult.

Dr. Matthews ask for more information on review burden. Dr. Corsaro said
that most of her reviewers did what was asked in writing their
critiques; however, many of them objected to the required meeting
discussions, particularly when no regulatory problems were found in an
application. Reviewers felt there was not much gained by doing so. Dr.
Matthews then summarized members\' comments: there was considerable
enthusiasm for the detailed checklist that was proposed. There was,
however, less enthusiasm for using regulatory consultants. The primary
reviewers should continue evaluating the science and the liability or
benefits the proposed studies could have for human or animal subjects.
It will be important to monitor the situation to see if this approach
becomes a real burden for reviewers.

**Study Section Reorganization**

Dr. Donald Schneider, Director, CSR Division of Molecular and Cellular
Mechanisms, provided an overview of CSR\'s reorganization activities.
After CSRAC accepted the report of the Panel on Scientific Boundaries
for Review (PSBR) in 2000, CSR began developing Study Section Boundaries
(SSB) teams for each proposed IRG. These teams consist primarily of
external scientist and have worked to develop the guidelines for the
proposed IRGs. Dr. Schneider explained teams recently drafted guidelines
for three new IRGs: Musculoskeletal, Oral and Skin Sciences (MOSS);
Oncological Sciences (ONC); and Biology of Development and Aging (BDA).
These guidelines have since been posted on CSR\'s Web site for a 90-day
public comment period. CSRAC members were asked to review these
guidelines to be sure they

\(1\) represent a cohesive area of science consistent with the PSBR
recommendations, (2) have clearly defined areas of shared interests with
other IRGs, (3) delineate study sections that are neither too broad nor
too narrow, (4) integrate basic science in its biological context as
much as possible, and (5) cluster like applications.

*Musculoskeletal, Oral and Skin Sciences IRG*

Dr. Michael Martin, Director, CSR Division of Physiological Systems,
discussed the proposed guidelines for MOSS, which will cover bone,
muscle, and the connective tissue to the skin that surrounds them. CSR
received five main comments during the public-comment period:

\(1\) Bone mineralization should be clustered and not split between
oral/dental sciences and bone sciences. (2) Immunology and infections of
the oral cavity should be clustered in the MOSS IRG. (3) Single-organ
aging research should be reviewed in the organ-based IRGs, including
muscle, exercise, osteoporosis, and arthritis aging research. Aging or
geriatric research that looks at multiple systems should be reviewed
together in the BDA IRG. (4) Musculoskeletal exercise and rehabilitation
represents too broad an area of science to be reviewed together in one
study section. (5) Systemic aspects of rheumatoid disorders should be
reviewed in a single study section with other aspects of arthritis.

Dr. Martin explained how these suggestions were integrated into the
guidelines presented to CSRAC. Representatives of the rehabilitation
sciences community and NIH program staff strongly supported the
suggestion for splitting musculoskeletal exercise and rehabilitation
sciences. A sixth study section was thus added to the five proposed by
the SSB team that drafted the MOSS guidelines: (1) Oral, Dental and
Craniofacial Sciences (ODCS); (2) Skeletal Biology and Diseases 1; (3)
Skeletal Biology and Diseases 2; (4) Skeletal Muscle Biology and
Exercise Physiology; (5) Musculoskeletal Rehabilitation Sciences; and
(6) Arthritis, Connective Tissue, and Skin. Dr. Martin said that the
level of shared interests within the IRG was at a manageable level. He
then said that the proposed study sections would offer multiple homes
for many applications, and a fair number of basic science applications
in the area of biochemistry could find homes in these study sections. He
concluded by saying the guidelines are largely consistent with the
original intent and scope proposed by PSBR. The guidelines cluster
according to organ system, and both basic and clinical sciences are
accommodated. Concerns about the overlap between Biology of Development
and Aging IRG and the Immunology IRG were resolved. CSR worked with the
American College on Rheumatology and the Alliance on Aging to resolve
these issues.

Two consultants were invited to the meeting to comment on the MOSS
guidelines: Dr. Tachi Yamada, Chairman, Research and Development,
SmithKline and Beecham Pharmaceuticals; and Dr. Adele Boskey, Professor
of Biochemistry and Cell and Molecular Biology, Weill Medical College of
Cornell University. Dr. Yamada said that the proposed guidelines were
quite logical and appropriate. He predicted that there would be few
problems since the boundaries are not harsh, and MOSS does not claim
ownership of all autoimmune diseases or all aspects of arthritis.

Dr. Boskey said that the guidelines were perhaps as good as they could
be. She continued by saying that members of the community need to be
educated about these new guidelines. They need to know that the study
sections in these areas have not changed drastically, and they should
know that they can request that their application be sent to a
particular study section. Dr. Boskey suggested that the 398 application
form be modified to highlight this option. She agreed with

Dr. Yamada that the guidelines work better now that many of the
\"shoulds\" have been changed to \"coulds.\" Drs. Boskey and Yamada also
agreed that CSR should attempt to present the guidelines in a more
simplified format so that applicants can easily understand them. Dr.
Boskey suggested that CSR do this by developing a Web-based
organizational diagram of its study sections.

*Oncological Sciences IRG*

Dr. Elliot Postow, Director, CSR Division of Clinical and
Population-Based Studies, explained that 44 scientists from the
extramural community and 7 NIH staff members served on the ONC SSB team.
They proposed 13 study sections, which were modified following the
public comment period: (1) Cancer Etiology, (2) Cancer Genetics, (3)
Tumor Cell Biology 1,

\(4\) Tumor Cell Biology 2, (5) Tumor Microenvironment, (6) Tumor
Progression and Metastasis, (7) Chemo/Dietary Prevention, (8) Diagnostic
Oncology, (9) Radiation Therapeutics and Biology, (10) Cancer
Immunopathology and Immunotherapy, (11) Drug Discovery and Molecular
Pharmacology, (12) Developmental Therapeutics, and (13) Clinical
Oncology.

Dr. Postow summarized the public comments. Many individuals asked for
very focused study sections for their area of research, while PSBR
counseled against creating entitlements for limited areas. A highly
organized group lobbied for the inclusion of supportive care and
end-of-life, which are areas covered by other IRGs. There was little
agreement where viral oncology should be located, so related boundaries
were made flexible. A number of individuals questioned different
boundaries, which in many instances were made clearer. Dr. Postow,
however, noted that more adjustments will be needed due to changes in
the volume of applications. He then discussed the principles used in
developing and revising the ONC guidelines. The first principle is a
study section must have the scientific ability to judge the significance
of its applications. He noted, however, that there are both individual
and global exceptions. The second principle is the people who understand
what an application informs should review it. He discussed how
applications for research in the areas of extracellular matrix,
cell-signaling, and apoptosis should be reviewed in an ONC study section
when they are related to cancer development or pathogenesis. Otherwise,
these applications should be reviewed in other IRGs. Dr. Postow then
focused on a number of exceptions. All applications related to vector
development, bio-informatics, bio-imaging, and risk prevention, are best
reviewed with other applications in these areas because of their
uniqueness.

In summarizing the final revisions of the ONC guidelines, Dr. Postow
said that (1) a paragraph was added to explain that lung tumors could
not be addressed until the lung boundaries team met, (2) the study
sections were reordered with a more logical development from etiology to
clinical research, (3) the Cancer Immunotherapy Study Section was
renamed as the Cancer Immunopathology and Immunotherapy Study Section,
(4) additional discussions were included on the influence of dietary
factors, (5) clinical trials research for the treatment of
therapy-related illnesses was added, (6) age-related factors were added,
and (7) many overlap statements were modified for consistency. He
concluded by saying he thought all 13 study sections were consistent
with PSBR recommendations, and the public comments were addressed.

Dr. Colvin complemented Dr. Postow and the SSB team for the "fantastic"
work they have done. He noted that the creation of a clinical oncology
study section previously had a remarkable effect on the field, and he
felt that some of the proposed ONC study sections may have the same kind
of effect on their respective fields. Dr. Colvin acknowledged that
adjustments would be made as the process evolved, and he reiterated his
support for the work being done.

Dr. Soybel said he liked how well the ONC guidelines follow PSBR
recommendations. He also emphasized the importance of keeping a fluid
structure that provides multiple homes for applications. Dr. Soybel
continued by noting that it is difficult to see where applications to
treat specific cancers, such as colon or lung cancers, would be
reviewed. The community will need to be educated as to why the ONC study
sections are the best venues for reviewing oncology research. Dr.
Ehrenfeld said that the oncology sciences community made a conscious
choice to structure these study sections around scientific areas versus
organ-specific cancers. She asked Dr. Soybel if he thought this would
have been a better approach. Dr. Soybel said he did not have any data to
suggest which approach would be better. He just emphasized how important
it is for applicants to know there is a study section where their
application can be reviewed fairly by their peers.

Dr. Yamada noted that the mechanisms of carcinogenesis or basic
pathophysiology of cancer is an area that overlaps those covered by
other IRGs. He then questioned whether cancer diagnostics should be
regarded as a specific area of oncology, and he suggested that the
corresponding ONC study section be reevaluated. Dr. Yamada then
questioned the notion that a given kind of cancer belongs in one study
section. For instance, gastric cancer is now associated with the
*Helicobacter pylori* infection, so it is of interest to experts in
multiple areas. He continued by saying cancer is not a defined thing but
a continuum, and understanding where cancer starts and organ system
disease begins is not easy. Dr. Yamada emphasized the importance of
letting applicants decide which group is the appropriate peer group for
their applications.

Dr. Matthews began a series of questions by asking why research on
behavioral interventions was not included in this IRG. Dr. Postow said
that they were not considered because they are so different from the
other research reviewed by this IRG. Dr. Matthews said she knows
researchers who combine dietary interventions with other behavioral
interventions, and she suggested that ONC would be an appropriate venue
for their research. She then noted that this IRG was a very large one
and asked if the boundaries team had considered dividing it into two
IRGs. Dr. Postow said that, toward the end of its deliberations, the
issue was raised to and rejected by the SSB Team. Dr. Matthews continued
her questions by asking if ONC would be a "captive" IRG, where most of
its applications are assigned to one NIH Institute. Dr. Postow said that
most of its applications would be assigned to NCI, and he added that,
despite the fact that PSBR cautioned against creating captive study
sections, the PSBR outline of IRGs encouraged the development of more
captive study sections than currently exist. Drs. Postow and Ehrenfeld
said that CSR still aspires to creating more diverse study sections. Dr.
Matthews concluded by summarizing the discussions. She said that there
is enthusiasm for the proposed IRG, but that CSR should reassess the
name for the Diagnostic Oncology Study Section.

# Biology of Development and Aging IRG

Dr. Schneider explained that BDA was proposed in response to public
comments on the draft PSBR report. He explained that PSBR subsequently
envisioned this IRG as one that included a broad collection of science
related to the biology of development and aging. The BDA SSB team
included 17 community members and representatives from five NIH
Institutes. He explained how this team proposed that BDA comprise four
study sections: (1) Development 1, (2) Development 2, (3) Cellular
Mechanisms in Aging and Development, and (4) Aging Systems and
Geriatrics. The first two study sections would review virtually the same
areas of development, although the first one would emphasize
gametogenesis or genogenesis while the second one would emphasize
pattern formation and early development.

In discussing public comments on the draft BDA guidelines, Dr. Schneider
said CSR had received a fair number of very positive remarks. Most of
the negative comments had to do with a fear that BDA would attempt to
review applications that have been appropriately reviewed in other study
sections. Many of these comments came from behavioral scientists,
epidemiologists, and neuroscientists as well as reproductive biologists.
Dr. Schneider explained that their fears were not really justified,
since most of their research would not be incorporated into BDA. He
noted that there was some overlap in reproductive science between BDA
and the Endocrinology, Metabolism, Nutrition and Reproductive Sciences
(EMNR) IRG, but that this area would not be split between the two IRGs.
For the most part, reproductive sciences \-- particularly endocrine
studies \-- will stay in EMNR while comparative development, early
development, and animal development studies will move to BDA. Dr.
Schneider then noted concerns that the development study sections would
be too aggressive in taking stem cell applications. He said that BDA
would not be the exclusive home for such applications. A few community
members suggested that CSR pull the development applications out of the
Cellular Mechanisms and Aging Development Study Section to create
another aging study section. Dr. Schneider said that development is the
intellectual "glue" that holds the proposed study section together, and
the inclusion of this area of science could have very positive
consequences. A few community members said that it was too bad that
Alzheimer's research and late-life dementias would not be reviewed in
the same IRGs. BDA's Aging Systems and Geriatrics Study Section will
review the latter while a neuroscience study section will continue to
review Alzheimer's research.

Dr. Schneider said that applicants would be encouraged to "vote with
their feet" and decide the degree to which these applications will be
reviewed together. Another concern raised about this study section was
its breadth. A representative from the Aging Institute responded by
designing a mock study section roster that could work using 27 reviewers
provided that some of the outlying applications were reviewed elsewhere.
Dr. Schneider acknowledged that BDA had a few shared interests with MOSS
and many more with EMNR. He concluded his presentation by saying he was
confident that the communities involved could sort out the differences.

Dr. Wickens focused his comments on how applicants will understand the
new guidelines and know which study section might be best for them. He
suggested a system where an applicant could enter keywords for their
research and receive a list of the related study sections.

Dr. Matthews noted that those from the behavioral and epidemiologic
communities would like BDA study sections to have reviewers with their
methodological expertise. She then emphasized the importance of
coordinating the development of the shared interest statements between
BDA and EMNR. Over all, she found the BDA study sections as designed
sensible and looks forward to seeing how they develop. Dr. Boskey said
she would like to know more about the overlap between BDA and MOSS;
however, she said the overlap section in the BDA guidelines was well
written and could serve as a model.

Dr. Matthews acknowledged that the BDA study sections could be expanded
as the EMNR guidelines are finalized and workload increases. She then
suggested that CSRAC accept the proposed IRG guidelines for BDA, ONC,
and MOSS with the understanding that CSR will reexamine the ONC
diagnostics study section and revisit the language used in the overlap
statements. Dr. Colvin moved that CSRAC do this, and the rest of the
committee agreed.

## 

## Mock Study Section Video

Dr. Matthews said CSRAC has had a long-term interest in a video that
would make the review process more transparent and could be used to help
train new reviewers. Dr. Michael Sayre, SRA, CSR Cell Development and
Function 1 Study Section, said that the mock study section video also
could train staff and inform NIH stakeholders and the public. CSR is
developing this mock study section video in collaboration with Dr. Chuck
Selden, Extramural Staff Training Officer at the NIH Office of
Extramural Research (OER). The mock study section was taped during two
days in July 2002. Instead of actors, the SRA was played by a real SRA:
Dr. Sharon Gubanich, with the ONC IRG. Real reviewers were also
recruited, including Dr. Dean Brenner who is chair of the Clinical
Oncology Study Section. Dr. Selden modified/disguised some real grant
applications and summary statements for three different mechanisms (R01,
R03, and K08), and the reviewers discussed them as they would in a
regular study section meeting. Drs. Sayre and Selden are working with a
film producer to reduce the 4 hours of tape to a 20-30 minute video.
CSRAC members were then shown an 11-minute clip from one of the reviews
to be included in the video.

Dr. Selden said that the completed video will include interviews with
two applicants: one talks about the experience of submitting a
successful application and the other talks about the experience of
submitting an unsuccessful one. A narrator will provide a brief
introduction to the grant review process, CSR, and the reviewers, as
well as the roles played by the SRA and study section chair. The
narrator also will discuss the review criteria. The video will then
present clips from the reviews of three different types of applications.

Dr. Matthews and Dr. Lucia Rothman-Denes said they found the clip
presented to be a little too negative and suggested that the final video
will need to be more balanced. Dr. Selden said that was their intention,
and Dr. Sayre added that the other two reviews to be included in the
video are more positive. Dr. Rothman-Denes suggested that the narrator
should mention how long it usually takes to review a grant, so
applicants will not think that their applications are dispatched in five
minutes. She concluded her remarks by saying she thought the completed
video would be a superb tool for helping new applicants understand the
review process, and it will be very well received by the community. Dr.
Pugh agreed with her comments and also suggested that the positive and
negative elements in the clip need to be more balanced. Dr. Leon said
that the video would be particularly helpful in training graduate
students and postdocs. Dr. Schachman said he was excited about the
video, and he emphasized the importance of highlighting a review
discussion that shows the enthusiasm reviewers have when they discuss a
very good proposal and experience the thrill of learning something new.
Dr. Matthews summarized the discussions by suggesting that the video be
modified to correct a disconnection between elements in the video that
could be confusing. On the whole, however, she said there was great
enthusiasm for the video and CSRAC members would be pleased to see the
video again as it is developed.

**Fellowship Study Sections and Outreach**

Dr. Richard Rodewald, SRA, F05 Fellowship Study Section, provided an
update on CSR\'s 12 fellowship study sections. These review groups were
developed to improve the review of fellowship applications, which not
long ago were usually reviewed in the regular study sections. He said
that the new fellowship study sections have completed three review
rounds, and they appear to be working well. A recent survey of the
fellowship SRAs reveals they are highly satisfied and proud of their
study sections. Dr. Rodewald added that a town meeting will be held in
October 2002 to collect feedback from Institute and Center staff and
others. He continued by saying that CSR reviews a little more than 80
percent of all NIH fellowship applications, and

90 percent of those reviewed by CSR are sent to the 12 fellowship study
sections. The consistency of review has improved significantly, and the
study sections are stabilizing, with

50 percent of the reviewers returning for the study section meetings.
Dr. Rodewald said that this was a reasonable turnover, given the
round-to-round variability of the applications submitted.

One problem afflicting these applications is that 15 percent of them
come with format or other problems that require follow-up by CSR staff
to correct. Dr. Rodewald said increasing CSR\'s outreach to this group
may not only help with these problems but also enhance the quality of
these applications. Postdocs have only a brief window of opportunity in
which to submit fellowship applications that can lead to funding, and
they often make needless and costly mistakes in preparing their
applications. He noted that applicants appreciate any mentoring they
receive, and such outreach will likely improve their later R01
applications. In reviewing current outreach efforts, Dr. Rodewald said
that CSR responds to inquiries from applicants, presents information via
meeting booths and displays, participates in outreach presentations and
workshops, and provides Web-based materials. All of these approaches
could be enhanced, but the most cost-effective one may be providing
information through the Web. CSR will thus work with Institute and
Center staff, OER, and other groups to develop a very informative and
engaging site to reach the postdoc community. Dr. Rodewald provided a
draft list of the kinds of information the proposed Web site could
provide, and then asked CSRAC members for their comments and
suggestions.

Dr. Williams said that he was enthusiastic about the plans to strengthen
CSR Web-based outreach to the postdoc community. He asked about the time
frame for this effort.

Dr. Rodewald said he hoped it could be implemented within a year, noting
the importance of soliciting feedback as the Web site is developed. Dr.
Williams then asked about the most common compliance problem with
fellowship applications. Dr. Suzanne Fisher, Director, CSR Division of
Receipt and Referral, said that using too small a font was the most
common problem. Contributing to this problem is the fact that fellowship
applications have a 10-page limit, which is lower than the limit for
other applications. OER is developing a new fellowship application form.
CSR has asked that the applicant\'s list of cited literature no longer
be counted against the 10-page limit. Dr. Williams concluded his
questions by asking if there should be a mechanism to ensure that there
was a core group of reviewers on each fellowship study section. Dr.
Rodewald said that he tries to maintain a core group of reviewers on his
fellowship study section, but he must recruit many temporary reviewers
because the scientific areas that need to be covered for a given set of
applications received vary greatly from round to round.

Dr. Wickens asked about the main goal of the proposed Web site. Dr.
Rodewald said that it would be designed to help applicants avoid common
errors. He added that sponsors often make the mistake of not providing a
strong statement as to the specific training their fellows will receive.
Dr. Colvin said that this was a tremendously important initiative for
the next generation of scientists. Dr. Leon suggested that CSR consider
posting a well-written fellowship application so that new applicants can
see the kinds of things and level of detail they should include in their
applications.

Dr. Matthews asked about the varying workloads of the different
fellowship study sections.

Dr. Rodewald explained that some of the study sections also review other
grants, and the SRAs who coordinate some of the smaller study sections
are responsible for other study sections.

Dr. Matthews also asked about the 12 percent of fellowship applications
that are reviewed by other Institutes and Centers (ICs). If CSR\'s
fellowship study sections continue to work well, the ICs may want to
move their fellowship applications to CSR. Dr. Rodewald said that a
certain percentage of IC-review applications could move to CSR if its
fellowship study sections continue to do well. Dr. Ehrenfeld discussed
the fellowship reviews for the behavioral and social sciences areas.
This community felt strongly that their fellowship applications should
be reviewed in the context of the scientific focus of the regular study
sections, and CSR decided not to create fellowship study sections for
these applications. She asked when CSR should reexamine this decision
and consider moving them into their own study sections. Dr. Matthews
asked if there were data on the rate of funding for these applications
vs. those reviewed in the fellowship study sections. Dr. Stanfield said
that CSR would collect the data and investigate this issue further.

**Neuroscience Evaluation**

Dr. Karl Malik, Acting Director, CSR Office of Evaluation, Planning and
Analysis, explained how CSR\'s neuroscience, AIDS, and behavioral and
social science study sections were reorganized after Congress passed
Public Law 103-321 in 1992, which moved the research components of the
Alcohol, Drug Abuse and Mental Health Administration to NIH. After
consultations with the affected NIH Institutes and the scientific
communities, CSR implemented a comprehensive reorganization of these
study sections with the goal of ensuring quality peer review that
identifies the most meritorious science for each institute to consider
for funding.

Dr. Malik then focused on the three new neuroscience IRGs that began
reviewing applications in 21 study section in 1998: Brain Disorders and
Clinical Neuroscience; Integrative, Functional, and Cognitive
Neuroscience; and Molecular, Cellular, and Developmental Neuroscience.
Three years later, CSR began a process to evaluate these IRGs and their
component study sections by surveying the 1,758 researchers who
submitted R01 applications to them between June 1 and November 1, 2000.
Applicants were asked to indicate their satisfaction with the review of
their applications and to indicate how well they thought reviewers
understood their applications. Those who had submitted applications
before the reorganization were also asked to assess how reviews may have
changed. Another survey was given to the 103 members of NIH program
staff who oversaw these applications to assess their views on the
reorganization.

Dr. Malik said that the applicants had a compliance rate of 85 percent
while program staff had a compliance rate of 74 percent. Applicants were
asked to indicate their satisfaction with five features of the review:
(1) assignment to a study section, (2) expertise of the reviewers,

\(3\) reviewers\' understanding of the research, (4) usefulness of
reviewer comments, and (5) time to receive the summary statement, as
well as their overall satisfaction. On all these points, they most
commonly indicated that they were completely or mostly satisfied. Dr.
Malik noted that satisfaction correlated with the funding status of the
applications. In regards to perceived changes, applicants commonly gave
similar ratings when assessing these five features before and after the
reorganization. Program staff were asked to indicate their satisfaction
with five reviewer characteristics: (1) scientific breadth, (2) depth of
scientific expertise,

\(3\) representation of all relevant disciplines, (4) demographic
diversity, and (5) ability to accommodate emerging areas of research.
All of these features were rated very highly, though the last one
received somewhat lower ratings than the others. As for perceived
changes, program staff tended to rate these characteristics the same or
better after the reorganization. A little over 25 percent of program
staff, however, indicated that scientific breadth was worse after the
reorganization. As for the quality of review, 95 percent of program
staff indicated that the new neuroscience study sections are between
good and excellent in identifying the most meritorious applications. In
summarizing the findings, Dr. Malik emphasized that the data analyzed to
date has not identified any areas where applicants or program staff are
mostly dissatisfied. Areas where applicant or program staff ratings are
relatively low represent concerns that transcend the neuroscience study
sections \-- reviewer comments and the review of new-investigator
applications. He noted that the CSR is already working to address these
concerns.

In discussing next steps, Dr. Malik said CSR will examine the applicant
survey data more closely to assess the responses from new investigators,
identify themes expressed in the open-ended comments section, and
determine if there are disaffected subgroups. Program staff survey data
and comments will be examined by IC. CSR will also attempt to synthesize
an overall evaluation by combining survey data, Working Group reports,
and other internal analyses.

Dr. Williams complemented CSR on the work done to evaluate these study
sections and good response rates that were achieved. He then asked if
there were any comparable data that would help put these results into
context. Dr. Malik said that NIH conducted a satisfaction survey for all
its study sections in 1997, and data from this survey will be used to
further analyze the neuroscience survey data. Dr. Williams concluded his
comments by asking if CSR would look at individual comments or other
data to shed light on areas where program staff responses were less
positive. Dr. Malik said that he hopes such questions could be addressed
as a more detailed analysis is completed.

Dr. Ehrenfeld said that she was excited about the fact that the first
major reorganization of CSR study sections appears to have occurred
without harming the review process. Dr. Pugh said he shared her
excitement, saying that the satisfaction level recorded was remarkable.
He was particularly impressed with the satisfaction ratings of unfunded
applicants. CSR appears to have done its job and done it well, and the
survey should serve as a model for future evaluations.

Dr. Pugh suggested that CSR consider other measures of success besides
satisfaction, such as the numbers of grants submitted and how
efficiently they were processed.

Dr. Stanfield said that CSR plans to use additional measures of success
from the Working Group reports and internal data. He then noted that
this survey was expensive, and it consumed a lot of staff time to
conduct this survey. It would be prudent to consider the extent to which
this kind of analysis should be conducted for the other reorganized
IRGs. Dr. Stanfield explained that this survey was funded with money NIH
sets aside for evaluations, but CSR cannot expect to receive these funds
on an ongoing basis. Dr. Ehrenfeld said that she had delayed conducting
similar surveys for the other reorganized IRGs because she wanted to
make a cost-benefit analysis.

Dr. Matthews asked CSRAC members to comment on the value of future
surveys. Dr. Soybel said that it will be important to let stakeholders
know that the reorganization of their IRGs did no harm and that certain
parts of the process may have been improved. Such validation would be
particularly important for the larger reorganizations, while it might
not be as important for the smaller reorganizations. Dr. Pugh agreed,
emphasizing the value of developing strong sets of data to support
subsequent steps in the reorganization. Dr. Matthews asked if anyone
wanted to argue to the contrary. When no one did, she said that CSRAC
members were very enthusiastic about continuing these surveys.

**Training of Study Section Reviewers and Chairs**

Dr. Sherry Dupere, SRA, CSR Cell Development and Function 5 Study
Section, named the members of an ad hoc committee established to advance
the training of study section reviewers and chairs: Dr. John Bishop,
SRA, CSR Integrative, Functional & Cognitive Neuroscience 5 Study
Section; Dr. Paul Strudler, SRA, CSR Radiation Study Section; and Dr.
Malik**.** Dr. Leon also worked to facilitate the work of this
committee. The goal is to identify effective approaches to training new
reviewers and chairs in the essential elements of NIH peer review. Dr.
Dupere said new reviewers and chairs need essential information about
the peer review process to be effective. She explained that different
kinds of information are needed at different times. She added that
different training mechanisms can be used to convey needed information,
such as documents, phone calls, e-mails, a hotline, the proposed video,
and review-meeting presentations. Dr. Dupere discussed how these
mechanisms fall into categories of training vehicles: (1)
didactic/reference vehicles \-- paper and online manuals and guidelines,
best practices, and quick-reference guides; (2) paradigm vehicles \--
tutorials, videos, and demonstrations, as well as models and sample
products; and (3) resolution/reinforcement vehicles \-- helpful hints,
hotlines, point persons, and frequently asked questions. In developing
the appropriate information, it is important that CSR does not
overburden reviewers and chairs, insult their intelligence by over
simplifying things, or overlook existing materials. Dr. Dupere said that
CSR would value CSRAC input in identifying effective materials and
mechanisms.

The training committee has developed a list of recommended vehicles for
training reviewers:

\(1\) a quick-reference guide with Web links, (2) a training progression
plan that incorporates policy updates, (3) a comprehensive overview, and
(4) a larger document based on the one that already exists which
contains all the essential pieces of information that would be used in
or linked to these other training vehicles. Dr. Dupere continued by
describing how her committee has drafted a quick-reference guide that is
being developed to address information needs during the four stages for
reviewer instruction: (1) during reviewer recruitment, (2) after review
materials are mailed, (3) before critiques are prepared, and (4) at the
review meeting. The third section of this guide was presented to CSRAC
to illustrate the plain language formatting and hyperlinking that will
be used to provide the needed information quickly.

With the assistance of Dr. Leon, the training committee has also begun
to develop sample critiques for showing reviewers what is expected of
them. These samples will illustrate the \"ideal\" length, tone, and
format that should be used. Dr. Dupere explained that samples could be
developed for each grant mechanism and would include annotations to
reinforce suggestions and directives provided in other training
materials as well as include separate sections for references and other
reminders. Copies of an annotated critique were provided along with a
related list of reminders and references. After discussing these
materials, Dr. Dupere explained how the training committee has also
updated the guidelines for study section chairs. The document has been
formatted in plain language and revised to include appropriate
hyperlinks and to cover new practices and policies such as electronic
review. Dr. Dupere concluded her presentation with a request for
feedback from CSRAC members.

Dr. Colvin said that he enjoyed the presentation and emphasized how
serving on a study section can be a very educational experience for new
investigators as well as more senior investigators. New investigators
also can learn a lot from the SRAs. He added that he thought the
proposed reviewer instructions on assessing innovation would be
particularly useful. Dr. Matthews noted that these instructions stated
that a proposal could be meritorious even if it were not innovative, and
she asked if reviewers could be confused and give an application a bad
rating. Dr. Dupere said that it is important that reviewers weight the
criteria for each proposal accordingly. She emphasized the importance of
a proposal's impact. Dr. Matthews noted that there are sometimes
"sleeper effects" in research, and it is often difficult to truly assess
impact beforehand.

Dr. Soybel mentioned that the neuorscience survey showed that 25-30
percent of the NIH program staff thought that their study sections did
not adequately address innovation in their reviews. He then focused on
how best to develop reviewer skills and proposed that CSR invite
first-time reviewers to come a day before their study section meeting to
participate in an orientation session. Dr. Dupere said that this was an
excellent suggestion. Dr. Matthews asked if such a program would be
feasible. Dr. Colvin said that it may be difficult for a young clinician
to take the extra time off from the clinic. Dr. Soybel said that it
could be managed, and he suggested that those who receive their first
R01 grant could be invited to such an orientation session. Dr. Dupere
said this proposal was worth exploring.

## Optimizing the Utility of Streamlining Applications

Dr. Richard Panniers, SRA, CSR Physiological Chemistry Study Section,
briefly discussed the benefits and disadvantages of streamlining the
review of applications that reviewers believe do not fall in the upper
half of those they are asked to consider. The practice was introduced as
a means to allow reviewers to focus discussion where there are critical
differences of opinion. The disadvantages frequently mentioned focus on
the limited comments that are given to the applicants. Dr. Panniers
continued by discussing the current standards of streamlining. A list of
applications proposed for streamlining is presented to reviewers before
the meeting. If a reviewer believes a certain application does not
belong on the list, it is removed at the meeting and discussed in its
regular order. Similarly, an application can be added to the list at the
meeting if all the reviewers agree.

Dr. Panniers explained that a few SRAs use additional or alternative
practices to increase the utility of streamlining: (1) using Web scores
before the meeting to facilitate the development of the streamlining
list, (2) soliciting nominations for streamlining at the meeting instead
of developing a list beforehand, (3) holding brief discussions of
streamlined applications either at the beginning of the meeting or as
the applications come up in the review, (4) discussing briefly a few of
these applications in the regular course of the meeting when members
feel it is desirable, or (5) allowing private discussions to reconcile
differences over applications proposed for streamlining. Dr. Panniers
added that electronic review now lets reviewers read each others\'
critiques and assess differences ahead of time. It is thus changing the
nature of streamlining. In focusing on how electronic review may be used
to optimize streamlining, he noted that SRAs vary in how much time they
give reviewers to submit their comments before "the curtain is lifted"
and they can see the comments submitted by others. Allowing reviewers to
read the critiques ahead of time will improve their confidence in the
streamlining process and the precision of their streamlining decisions.
Dr. Panniers asked the committee for advice on how much time should be
given to submit and view critiques, how reviewers can be encouraged to
read the other critiques, and whether reviewers should be allowed to
change their reviews before their meeting.

Dr. Panniers noted that one of the main concerns about streamlining is
that it can result in unclear critiques. He explained how he reviewed
the summary statements for 200 streamlined applications, which included
465 critiques. He found seven positive critiques that were out of sync
with the streamlining decision, and he suggested that the reviewers
could have written addenda. In reading all the critiques, he found that
5 percent lacked an overall evaluation, which can make a big difference
in understanding why an application was streamlined. Possible solutions
include (1) providing initial Web scores to the applicants, although
these scores could be confusing and reviewers do not usually provide
scores when streamlining is recommended, (2) giving reviewers a
checklist, although the richness of the critiques could be diminished
and the reviewers would be burdened by yet more paper, (3) ensuring that
critiques are completed with a brief, overall evaluation section, and
(4) having SRAs or reviewers provide an addendum to clarify unclear
critiques. Another concern about streamlining has to do with presenting
an application for streamlining while a reviewer who is in conflict with
it is in the room.

Dr. Panniers said that this should not be a problem as long as reviewers
do not discuss the particular merits of the application. In concluding
his discussions of concerns, he talked about the stigma associated with
streamlining. Unscored applications are presently given the label
\"UN.\" They may be better than many applications that receive scores
above the 50th percentile. Applications are streamlined because there is
consensus at the outset that they are not competitive. To remove the
label and level the playing field, all applications in the lower half
(scored or unscored) can simply be marked as \"\>50 percentile.\"

In summary, Dr. Panniers listed the questions that remain to be
answered: How can reviewers be encouraged to take advantage of the
electronic review system and read critiques ahead of time? What is the
minimum number of days needed for reviewers to read these critiques?
Should reviewers be allowed to amend their critiques before the meeting
to reflect changes in their thinking? Should CSR pursue a checklist for
reviewers? If there is no discussion of merit, can reviewers with
conflicts remain in the room? Should private discussions about
applications to be streamlined be permitted before the meeting? Should
CSR provide preliminary Web scores or mark both scored and unscored
applications above the 50^th^ percentile as \"\>50 percent.\"

Dr. Rothman-Denes said it was crucial that reviewers be allowed to amend
their critiques when they conflict with streamlining decisions. However,
she was concerned about letting reviewers change their critiques before
their meetings, noting that younger reviewers could be intimidated by
the reviews of senior members and inappropriately alter a good review
too soon.

Dr. Rothman-Denes also said that she liked the idea of unscoring
applications that would fall at or above the 50^th^ percentile, and she
thought a checklist could be useful for new reviewers.

Dr. Soybel agreed with her on the value of marking the lower 50th
percent of applications as \>50 percent and the usefulness of a
checklist for new reviewers. However, he said that he also did not favor
allowing reviewers the opportunity to alter their critiques before the
meeting. If clarification is needed to explain the streamlining
decision, it should be explained in an addendum added at or after the
meeting. Dr. Wickens gave his support for using an addendum when an
explanation is needed to clarify a decision to streamline an
application. He said he did not like the proposed checklist, which would
have to be complicated to cover all the important nuances. Dr. Matthews
agreed, saying that the written review should contain all the valuable
details. A checklist would be redundant.

After saying she like the idea of using addenda, Dr. Matthews said she
favored using electronic review to generate preliminary scores that
could be used to determine the 50 percent line and develop a
streamlining list. Giving these scores to the applicants could also
diminish the trauma associated with streamlining and let applicants know
how close they may be to producing a successful application. Dr. Pugh
also supported the idea of using preliminary scores to determine the
50^th^ percentile instead of pushing reviewers to identify those above
it. He then emphasized the importance of written critiques. Though he
did not oppose the development of a checklist, he said that it should
not be a substitute for a well-written critique. Dr. Williams gave his
support for using preliminary scores to determine what applications
should be streamlined and to help applicants know how close their
applications are to the cut-off line. He also endorsed the use of
addenda. Dr. Stanfield noted that reviewers who nominate applications
for streamlining usually do not score them. Dr. Leon emphasized the
usefulness of having reviewers provide scores for streamlined
applications and fine-tuning them if necessary.

Dr. Matthews summarized the sentiments of CSRAC members. Members agreed
that addenda should be developed after the review meeting to explain
what happened when a reviewer who wrote a positive critique decided that
an application should be streamlined. A fair number of members supported
the use of scores to identify and place those applications at or above
the

50 percentile. She said that an additional round of CSRAC discussions
would be useful to address more fully the issues and questions posed.

## 

**Agenda Items for the Next Meeting**

Dr. Matthews listed possible agenda items for the next CSRAC meeting:
(1) the definition of innovation, (2) ways to better inform customers
about CSR's reorganization, (3) the thoughts of the new NIH Director,
(4) future evaluations of newly reorganized IRGs, (5) new investigator
issues, (6) the role of K awards, (7) the review of behavioral sciences
fellowships, (8) the retention of principle investigators, (9) the
feasibility of shortening the review cycle,

\(10\) reviewer workload, and (11) the review of any new IRG guidelines.
She then thanked CSR staff and CSRAC members for a good meeting. Dr.
Ehrenfeld added her thanks, saying she was very please and proud of the
work CSRAC and her staff have done.

With no other business to address, CSRAC adjourned the meeting at 12:03
p.m.

We do hereby certify that, to the best of our knowledge, the foregoing
minutes of the 30th meeting of CSRAC are accurate and complete. The
minutes will be considered at the 31st meeting of the Advisory
Committee, and any corrections or comments will be made at that meeting.

> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
>
> Brent Stanfield, Ph.D.
>
> Executive Secretary
>
> Center for Scientific Review Advisory Committee
>
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
>
> Ellie Ehrenfeld, Ph.D.
>
> Director
>
> Center for Scientific Review
>
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
>
> Karen Matthews, Ph.D.
>
> Chair
>
> Center for Scientific Review Advisory Committee