Committee of Visitors (COV) Report for the National Science Foundation
(NSF) Graduate Research Fellowship Program (GRFP)

June 17-18, 2003

Committee Members:

Sally Frost Mason, Purdue University (Chair)

Dereje Agonafer, University of Texas, Arlington

Richard S. Givens, University of Kansas

Joseph Jones, Texas Southern University

Margaret Petrochenkov, National Research Council

Amy Cheng Vollmer, Swarthmore College

##### FY 2003 REPORT TEMPLATE FOR

#####  NSF COMMITTEES OF VISITORS (COVs)

+-----------------------------------------------------------------------+
| ##### Date of COV: June 17-18, 2003                                   |
+-----------------------------------------------------------------------+
| **Program/Cluster: NSF GRFP**                                         |
+-----------------------------------------------------------------------+
| **Division: DGE**                                                     |
+-----------------------------------------------------------------------+
| **Directorate: EHR**                                                  |
+-----------------------------------------------------------------------+
| **Number of actions reviewed by COV: Awards: 36 Declinations: 33**    |
|                                                                       |
| **Other: 36 Honorable Mention (HM)**                                  |
+-----------------------------------------------------------------------+
| **Total number of actions within Program/Cluster/Division during      |
| period being reviewed by COV: 2000 Awards: 850 Declinations: 2836     |
| Other: 946 HM**                                                       |
|                                                                       |
| **2001 903 3560 1097 HM**                                             |
|                                                                       |
| **2002 900 4377 1278 HM**                                             |
|                                                                       |
| **2003 900 4933 1955 HM**                                             |
+-----------------------------------------------------------------------+
| **Manner in which reviewed actions were selected: By consensus of COV |
| panel discussions.**                                                  |
+-----------------------------------------------------------------------+

Table of Contents

Report narrative.................................................... 3

Introduction ................................................ 3

The application process .................................. 4

The application ................................... . 5

Letters of reference ............................... 7

The review process ....................................... 9

Panel reviewer selection .......................... 9

Panel selection and preparation ...............\... 10

Program management .................................... 12

Other recommendations .................................. 13

Stipend levels ...................................... 13

Award expansion ................................. 13

Enhancing access ................................. 13

Other mentoring issues........................... 15

Interactions with other NSF programs ......... 15

The educational component ..................... 16

Executive summary of recommendations...................... 17

Core questions and report template responses ................. 20

Appendix A ....................................................... 36
Appendix B ....................................................... 38

Appendix C ....................................................... 39

**Introduction**

The Committee of Visitors for the NSF Graduate Research Fellowship
Program (GRFP) met on June 17 and 18, 2003. Committee members were:
Sally Mason (Chair) from Purdue University, Amy Cheng Vollmer from
Swarthmore College, Dereje Agonafer from the University of Texas
Arlington, Margaret Petrochenkov from the National Research Council,
Richard Givens from the University of Kansas, and Joseph Jones from
Texas Southern University. Five of the six members of this committee had
served previously on GRFP review panels, two members had chaired GRFP
review panels, and one committee member had extensive experience with
similar types of review panels for other funding organizations. After a
brief orientation from NSF staff members, including James Lightbourne,
Wynn Jennings, Elmima Johnson, Allan Headley and Eric Sheppard,
discussion turned to program-specific questions posed by the GPRA.
Additionally, we addressed four points of consideration raised by Wynn
Jennings during our orientation: (1) quality maintenance; (2) indicators
and predictors of success during proposal evaluation; (3) the rising
number of applications (10,000+); and (4) how to motivate students from
underrepresented groups to apply.

Allan Headley provided an overview of the changes made to the program
since the last COV report in 1999. These included: (1) expansion of
those eligible to apply as a result of an interruption or change in
graduate training; (2) the ability of fellowship holders to accept other
fellowships in addition to the NSF GRF; (3) elevation of the stipend
level to \$27,500 per annum; and (4) the fact that this is the first COV
review since the minority graduate fellowship programs were folded into
the GRFP. The COV review in 1999 was conducted too soon after the
elimination of these programs to assess impact. Eric Sheppard followed
up with a presentation that included very useful and recent longitudinal
data on the application and success rate of minority and women GRF
applicants.

From the outset, we were impressed by the levels of responsiveness of
the NSF, the NSF staff who responsibly manage this program, and the ORAU
who manage the review process to suggestions from previous COVs. *Our
suggestions are not meant to imply criticism or a sense that the program
is not well managed or well run.* To the contrary, *all of our
observations suggest that this is a healthy and thriving program that
deserves the support of all interested constituents*, from Congress to
fellowship recipients. Our suggestions are intended to provide
fine-tuning and enhancement opportunities, not all of which may be
feasible, practical, or affordable. They are offered in the spirit of
extremely positive support for a program that is truly a national
treasure in the scientific community.

The rate of compensation paid to fellowship holders now places the NSF
GRF in a strategically prestigious position. Only one other federal
fellowship program compensates students as handsomely, which will
unquestionably make the NSF fellowships highly desirable to students and
institutions of higher education. We believe that with these increased
stipends comes increased accountability and enhanced expectations. This
belief is commonly threaded throughout the body of this report and is
explicitly stated at appropriate points in the text.

**The Application Process**

The COV believes that the enhanced stipend associated with GRFs warrants
a much closer scrutiny of expectations of both the applicant and the
host institution. Consequently, we suggest a number of adjustments to
the application, review, and reporting processes.

**The Application:** A review of the application, including the student
application form and the supporting documents for each application, led
to several suggestions that we believe would strengthen a reviewer's
ability to evaluate students. Additionally, we were intent upon
addressing issues that might help reviewers identify both the best and
brightest students, as well as those who demonstrate leadership
potential. The issue of leadership potential was raised by NSF staff
members and reinforced as a worthy goal for this program by all of the
COV members.

Over the years the application itself has been greatly improved by
providing more opportunities for students to demonstrate how their
interest in science has evolved, what "greater impact" a fellowship
might have on their career, and what research project they intend to
pursue as graduate students. After extensive discussion, we determined
that these components of the proposal are extraordinarily useful to
reviewers but could be strengthened in several ways by more explicit
instructions to applicants. We urge the NSF staff to consult with those
who have expertise in survey construction to develop a series of "bullet
points" for items 15 and 16 on the application form and on the
corresponding instruction page in order encourage students to address
issues relative to (a) potential leadership and (b) past accomplishments
in research. Strong wording suggesting that students who fail to address
the "bullet points" will likely not rank as highly as those who do is
also encouraged. \[We provide a more detailed example of suggested
changes to the application form in Appendix A of this document.\]

Regarding leadership, students should be encouraged to describe examples
and ideas that are not necessarily confined to the science, mathematics
or engineering disciplines they have chosen to pursue. The goal is to
assess leadership potential by attempting to determine whether a student
demonstrates qualities such as persistence, drive, determination, risk
taking, reflection, introspection, curiosity, ingenuity and any other
indicators of potential success as a future leader in whatever career
they choose to pursue.

Regarding past accomplishments, we would encourage instructions for
students to cite explicitly any publications, abstracts, research
awards, inventions, or conference presentations that they have completed
or have in progress. The COV did not feel that it had the expertise to
develop the appropriate wording for these sections of the application,
but we did feel strongly that this would greatly improve a reviewer's
ability to assess the potential of an applicant. \[Refer, again, to
Appendix A.\]

Concomitantly, the COV felt that three changes could be made to the
research proposal portion of the application that would strengthen the
reviewer's ability to assess quality. First, ask explicitly for a title
for the research proposal, including key words,

and an hypothesis that is to be developed by the proposed research.
Second, request the inclusion of appropriate literature citations within
the body of the proposal for those portions of the narrative that relate
to background literature upon which the hypothesis is based. In so
doing, the applicant is to be encouraged to state the uniqueness and
creativity of the proposed research. Third, a simple outline of the
basic elements of a research proposal might be provided to help students
construct a cogent and consistent narrative. And finally, students
should be instructed to assure the NSF that all narrative portions of
the application, including the research portion, are exclusively their
own. Therefore, a "statement of integrity" attesting to the original
nature of the student's research proposal should be included within the
research narrative.

The COV believes that it might be useful for the NSF to develop a
statement regarding the ethics of scientific discourse including
proposal writing. Student applicants should be provided with this
statement, and research advisors who write in support of applicants
should also be asked to provide a statement of the role that they have
played in the development of the research proposal.

**Letters of Reference:** The COV feels that letters of reference are
critical to assessment of a student's ability. We had several
suggestions for improving the utility of the letters of reference.
First, we strongly urge the NSF to reconsider using the "quantitative"
rankings of candidates on the reference form. There is no norm for this
type of ranking, and savvy recommenders know that they must check the
top 1-2% line in order for their candidate to be competitive. We suggest
eliminating the "check box" item that ranks students and replacing it
with an explicit request that letter writers address how this student
compares with others that they have known well, and whether this
statement is based on the student's quantitative and creative
performance in class, in the laboratory, or both. Second, we urge the
NSF to develop a "fill-in" form for reference writers that asks the
writer to provide examples and comment on the potential of the applicant
to (a) succeed in graduate education, (b) conduct original research, (c)
communicate effectively, (d) function in potential leadership roles, (e)
work cooperatively with peers and supervisors, and (f) make unique
contributions to their chosen discipline and to society in general.
Finally, letter writers should be provided a space to comment on any
unique or special talents or attributes that might distinguish this
student from others.

Letter writers should also be instructed that a "generic" letter of
reference for a student applicant would place the student at a
disadvantage among peers. If letter writers choose not to use the form,
they should be instructed to follow the recommended format closely.
Moreover, the letter writers should be provided with the narrative
portion of the student's application, and each writer should be urged to
read these narratives prior to writing the recommendation.

As stated earlier, research advisors who write letters in support of
their student's GRF application should also be asked to sign a
"statement of integrity" attesting to the original nature of the
student's research proposal. They should comment explicitly on the role
they played, if any, in assisting the student with the proposal.

While we acknowledge that this places an additional burden upon both
applicants and letter writers, recognition and prestige associated with
the GRF awards, especially the level of funding, warrants a more
rigorous application and evaluation process. We are aware, however, that
this may discourage some prospective applicants who are not willing to
exercise meticulous attention to detail in the application process. The
effect may be a reduction in the rapid growth in numbers of applications
recently experienced by the GRFP. In light of the added responsibility
to the letter writers, the NSF could include a narrative urging the
applicant to give letter writers sufficient time to prepare their
recommendations.

**The Review Process**

The COV panel members, to a person, were uniformly impressed by the
level of professionalism, consistency, and timeliness provided by the
ORAU and NSF in what is an enormously complex and preparation-intensive
review process. The experience of the COV members as panel reviewers
and/or panel chairs contributed to several suggestions that might
improve upon this already well-run and well-organized process.

**Panel Reviewer Selection:** The complexity of constructing the number
of panels required to review more than 9,000 proposals is daunting. And
yet, the ORAU has done an exemplary job of creating and maintaining
diverse representation on the panels. The importance of diversity of the
reviewers with regard to gender, ethnicity, and type of institution
cannot be overstated. We encourage a continued high level of dedication
and effort by the staff as they strive to maintain broadly
representative review panels.

We have no doubt that a diverse panel sends a strong message to all
panel members about the value of diversity in the academy. COV members
suggested that their own experiences were positively influenced by the
willingness of panel members from underrepresented groups to present
unique and sometimes personal viewpoints on candidate's qualifications.

Leadership is an additional criterion that might be used in the reviewer
selection process. Those scientists and engineers who have strong
research credentials and who have served in academic or professional
leadership roles may be able to help identify student applicants who
possess potential leadership qualities.

A final observation that appears at variance when reviewing the panel
composition warrants more attention. In some areas (notably biological
and physical sciences), it appeared that the very institutions that
would most likely be the beneficiaries of GRFs were least well
represented on these panels. While there may be legitimate reasons that
are not apparent to the COV members, it is reasonable to expect a higher
level of representation on the panel from the twenty or thirty
institutions that most benefit from the GRF awards. We do not suggest
this be done in lieu of maintaining diversity on the panel but in
addition to this. \[See Appendix B for a suggestion on potential
reorganization of the life sciences panels given that the Howard Hughes
Medical Institute program has now been terminated.\]

**Panel Selection and Preparation:** Serving as a GRF reviewer is a
personally rewarding experience and at the same time it involves a
relatively modest commitment of time. There is no massive amount of
preparation for panel meetings; the bulk of the work is accomplished in
an intensive two-day meeting. Furthermore, the service benefits the
panel members in several ways. For example, faculty learn to be both
better mentors and better letter writers for applicants by participating
in the review process.

By far the most effective means of communicating this very positive
message about the multiple benefits of serving on a GRFP panel is by
word of mouth from those who have served previously. All panel members
should be asked to encourage their colleagues to disseminate information
about the program and to provide recommendations to the ORAU for
potential panel members.

Members of the COV felt strongly that the calibration exercise, while
time consuming, was an essential component of preparing panel members to
be effective and responsible reviewers. Panel leaders should require
that all panel members to adopt the common practice of reading the first
four or five applications before assigning a score. This is especially
important for new panel members who may be unfamiliar with the sometimes
narrow range of high quality proposals that they might read. Panel
leaders should also be instructed on how to deal proactively with
"rogue" panel members who can, if not monitored, skew the results on the
applications that they review. Panel leaders must be able to counsel a
panel member who is reluctant or unable to adapt to the norms and ranges
of the review process. The ORAU lead panel assistant must be prepared to
offer counsel and assistance to the panel leader. A caucus of all panel
chairs and lead panel assistants on the first day (during the lunch
hour) might help to identify panels with difficult members and suggest
strategies. This may be of particular help to new panel chairs.

Because we have placed new emphasis on attempting to determine
leadership potential of applicants, we suggest that instructions
describing several criteria for evaluating leadership potential be
presented to panel members in the orientation session. They should be
directed to read the statements written by each candidate wherein these
issues are discussed directly (currently items 15 and 16 in the
application form) and to evaluate the strength of the applicant's
response to the newly worded request for information. With the GRE
scores now an optional item on the application, reviewers need to be
ever vigilant not to disproportionately disadvantage applicants without
GRE scores over those with GRE scores.

Finally, the COV felt that adjustments to feedback given to applicants
might be appropriate. Specifically, we suggest that a list of common
critiques of proposals be provided as check boxes, along with a discrete
amount of space for written comments by the panel reviewers. \[See
Appendix C.\]

**Program Management**

The level of organization and support provided by the ORAU and the NSF
is truly outstanding. Points relevant to the application process and to
program management that were addressed in the last COV report were taken
very seriously by the staff at both ORAU and the NSF. The COV was
especially impressed with results presented by Eric Sheppard of efforts
to increase the diversity of the applicant pool and awardees. His
efforts at increasing outreach have been tremendously successful, and
the results (more minority fellowship winners) cannot be overstated.

When the minority fellowship programs were folded into the GRFP, many
feared that minority applicants might not be effectively recruited,
represented, or evaluated. In the years immediately following the merger
of these programs, the numbers of minority awardees were indeed
alarmingly small. In subsequent years, thanks to the efforts of NSF
staff, significant increases in fellowships awarded to these groups have
occurred. We applaud these efforts and encourage expansion and
continuation of efforts to increase access to underrepresented groups to
the GRFP. We offer some suggestions for enhancing these efforts later in
this report.

We also commend the ORAU and NSF for their efforts to provide the names
of panelists to all participants in the review process prior to the
panel meetings. This has greatly facilitated networking opportunities
and a sense of common purpose across panels.

# Other Recommendations

**Stipend Levels:** The GRF award stipend is currently \$27,500 per
annum. This is a substantial sum for a graduate fellowship; indeed, few
other graduate fellowships come close to this amount of compensation.
One unintended consequence of a stipend at this level is the pressure
that this may place on university budgets since few universities are
capable of matching or even approaching a match of this sum for graduate
stipends. Thus, we recommend that before any further increases in
stipend level are considered, the NSF thoroughly examine the external
consequences of the \$27,500 stipend and what intended and unintended
consequences this might engender. Our recommendation is that stipend
levels remain at \$27,500 until the next COV review.

**Award Expansion:** At the same time, we encourage the NSF to consider
two possibilities for expansion of the program. The first is to increase
the number of awardees by 100 for each of the next three years, so that
there will be 1200 fellows in 2006. Second, we recommend considering a
second level award, to those students primarily in QG2 who are
themselves highly meritorious. This would be a one-time award of
\$3000-5000 as a true "honorable mention." An award winner who declines
the GRF might also be eligible for this category of support. The funds
should be available to the student to use over several years to support
professional development such as travel to a conference or research
site, membership in a professional society, subscriptions to
professional publications, or to purchase educational materials.

**Enhancing Access and Raising the Level of Awareness, Particularly
Among Underrepresented Groups:** Our recommendations place additional
responsibilities on the shoulders of the applicants to do even more work
in applicant proposal preparation. We do not wish to see this limit the
ability of students from small liberal arts schools, minority-serving
institutions (MSIs), or less research intensive institutions of higher
education placed at a disadvantage in the application process. We
suggest several ways in which this might be mitigated.

First, given the level of the stipends provided to fellowship holders,
the NSF might consider including an option of one year (preferably in
the second or final year of their award) of service for each current
fellowship holder as a mentor to aspiring fellows. The fellow's name
would be entered into a database of potential mentors, along with their
discipline, home institution, email address, and perhaps a brief
biography. As long as their name is active in the database, they would
be available to potential applicants to answer questions and help mentor
those who are preparing an application. We would especially encourage
the NSF staff to provide this mentoring service to students from
underrepresented groups or who are at institutions that may not have the
infrastructure to support significant research.

Second, the NSF staff should consider offering or helping to support
grant-writing workshops at national conferences or regionally across the
country, and by webcasting, so that aspiring applicants can have access
to help in the preparation of an application. Some of this activity
already occurs in professional societies or on campuses and could be
used as a model for the NSF. This should be a mandatory activity for
institutions that can provide students from underrepresented groups and
is perhaps where the NSF should initially focus its efforts to see
whether such an activity is feasible and useful. If grant-writing
workshops are initiated, an applicant might be asked to note on their
application whether they were the beneficiaries of such an activity.

Finally, the COV discussed the merits of finding ways to encourage
institutions to mentor students to be successful applicants. One
suggestion is that for baccalaureate-granting schools who are successful
for the first time, there might be a modest monetary award given from
the NSF. At the very least, a letter or plaque of congratulations signed
by the appropriate NSF officials could be sent to acknowledge and
encourage this achievement.

**Other Mentoring Issues:** In addition to our suggestion of using
fellowship holders as grant-writing mentors for aspiring applicants, we
discussed the importance of mentoring in general. Mentoring is
especially critical for students from underrepresented groups, and in
its outreach efforts, NSF can focus especially on the role that faculty
can play in this process. Efforts to develop web-based or face-to-face
meetings with NSF staff and faculty on selected campuses for the
purposes of training and encouraging mentors is something that might be
seriously considered. This should be coupled with ongoing efforts to
continue improving the number of quality applicants from
underrepresented groups.

**Interactions with Other NSF Programs:** A point raised during the last
COV review and one that we would strongly reinforce is the potential for
interaction and activity especially between the GRFP and the REU
program. The overlapping missions and goals of these two programs should
naturally draw them closer together both functionally and
administratively. It is apparent that undergraduate research experiences
are often a critical and deciding factor in the success of future
scientists and engineers. The large, institutionally based REU programs
provide fertile ground for potential GRF awardees. As part of a summer
REU experience, the directors of these programs might be apprised that
the NSF would like to spend a few hours one afternoon orienting students
to the opportunities for the GRFP. This might involve a staff member
visiting an ongoing REU program, either in person or by webcast, or a
local faculty volunteer who would be willing to present such a program.

Similarly, the large center grants from the various directorates of the
NSF often have an outreach and/or undergraduate education component
associated with funding. One activity that could be of benefit to groups
of undergraduates involved in these research projects is again an
educational seminar on the GRFP, as described above. There are likely
other programs across the NSF where similar interactions can be fostered
of which we are unaware.

We hope that better ways of interfacing between the various NSF
directorates can be found in order to facilitate access and
encouragement of students to the broad variety of programs available
through the NSF, but especially to the GRFP.

**The Educational Component:** This COV felt strongly that we should not
lose sight of the responsibility of the NSF to train responsible
scientists, citizens, and educators. We were especially concerned that
GRF winners need to understand the importance of education, teaching,
and community service. The tendency to focus exclusively on research to
the exclusion of all other activities in graduate education at some
research universities does not serve these outstanding students well in
the long run. They must continue to hone their communication skills,
and, if they plan a career in education, they must begin to understand
learning, pedagogy, and teaching. Faculty at research-intensive
universities should be advised to encourage fellowship holders to teach,
to engage in community service, and to pursue careers that might be
considered nontraditional in the eyes of many traditional faculty. The
end result may help to create a more scientifically literate and better
informed core of individuals who pursue leadership both within and
outside of academia. The NSF should be proactive in exploring ways to
provide incentives to both student award recipients and faculty mentors
to pursue these worthy goals.

**Executive Summary of Recommendations**

As mentioned throughout this document, the COV felt strongly that the
handsome stipend awarded to GRF fellows increases the responsibilities
of the applicants and the subsequent fellowship holders. The panel feels
that these additional responsibilities should include mentoring
potential applicants, testimonial support, and encouraging and advising
new applicants, especially with the goal of increasing the number of
applicants from underrepresented groups for the GRF.

Our specific recommendations are summarized as follows:

# Application Process

-   Reword items 15 and 16 on the application form to request evidence
    of the applicant\'s leadership potential and to provide reviewers
    with useful evidence of past experience in research.

-   Provide a more structured format for the research proposal that
    would include the title, key words, hypothesis, research plan,
    literature citations, and, most importantly, a statement of
    integrity attesting to the originality of the applicant's proposal.

-   Require research advisors to provide a similar statement of the
    integrity attesting to the originality of the applicant's proposal.

-   Provide a structured format for the reference letters, and eliminate
    the "check box" rating of candidates with regard to other students
    the referee has known. Referees should be asked to provide this
    information in narrative form by comparing the applicant with former
    successful applicants for the GRF fellowships and/or other
    outstanding students.

-   Provide explicit instructions to the referees about the importance
    that the letters play in the panel's ranking of the GRF candidates.

# Review Process

-   Above all, continue the strong emphasis on diversity of the panels.

-   When possible, encourage more participation on the panels of faculty
    from schools who receive large numbers of the GRF awardees.

-   Develop well-grounded criteria for assessing leadership in an
    application.

-   Expand panel preparation exercises by including advice on how to
    ascertain leadership potential from the application materials and
    insist on delaying the assignment of scores for applications until a
    panelist has read the first four or five.

# Program Management

-   The NSF management was praised for the success in increasing
    minority applicants and awardees.

-   Overall program management by the NSF and ORAU should be continued
    at its current outstanding level.

# Other Recommendations

-   Freeze the current stipend level at \$27,500 for the next three
    review periods.

-   Increase the number of awards by 100 per year for the next three
    review periods to reach a goal of 1200 awards per year.

-   Initiate a new type of award for applicants ranked in the QG2 and
    Honorable Mention categories that will provide \$3-5,000 one time
    awards to be used for travel to professional meetings, membership in
    professional organizations, journal subscriptions, or for purchase
    of educational materials.

-   Increase access and awareness of the GRF program through contacts
    with various organizations that represent MSIs.

-   Recognize baccalaureate institutions with first-time successful
    applicants.

-   Improve the awareness of the GRF program in general and maintain
    access to the program from underrepresented groups by providing GRF
    fellow mentors to prospective applicants and grant-writing workshops
    when and wherever possible.

-   Encourage more intra-directorate communication and activity,
    especially between the REU and GRF program directors and with other
    NSF programs that impact or overlap with the mission and goals of
    the GRF program.

-   Provide incentives and support for the GRF fellows to participate in
    service and teaching opportunities, as well as encouraging their
    mentors to appreciate the value of having talented students who
    might choose to pursue more nontraditional, nonacademic, or
    professional career trajectories.

-   Identify persons at MSIs willing to serve as mentors for students
    seeking GRFs.

-   Develop a communication scheme for organizing and informing mentors
    about cogent aspects of the GRF application wherein mentors can
    facilitate students in the process of writing quality applications.

-   Increase the interface with a diversity of organizations that can
    assist in informing students, especially minorities, about GRF as
    well as other appropriate NSF programs.

-   Seek ways of encouraging inter-institutional collaboration between
    GRF recipient institutions and MSIs that train GRF awardees.

**PART A. INTEGRITY AND EFFICIENCY OF THE PROGRAM'S PROCESSES AND
MANAGEMENT**

Briefly discuss and provide comments for *each* relevant aspect of the
program\'s review process and management. Comments should be based on a
review of proposal actions (awards, declinations, and withdrawals) that
were *completed within the past three fiscal years*. Provide comments
for *each* program being reviewed and for those questions that are
relevant to the program under review. Quantitative information may be
required for some questions. Constructive comments noting areas in need
of improvement are encouraged. Please do not take time to answer
questions if they do not apply to the program.

A.1 Questions about the quality and effectiveness of the program's use
of merit review procedures. **Provide comments in the space below the
question**. **Discuss areas of concern in the space provided.**

+-----------------------------------------------------------+----------+
| **QUALITY AND EFFECTIVENESS OF MERIT REVIEW PROCEDURES**  | **YES,   |
|                                                           | NO,**    |
|                                                           |          |
|                                                           | **DATA   |
|                                                           | NOT      |
|                                                           | AV       |
|                                                           | AILABLE, |
|                                                           | or NOT   |
|                                                           | APPL     |
|                                                           | ICABLE** |
+-----------------------------------------------------------+----------+
| Is the review mechanism appropriate? (panels, ad hoc      | Yes      |
| reviews, site visits)                                     |          |
|                                                           |          |
| Comments: See section entitled **The Review Process** in  |          |
| the narrative.                                            |          |
+-----------------------------------------------------------+----------+
| Is the review process efficient and effective?            | Yes      |
|                                                           |          |
| Comments: See section entitled **The Review Process** in  |          |
| the narrative.                                            |          |
+-----------------------------------------------------------+----------+
| Are reviews consistent with priorities and criteria       | Yes      |
| stated in the program's solicitations, announcements, and |          |
| guidelines?                                               |          |
|                                                           |          |
| Comments: See section entitled **The Review Process** in  |          |
| the narrative and **Appendix C**.                         |          |
+-----------------------------------------------------------+----------+
| Do the individual reviews (either mail or panel) provide  | Yes      |
| sufficient information for the principal investigator(s)  |          |
| to understand the basis for the reviewer's                |          |
| recommendation?                                           |          |
|                                                           |          |
| Comments: See section entitled **The Review Process** in  |          |
| the narrative and **Appendix C**.                         |          |
+-----------------------------------------------------------+----------+
| Do the panel summaries provide sufficient information for | Yes      |
| the principal investigator(s) to understand the basis for |          |
| the panel recommendation?                                 |          |
|                                                           |          |
| Comments: See **Appendix C** for our suggestions about    |          |
| how to improve upon providing feedback to the applicants. |          |
+-----------------------------------------------------------+----------+
| Is the documentation for recommendations complete, and    | Yes      |
| does the program officer provide sufficient information   |          |
| and justification for her/his recommendation?             |          |
|                                                           |          |
| Comments: However, the COV felt that a more complete set  |          |
| of feedback needs to be provided particularly to          |          |
| unsuccessful applicants. Specific suggestions can be      |          |
| found in **Appendix C**.                                  |          |
+-----------------------------------------------------------+----------+
| Is the time to decision appropriate?                      | Yes      |
|                                                           |          |
| Comments: Nothing further to comment on.                  |          |
+-----------------------------------------------------------+----------+
| Discuss issues identified by the COV concerning the       |          |
| quality and effectiveness of the program's use of merit   |          |
| review procedures:                                        |          |
|                                                           |          |
| Discussed in general in section entitled **The            |          |
| Application Process** and **The Review Process** of the   |          |
| narrative.                                                |          |
+-----------------------------------------------------------+----------+

A.2 Questions concerning the implementation of the NSF Merit Review
Criteria (intellectual merit and broader impacts) by reviewers and
program officers. **Provide comments in the space below the question.
Discuss issues or concerns in the space provided.**

+----------------------------------------------------------+-----------+
| **IMPLEMENTATION OF NSF MERIT REVIEW CRITERIA**          | **YES,    |
|                                                          | NO,**     |
|                                                          |           |
|                                                          | **DATA    |
|                                                          | NOT       |
|                                                          | A         |
|                                                          | VAILABLE, |
|                                                          | or NOT    |
|                                                          | APP       |
|                                                          | LICABLE** |
+----------------------------------------------------------+-----------+
| Have the individual reviews (either mail or panel)       | Yes       |
| addressed whether the proposal contributes to both merit |           |
| review criteria?                                         |           |
|                                                          |           |
| Comments: No further comments provided.                  |           |
+----------------------------------------------------------+-----------+
| Have the panel summary reviews addressed whether the     | Not       |
| proposal contributes to both merit review criteria?      | A         |
|                                                          | pplicable |
| Comments: The GRFP relies on individual, not panel       |           |
| summary, reviews.                                        |           |
+----------------------------------------------------------+-----------+
| Have the *review analyses* (Form 7s) addressed whether   | Not       |
| the proposal contributes to both merit review criteria?  | A         |
|                                                          | pplicable |
| Comments: The GRFP uses an aggregate recommendation memo |           |
| and not an individual review analysis.                   |           |
+----------------------------------------------------------+-----------+
| Discuss any issues or concerns the COV has identified    |           |
| with respect to NSF's merit review system.               |           |
|                                                          |           |
| The COV agreed that the merit review system seems to be  |           |
| working well. Fellowships are awarded exclusively based  |           |
| on merit and according to the two criteria spelled out   |           |
| in the guidelines. We thus focused on details of the     |           |
| process itself.                                          |           |
+----------------------------------------------------------+-----------+

A.3 Questions concerning the selection of reviewers. Provide comments in
the space below the question. Discuss areas of concern in the space
provided.

+---------------------------------------------------------+------------+
| Selection of Reviewers                                  | YES , NO,  |
|                                                         |            |
|                                                         | DATA NOT   |
|                                                         | AVAILABLE, |
|                                                         |            |
|                                                         | or NOT     |
|                                                         | APPLICABLE |
+---------------------------------------------------------+------------+
| Did the program make use of an adequate number of       | Yes        |
| reviewers for a balanced review?                        |            |
|                                                         |            |
| Comments: See the final paragraph of the section        |            |
| entitled **Panel Reviewer Selection** in the narrative. |            |
+---------------------------------------------------------+------------+
| Did the program make use of reviewers having            | Yes        |
| appropriate expertise and/or qualifications?            |            |
|                                                         |            |
| Comments: No further comment provided.                  |            |
+---------------------------------------------------------+------------+
| Did the program make appropriate use of reviewers to    | Yes        |
| reflect balance among characteristics such as           |            |
| geography, type of institution, and underrepresented    |            |
| groups?                                                 |            |
|                                                         |            |
| Comments: See the section entitled **Panel Reviewer     |            |
| Selection** of the narrative.                           |            |
+---------------------------------------------------------+------------+
| Did the program recognize and resolve conflicts of      | Yes        |
| interest when appropriate?                              |            |
|                                                         |            |
| Comments: The COV members all agreed that ORAU was      |            |
| doing an excellent job of determining potential COIs    |            |
| and effectively managing these. The topic is clearly    |            |
| addressed in the "Guide for Panelists" distributed      |            |
| prior to convening of a panel.                          |            |
+---------------------------------------------------------+------------+
| Discuss any concerns identified that are relevant to    |            |
| selection of reviewers.                                 |            |
|                                                         |            |
| See the section entitled **The Review Process** of the  |            |
| narrative.                                              |            |
+---------------------------------------------------------+------------+

A.4 Questions concerning the resulting portfolio of awards under
review**. Provide comments in the space below the question. Discuss
areas of concern in the space provided.**

+------------------------------------------------------+---------------+
| **RESULTING PORTFOLIO OF AWARDS**                    | **A           |
|                                                      | PPROPRIATE,** |
|                                                      |               |
|                                                      | **NOT         |
|                                                      | A             |
|                                                      | PPROPRIATE,** |
|                                                      |               |
|                                                      | **OR DATA NOT |
|                                                      | AVAILABLE**   |
+------------------------------------------------------+---------------+
| Overall quality of the research and/or education     | Appropriate   |
| projects supported by the program.                   |               |
|                                                      |               |
| Comments: No further comment provided.               |               |
+------------------------------------------------------+---------------+
| Are awards appropriate in size and duration for the  | Appropriate   |
| scope of the projects?                               |               |
|                                                      |               |
| Comments: See sections entitled **Program            |               |
| Management, Stipend Levels,** and **Award            |               |
| Expansion** in the narrative.                        |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| -   High Risk Proposals?                             |               |
|                                                      |               |
| Comments: We broadly interpreted "high risk" to      |               |
| include proposal from individuals at MSIs. Please    |               |
| see section entitled **Enhancing Access and Raising  |               |
| the Level of Awareness, Particularly in              |               |
| Underrepresented Groups** in the narrative. A        |               |
| similar answer (appropriate) could be applied to     |               |
| consideration of the science being proposed. We      |               |
| agreed that determining "risk" in this latter case   |               |
| would be extremely subjective.                       |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| -   Multidisciplinary Proposals?                     |               |
|                                                      |               |
| Comments: No further comment is provided.            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| -   Innovative Proposals?                            |               |
|                                                      |               |
| Comments: No further comment is provided.            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | NA            |
| balance of:                                          |               |
|                                                      |               |
| -   Funding for centers, groups and awards to        |               |
|     individuals?                                     |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | NA            |
| balance of:                                          |               |
|                                                      |               |
| -   Awards to new investigators?                     |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| -   Geographical distribution of Principal           |               |
|     Investigators?                                   |               |
|                                                      |               |
| Comments: We assumed that fellowship holders with    |               |
| PIs in this case.                                    |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| -   Institutional types?                             |               |
|                                                      |               |
| Comments: Again, we assumed that the baccalaureate   |               |
| institution of the fellowship holder was the         |               |
| appropriate entity to consider here.                 |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Yes           |
| balance of:                                          |               |
|                                                      |               |
| -   Projects that integrate research and education?  |               |
|                                                      |               |
| Comments: See comments in the section entitled **The |               |
| Educational Component** in the narrative. We suggest |               |
| encouraging teaching experience for those fellowship |               |
| holders who aspire to jobs in academe in particular. |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance:                                             |               |
|                                                      |               |
| -   Across disciplines and subdisciplines of the     |               |
|     activity and of emerging opportunities?          |               |
|                                                      |               |
| Comments: No further comment is provided.            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have appropriate          | Appropriate   |
| participation of underrepresented groups?            |               |
|                                                      |               |
| Comments: See comments in section entitled           |               |
| **Enhancing Access and Raising the Level of          |               |
| Awareness, Particularly Among Underrepresented       |               |
| Groups** in the narrative.                           |               |
+------------------------------------------------------+---------------+
| Is the program relevant to national priorities,      | Appropriate   |
| agency mission, relevant fields and other customer   |               |
| needs? Include citations of relevant external        |               |
| reports.                                             |               |
|                                                      |               |
| Comments: The program has been a mainstay of         |               |
| technical workforce development in science,          |               |
| engineering, technology and mathematics for many     |               |
| years. It effectively addresses the national needs   |               |
| and priorities for a highly trained scientific       |               |
| workforce, which have been issues raised by policy   |               |
| makers and politicians through many administrations. |               |
| The program is old enough now that former fellowship |               |
| holders can be found in many prominent places within |               |
| the public and private sectors. NSF should not       |               |
| hesitate to use examples of successful "alumni" to   |               |
| further the goals of this program.                   |               |
+------------------------------------------------------+---------------+
| Discuss any concerns identified that are relevant to |               |
| the quality of the projects or the balance of the    |               |
| portfolio.                                           |               |
|                                                      |               |
| Other than thoughts in the section entitled **Other  |               |
| Recommendations** of the narrative and in **Appendix |               |
| A**, which are addressed in the questions above, the |               |
| COV had no additional concerns of note.              |               |
+------------------------------------------------------+---------------+

A.5 Management of the program under review**. Please comment on:**

+-----------------------------------------------------------------------+
| Management of the program.                                            |
|                                                                       |
| Comments: See comments in section entitled **Program Management** of  |
| the narrative.                                                        |
+-----------------------------------------------------------------------+
| Responsiveness of the program to emerging research and education      |
| trends.                                                               |
|                                                                       |
| Comments: The NSF staff have been extraordinarily responsive not just |
| to trends but to suggestions from former COVs and other reviews that  |
| have been done.                                                       |
+-----------------------------------------------------------------------+
| Program planning and prioritization process (internal and external)   |
| that guided the development of the portfolio under review.            |
|                                                                       |
| Comments: While we have threaded some suggestions on program planning |
| and prioritization throughout the text of the narrative, the NSF has  |
| in general done an outstanding job on these issues from this          |
| program's inception.                                                  |
+-----------------------------------------------------------------------+
| Discuss any concerns identified that are relevant to the management   |
| of the program.                                                       |
|                                                                       |
| None identified.                                                      |
+-----------------------------------------------------------------------+

##### PART B. RESULTS : OUTPUTS AND OUTCOMES OF NSF INVESTMENTS

**NSF investments produce results that appear over time. The answers to
questions for this section are to be based on the COV's study of award
results, which are direct and indirect accomplishments of projects
supported by the program. These projects may be currently active or
closed out during the previous three fiscal years. The COV review may
also include consideration of significant impacts and advances that have
developed since the previous COV review and are demonstrably linked to
NSF investments, regardless of when the investments were made.
Incremental progress made on results reported in prior fiscal years may
also be considered.**

The following questions are developed using the NSF outcome goals in the
FY 2003 Performance Plan. The COV should look carefully at and comment
on (1) noteworthy achievements of the year based on NSF awards; (2) the
ways in which funded projects have collectively affected progress toward
NSF's mission and strategic outcomes; and (3) expectations for future
performance based on the current set of awards. NSF asks the COV to
provide comments on the degree to which past investments in research and
education have contributed to NSF's progress towards its annual
strategic outcome goals and to its mission:

-   To promote the progress of science.

-   To advance national health, prosperity, and welfare.

-   To secure the national defense.

-   And for other purposes.

**B. Please provide comments on the activity as it relates to NSF's
Strategic Outcome Goals. Provide examples of outcomes (nuggets) as
appropriate. Examples should reference the NSF award number, the
Principal Investigator(s) names, and their institutions.**

+-----------------------------------------------------------------------+
| **B.1 [NSF OUTCOME GOAL for PEOPLE]{.underline}: Developing "a        |
| diverse, internationally competitive and globally engaged workforce   |
| of scientists, engineers, and well-prepared citizens."**              |
|                                                                       |
| Comments: See comments in section entitled **Enhancing Access and     |
| Raising the Level of Awareness, Particularly Among Underrepresented   |
| Groups** of the narrative. Moreover, the COV reviewed the 2003        |
| nuggets provided to us and was further persuaded that all stated      |
| goals were being met, particularly since each nugget included a       |
| description of the goals. Clearly, the students who are recipients of |
| these awards end up making significant contributions to the           |
| "scientific" work force and as "global" representatives of our        |
| scientifically trained citizenry.                                     |
+-----------------------------------------------------------------------+
| **B.2 [NSF OUTCOME GOAL for IDEAS]{.underline}: Enabling "discovery   |
| across the frontier of science and engineering, connected to          |
| learning, innovation, and service to society."**                      |
|                                                                       |
| Comments: See comments in section entitled **Other Recommendations**  |
| of the narrative, but especially see **Other Mentoring Issues,        |
| Interactions with Other NSF Programs,** and **The Educational         |
| Component**.                                                          |
+-----------------------------------------------------------------------+
| **B.3 [OUTCOME GOAL for TOOLS:]{.underline} Providing "broadly        |
| accessible, state-of-the-art and shared research and education        |
| tools."**                                                             |
|                                                                       |
| Comments: While "tools" are not the goal of this program (people      |
| are), we nevertheless suggest a number of ways in which "tools"       |
| (e.g., webcasting) could be used to enhance the program.              |
+-----------------------------------------------------------------------+

##### PART C. OTHER TOPICS

**C.1 Please comment on any program areas in need of improvement or gaps
(if any) within program areas.**

See section entitled **Executive Summary of Recommendations**

**C.2 Please provide comments as appropriate on the program's
performance in meeting program-specific goals and objectives that are
not covered by the above questions.**

Covered in the narrative.

**C.3 Please identify agency-wide issues that should be addressed by NSF
to help improve the program\'s performance.**

See especially section entitled **Other Recommendations** in the
narrative.

**C.4 Please provide comments on any other issues the COV feels are
relevant.**

Covered in the narrative (**Other Recommendations**).

**C.5 NSF would appreciate your comments on how to improve the COV
review process, format and report template.**

See **Appendices A and C.**

**SIGNATURE BLOCK:**

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

For the NSF GRFP COV

Sally Frost Mason

Chair

### Appendix A

More detailed suggestions from the COV regarding how segment of the
application form might be modified:

[Application form:]{.underline}

**Information form: NSF Form 289**

> \(a\) Include names of people from whom letters were requested, and
> identify actual research mentor, if known.
>
> \(b\) For item 15, specify more clearly what information is being
> requested: e.g., Describe any personal, professional, or education
> experience or situations that have contributed to your desire to
> pursue advanced study in science, mathematics, or engineering.
> Experiences need not be limited to the academic realm.
>
> \(c\) For item 16, again specify more clearly what information is
> requested: e.g., Describe how you would address the following issues
> as you advance your career: mentoring, integrating research and
> education, advancing diversity in science, enhancing scientific and
> technical understanding, benefiting society. Include details from your
> life that demonstrate your persistence, drive, determination, risk
> taking, reflection, introspection, curiosity, ingenuity, and
> leadership.

*Note: the importance of these 2 items might warrant a separate form*

**Proposed Plan of Research, NSF form 306:**

> \(a\) Instructions for this might read: In a clear, concise, and
> original statement, describe research topics you may pursue while on
> fellowship tenure. Please follow the following format:

Title of research project and key words

> Statement of hypothesis or question to be addressed by the research

Strategy, methodology, and controls

Anticipated results, findings

Appropriate literature citations

> Statement of integrity: a statement attesting to the original nature
> of your proposal. If you have shown this plan to an advisor, explain
> his/her role in the formulation of this plan.

**Previous Research Experience, NSF Form 1259:**

> \(a\) Instruction for this form might read: Describe any scientific
> research activities in which you have participated. For each
> experience, describe the purpose of the research, your role, and what
> you gained from each experience. Include:
>
> Experience in undergraduate research programs
>
> Research experience gained through summer or part-time employment or
> in work-study programs
>
> Other research activities, either academic or job-related
>
> Meetings attended and any presentations (authors, title of
> paper/poster, name, month and year of meeting
>
> Published papers (formal citation format, including title of paper)
>
> Papers in review, preparation (formal format, including name of
> journal of record)

(b) Distinguish between undergraduate and graduate research experiences.
    If you have had no direct research experience, describe any
    activities that you believe have prepared you to undertake research.

**Reference Letters, NSF Form 299**

> Suggestions include:

(a) Provide a separate form for the research mentor, which includes a
    statement of integrity that asks them to comment on their
    contribution to the research proposal.

(b) Eliminate the numerical questions about how a student ranks among
    his/her peers.

(c) Provide a form with sections or subheadings, asking for a
    description and/or evidence of the candidate's (1) creativity and
    curiosity, (2) technical skill, (3) knowledge of the broad field and
    specific topic, (4) motivation and emotional maturity, (5) ability
    to work both independently and cooperatively, and (5) potential for
    leadership.

**Appendix B**

Because there are no longer conflicts with the HHMI fellowship program,
the COV suggests that the life sciences panels could be better
reconfigured as follows (these would also be more closely aligned with
graduate programs):

Physiology, neuroscience

Microbiology, molecular biology, cell biology

Genetics, developmental biology

Environmental life sciences

Ecology

Evolutionary biology/genomics

Plant and animal biology

Biochemistry, biophysics, structural biology

**Appendix C**

Suggestions for reviewers comments in order to provide more constructive
and meaningful feedback to applicants, especially those who are not
successful and/or who may wish to reapply:

In order to provide constructive critiques and avoid conveying negative
and sometimes potentially hurtful comments, the COV discussed the
following suggestions for changing the reviewer's comment sheet. It is
the general intent and not the exact wording or formatting of these
evaluative comments that we hope the GRFP staff will take into
consideration. There should still be space allotted for general comments
on the reviewer's forms as well as the suggestions below.

-   Design a form with the following headings in a horizontal row across
    the top of a page, followed by the statements listed below, with
    check boxes in each column:

    -   Strongly agree

    -   Agree

    -   Neutral

    -   Disagree

    -   Strongly disagree

    -   Not possible to evaluate

-   The statements to be checked, aligned in a far right-hand column,
    might include:

    -   Applicant has a strong academic record in coursework appropriate
        to chosen field

    -   Applicant's research proposal has a well defined and well
        supported hypothesis

    -   Proposed research plan is thoughtfully laid out

    -   Applicant shows knowledge and command of the primary literature

    -   Application is scientifically sound and accurate

    -   Application demonstrates creativity/initiative on the part of
        the applicant

    -   Contributions of applicant and research mentor (if identified)
        to the proposed research plan are clearly delineated in the
        "statements of integrity"

    -   Applicant has demonstrated research experience/motivation

    -   Applicant has demonstrated leadership skills, experience,
        potential

    -   Applicant has demonstrated involvement in and commitment to
        teaching and service

    -   Letters of support are consistent in their strong support of the
        applicant's scientific research experience/qualifications

    -   Letters of support are consistent in their strong support of the
        applicant's leadership experience/potential

    -   Application is well presented in a polished form, largely free
        of typographical and grammatical errors, slang and inappropriate
        language use