**[NSB-00-78]{.underline}**

Report to the National Science Board

on the

National Science Foundation's

Merit Review System

Fiscal Year 1999

![](media/image1.png){width="2.279166666666667in"
height="2.279166666666667in"}

# FY 1999 Report on the NSF Merit Review System

####### TABLE OF CONTENTS

HIGHLIGHTS

+---+---------------------------------------------------------------+---+
| 1 | [Proposals and Awards]{.underline}                            | 3 |
| . | \...\...\...\.                                                |   |
|   | ..\...\...\...\...\...\...\...\...\...\...\...\...\...\...\.. | 3 |
|   | .\...\...\...\...\...\...\...\...\...\...\...\...\...\...\... |   |
|   |                                                               | 4 |
|   | Competitively Reviewed Proposals, Awards and Funding Rates    |   |
|   | \...\...\...\...\...\...\...\...\...\...\...\....             | 5 |
|   |                                                               |   |
|   | Characteristics of Principal Investigators                    |   |
|   | \...\...\...\...\...\...\...\...\...                          |   |
|   | \...\...\...\...\...\...\...\...\...\...\...\...\...\...\.... |   |
|   |                                                               |   |
|   | Award Amounts                                                 |   |
|   | \...\...\...\...\...\...\..                                   |   |
|   | .\...\...\...\...\...\...\...\...\...\...\...\...\...\...\... |   |
|   | \...\...\...\...\...\...\...\...\...\...\...\...\...\...\.... |   |
+---+---------------------------------------------------------------+---+
| 2 | [Methods of Proposal Review]{.underline}                      | 6 |
| . | \...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\ |   |
|   | ...\...\...\...\...\...\...\...\...\...\...\...\...\...\..... | 6 |
|   |                                                               |   |
|   | Review Processes Used at NSF                                  | 8 |
|   | \...\...\...\...\...\...\...\...\...\...\...\...\...\...\     |   |
|   | ...\...\...\...\...\...\...\...\...\...\...\...\...\...\..... | 9 |
|   |                                                               |   |
|   | Reviews and Reviewers                                         | 1 |
|   | \...\...\..                                                   | 0 |
|   | .\...\...\...\...\...\...\...\...\...\...\...\...\...\...\... |   |
|   | \...\...\...\...\...\...\...\...\...\...\...\...\...\...\.... | 1 |
|   |                                                               | 1 |
|   | Reviewer Proposal Ratings                                     |   |
|   | \...                                                          |   |
|   | \...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\ |   |
|   | ...\...\...\...\...\...\...\...\...\...\...\...\...\...\..... |   |
|   |                                                               |   |
|   | NSF Program Officers                                          |   |
|   | \...\...\...\..                                               |   |
|   | .\...\...\...\...\...\...\...\...\...\...\...\...\...\...\... |   |
|   | \...\...\...\...\...\...\...\...\...\...\...\...\...\...\.... |   |
|   |                                                               |   |
|   | Assuring Objectivity in the Merit Review Process              |   |
|   | \...\...\...\...\                                             |   |
|   | ...\...\...\...\...\...\...\...\...\...\...\...\...\...\..... |   |
+---+---------------------------------------------------------------+---+
| 3 | [Other Issues Related to Merit Review]{.underline}            | 1 |
| . | \...\...\...\...\...\...\...\...\...\...\...                  | 2 |
|   | \...\...\...\...\...\...\...\...\...\...\...\...\...\...\.... |   |
|   |                                                               | 1 |
|   | Committee of Visitors (COV)                                   | 2 |
|   | \...\...\...\...\...\...\...\...\...\...\...\...\...\...\...  |   |
|   | \...\...\...\...\...\...\...\...\...\...\...\...\...\...\.... | 1 |
|   |                                                               | 3 |
|   | Government Performance and Results Act (GPRA)                 |   |
|   | \...\...\...\..                                               | 1 |
|   | .\...\...\...\...\...\...\...\...\...\...\...\...\...\...\... | 4 |
|   |                                                               |   |
|   | Use of Preliminary Proposals                                  | 1 |
|   | \...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\ | 5 |
|   | ...\...\...\...\...\...\...\...\...\...\...\...\...\...\..... |   |
|   |                                                               | 1 |
|   | Cost Sharing Policy Revision                                  | 5 |
|   | \...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\ |   |
|   | ...\...\...\...\...\...\...\...\...\...\...\...\...\...\..... | 1 |
|   |                                                               | 6 |
|   | Small Grants for Exploratory Research (SGER)                  |   |
|   | \...\...\...\...\...\..                                       | 1 |
|   | .\...\...\...\...\...\...\...\...\...\...\...\...\...\...\... | 7 |
|   |                                                               |   |
|   | Accomplishment Based Renewals and Creativity Extensions       |   |
|   | \...\...\...\...\...\...\...\...\...\...\...\...\...\...      |   |
|   |                                                               |   |
|   | Exemptions to the Merit Review Process                        |   |
|   | \...\...\...\...\...\...\...\...\...                          |   |
|   | \...\...\...\...\...\...\...\...\...\...\...\...\...\...\.... |   |
+---+---------------------------------------------------------------+---+

**FY 1999 Report on the NSF Merit Review System**

## HIGHLIGHTS

  ----- -------------------------------------------------------------------
  1\.   During FY 1999 NSF took action on 28,504 competitively reviewed
        proposals, and provided funding to 9,112 of them. This resulted in
        an overall funding rate of 32 percent. These numbers have changed
        very little during the past five years. In FY 1999, Directorate
        funding rates ranged from 27 percent to 42 percent.

  2\.   The funding rates for proposals from minority Principal
        Investigators (PIs) were below the NSF average in FY 1999, and have
        been for seven of the past eight years. The number of proposals
        received yearly from minority PIs has decreased by 5 percent since
        FY 1992.

  3\.   Since FY 1992, the funding rates for proposals received from female
        PIs and male PIs have been similar. (In FY 1999 it was 31 percent
        for females compared to 32 percent for males.) The number of
        proposals received from female PIs increased by 19 percent during
        that seven year period.

  4\.   There continues to be considerable disparity in the funding rates
        of proposals from "new PIs" and "prior PIs" (23 percent and 39
        percent, respectively, in FY 1999).

  5\.   The average and median award size increased by 3.3 percent and 3.5
        percent respectively. In FY 1999 the average award size for
        research grants was \$92,788 and the median was \$73,335. In NSF's
        FY 2001 Government Performance and Results Act (GPRA) Performance
        Plan, a specific goal is to increase the average annualized award
        size for research projects to \$108,000.

  6\.   Since 1990, the percentage of NSF proposals reviewed by panel-only
        has increased from 36 to 47 percent, while the use of mail-only
        review has decreased from 33 to 18 percent. The use of
        mail-plus-panel review increased from 32 to 35 percent during this
        period. This may reflect a number of factors, such as the growing
        number of multidisciplinary proposals received and the declining
        response rate of mail reviewers.

  7\.   NSF received a total of more than 246,000 reviews in FY 1999, for
        an average of 8.3 reviews per proposal. The response rate to mail
        review requests has decreased to 59 percent from 62 percent in FY
        1998.

  8\.   In FY 1999, over 8,000 proposals with average summary ratings
        between Very Good and Excellent were declined. The judgment of NSF
        staff is essential to making this difficult separation between
        awards and declines. The data also indicate that a large number of
        potentially fundable proposals are declined each year.

  9\.   The use of preliminary proposals has increased in frequency over
        the past several years in NSF programs. In FY 1999, NSF acted on
        1,594 preliminary proposals that were logged into the proposal
        processing system. Of these, NSF encouraged the submission of full
        proposals in 221 cases and discouraged submission in 1,373 cases.
  ----- -------------------------------------------------------------------

**FY 1999 Report on the NSF Merit Review System**

Among the Federal agencies, NSF has a unique mission: to strengthen the
overall health of U.S. science and engineering across a broad and
expanding frontier. NSF invests in the best ideas from the most capable
people, determined by competitive merit review. The merit review system
is at the very heart of NSF\'s selection of the projects through which
its mission is achieved. NSF evaluates proposals for research and
education projects using two criteria: the intellectual merit of the
proposed activity and the broader impacts of the proposed activity on
society.

The *FY 1999 Report on the NSF Merit Review System* responds to a
National Science Board (NSB) policy endorsed in 1977 and amended in
1984, requesting that the Director of the National Science Foundation
(NSF) submit an annual report on the NSF proposal review system. This
report provides summary information about levels of proposal and award
activity and the process by which proposals are reviewed and awarded.

# Proposals and Awards

## Competitively Reviewed Proposals, Awards and Funding Rates

During FY 1999, NSF took action on 28,504 competitive, merit reviewed
research and education proposals, as shown in **Text Figure 1**. The
number of proposals reviewed annually by NSF has been reasonably stable
at around 30,000 proposals since 1995.

NSF funding was awarded to 9,112 of the proposals, resulting in an
overall funding rate of 32 percent. The number of awards made each year
has varied between approximately 9,000 and 10,000. The overall funding
rate has varied little since 1995. However, funding rates among
directorates[^1] varied considerably, ranging from 27 percent to 42
percent as shown in **Appendix Table 1.**

**Text Figure 1**

### NSF Proposal, Award and Funding Rate Trends

Fiscal Year

  ------------------ ---------- ---------- ---------- ---------- ----------
                     **1995**   **1996**   **1997**   **1998**   **1999**

  **Proposals**      30,700     30,231     30,189     28,318     28,504

  **Awards**         9,563      9,071      9,864      9,279      9,112

  **Funding Rate**   31%        30%        33%        33%        32%
  ------------------ ---------- ---------- ---------- ---------- ----------

In addition to funding proposals that were competitively reviewed during
FY 1999, NSF awarded 6,896 continuing grant increments (CGIs) based on
proposals which had been competitively reviewed in earlier years. CGIs
are funded in annual increments from current year appropriations. The
CGI procedure complements the other major type of NSF award instrument
-- standard grants -- where all funds for a multiple year project are
obligated in the initial award. NSF policy limits the amount of next
year's CGI commitments to 65 percent of a program's current fiscal year
operating plan.

## Characteristics of Principal Investigators

Trends in funding rate for all PIs, female and minority PIs, and prior
and new PIs are shown in **Text Figure 2**. Proposals, awards, funding
rates and trends by PI characteristics are presented in **Appendix Table
2**.

The differences in funding rates of proposals from female PIs and male
PIs have been minor over the past few years. However, the funding rates
for proposals from minority PIs have been below the overall NSF funding
rate for seven of the past eight years.

The number of proposals received from female Principal Investigators
(PIs) has increased by 19% since 1992, as shown in Appendix Table 2. The
same has not been true for PIs from minority groups, which has decreased
by 5%. During FY 1999, about 19 percent of competitively reviewed
proposals were from female PIs down from 20 percent in FY 1998, and five
percent were from minority PIs (level with FY 1998.)

![](media/image2.wmf)Forty-one percent of the competitively reviewed
proposals in FY 1999 were from PIs who had never received an NSF award
('new PIs'), up slightly from FY 1998. There continues to be a wide
disparity in the funding rates of \"new PIs\" and \"prior PIs\" (23
percent and 39 percent, respectively in FY 1999). As indicated in
Appendix Table 2, in FY 1999 the number of awards to new PIs decreased
from 3,041 to 2,675, or 12 percent.

In order to encourage the proposal and award process to be open to new
people and new ideas, NSF has established an FY 1999 GPRA performance
goal of 30 percent of competitive research grants going to new
investigators. The FY 1999 result was 27 percent. The FY 2000/01 goals
will continue to be 30%. The Agency is committed to maintaining openness
in the system and will strive to increase the percentage of awards to
new investigators. NSF will explore whether the pool of new
investigators is smaller than in previous years, whether they are
submitting fewer proposals, etc., and use this information to design
future strategies.

[Award Amounts]{.underline}

The median annual award amount (adjusted for multiple year projects)
among competitive research awards made during FY 1999 was \$73,335, a
3.5 percent increase from FY 1998. The average annualized award amount
in FY 1999 was \$92,788, an increase of 3.3 percent from FY 1998. The
difference between the median and average award amounts reflects the
effect of numerous small awards on the median, and a few large awards
for centers, facilities, and large systemic initiatives on the average
award amount.

Award amounts have been consistent over the past decade, when adjusted
to constant dollars as measured by the Consumer Price Index. There are
considerable differences among directorates, as shown in **Text Figure
3**. Data on median and average award amounts from FY 1995-1999 are
presented by directorate in **Appendix Table 3**.

![](media/image3.wmf)Adequate award size is important both to getting
high quality proposals and to ensuring that proposed work can be
accomplished as planned. Larger awards also enable scientists and
engineers to devote a greater portion of their time to research. In
NSF's FY 2001 Government Performance and Results Act (GPRA) Performance
Plan, a specific goal is to increase the average annualized award size
for research projects to \$108,000, and the average award duration from
2.7 to 3 years.

# Methods of Proposal Review

The involvement of knowledgeable peers from outside the Foundation in
the review of proposals is the keystone of NSF's proposal review system.
Their judgements of the extent to which proposals address established
criteria are vital for informing NSF staff and influencing funding
recommendations. For this reason, NSF's system of proposal review can
accurately be characterized as "merit review with peer evaluation."

## Review Processes Used at NSF

NSF programs obtain external peer review by two principal methods, mail
and panel. In addition to mail and panel reviews, site visits by NSF
staff and external peers are often used to review proposals for large
facilities, centers, and systemic reform initiatives. NSF program
officers are given discretion in the specific use of review methods,
subject to supervisory approval. For example, some programs try to
manage proposal pressure by requiring submission of preliminary
proposals. Review of preliminary proposals varies widely, ranging from
non-binding advice from program officers to proposers, to
recommendations from external reviewers. The use of preliminary
proposals has increased in frequency over the past several years in NSF
programs. In FY 1999, NSF acted on 1,594 official preliminary proposals
that were logged into the proposal processing system. Of these, NSF
encouraged the submission of full proposals in 221 cases and discouraged
submission in 1,373 cases.

In "mail-only" reviews, peers are sent proposals and asked to submit
written comments to NSF by postal mail, facsimile, electronic mail, or
through FastLane, NSF's Web-based system for electronic proposal
submission and review. These mail reviews are then used by the NSF
program officer directly to support a recommendation for award or
decline.

"Panel-only" review refers to the process of soliciting reviews only
from those peers who meet in a panel review setting to discuss their
reviews and provide advice directly to the program officer. Most
programs that use this process provide proposals to panelists and
receive their reviews prior to the panel meeting. Other programs provide
panelists with access to the proposals at the beginning of the panel
meeting, allowing them a period of time during which they prepare their
reviews at the meeting.

Many proposals submitted to NSF are reviewed using some combination of
these two processes ("mail-plus-panel" review). Those programs that
employ the mail-plus-panel review process have developed several
different configurations, such as:

-   A peer is asked to submit a written mail review and also serve as a
    panelist, in effect contributing two reviews for each proposal; and

```{=html}
<!-- -->
```
-   A peer is asked to participate only as a panelist, with
    responsibility only for reviewing and discussing mail reviews
    written by others and providing verbal and/or written advice to the
    program officer.

The use of various review processes has changed markedly over time. The
percentage of NSF proposals reviewed by panel-only has increased from 36
to 47 percent of all proposals since FY 1990. There has been a steady
decline in the use of mail-only review from 33 to 18 percent during the
past decade. The use of mail-plus-panel review increased from 32 to 35
percent during the past ten years. These trends are shown in **Text
Figure 4**, and the corresponding data are presented in **Appendix Table
4**. These trends most likely result from the increasing complexity and
multidisciplinarity of proposals and the need to better manage the
proposal workload.

Directorate-level data on the use of different review processes during
FY 1999 are presented in **Appendix Table 5**. Directorates vary in
their use of proposal review methods. Mail-plus-panel review was the
most common review process used in the BIO, GEO, and SBE Directorates.
Mail-only review was the predominant mode of review in MPS.

![](media/image4.wmf)

Panel-only review was the most commonly used method in CISE, ENG and
EHR. These trends have major implications for the way NSF conducts its
business. For example, as indicated in **Text Figure 5**, there is a
strong relationship between time to decision and type of review. In FY
1999, 65% of all proposals reviewed by panel-only were processed within
six months, compared to 54% for mail-plus-panel and 50% for mail-only.

> **Text Figure 5**
>
> **Time to Decision by Type**
>
> **FY 1995 -- 1999**

+---------+-----------------+---------+----------+----------+---------+
| **FY**  | **Review        | **0-6** | *        | **       | *       |
|         | Method**        |         | *\>6-9** | \>9-12** | *\>12** |
|         |                 | **M     |          |          |         |
|         |                 | onths** | **       | **       | **M     |
|         |                 |         | Months** | Months** | onths** |
+---------+-----------------+---------+----------+----------+---------+
| *       | Mail-plus-Panel | 54%     | 36%      | 7%       | 3%      |
| *1999** |                 |         |          |          |         |
|         | Mail Only       | 50%     | 36%      | 10%      | 4%      |
|         |                 |         |          |          |         |
|         | Panel Only      | 65%     | 27%      | 6%       | 1%      |
+---------+-----------------+---------+----------+----------+---------+
| *       | Mail-plus-Panel | 56%     | 36%      | 6%       | 2%      |
| *1998** |                 |         |          |          |         |
|         | Mail Only       | 42%     | 43%      | 12%      | 4%      |
|         |                 |         |          |          |         |
|         | Panel Only      | 67%     | 27%      | 5%       | 1%      |
+---------+-----------------+---------+----------+----------+---------+
| *       | Mail-plus-Panel | 58%     | 34%      | 6%       | 3%      |
| *1997** |                 |         |          |          |         |
|         | Mail Only       | 42%     | 38%      | 12%      | 8%      |
|         |                 |         |          |          |         |
|         | Panel Only      | 70%     | 25%      | 3%       | 2%      |
+---------+-----------------+---------+----------+----------+---------+
| *       | Mail-plus-Panel | 44%     | 40%      | 11%      | 4%      |
| *1996** |                 |         |          |          |         |
|         | Mail Only       | 35%     | 42%      | 15%      | 8%      |
|         |                 |         |          |          |         |
|         | Panel Only      | 40%     | 48%      | 10%      | 2%      |
+---------+-----------------+---------+----------+----------+---------+
| *       | Mail-plus-Panel | 48%     | 40%      | 8%       | 5%      |
| *1995** |                 |         |          |          |         |
|         | Mail Only       | 39%     | 38%      | 13%      | 11%     |
|         |                 |         |          |          |         |
|         | Panel Only      | 51%     | 38%      | 8%       | 3%      |
+---------+-----------------+---------+----------+----------+---------+

> Note: FY 1999 numbers include 1,570 reviewed preproposals 1,554 were
> reviewed by panel and 16 by mail. Source: Custom program by Special
> Data Group, February 23, 2000.

## Reviews and Reviewers

NSF policy states that each recommendation for final action on a
proposal must be accompanied by at least three external reviews, unless
the requirement has been waived under special circumstances (see,
"Exemptions to the Merit Review Process," below). The total numbers of
reviews and the average numbers of reviews per proposal obtained by
these different methods are presented in **Text Figure 6**. Some of this
difference is reflected in the fact that panel counts refer to all the
participants on a given panel, not just the lead reviewers.

Directorate-level data for FY 1999 are presented in **Appendix Table
6**. There is considerable variation in the number of reviews per
proposal among the directorates, ranging from 15.7 (BIO) to 4.0 (ENG).

### Text Figure 6

**Reviews per Proposal, FY 1999**

  ------------------- ------------- --------------------- --------------- ----------------
                      **All         **Mail-plus-Panel**   **Mail-Only**   **Panel-Only**
                      Methods**                                           

  **\# of Reviews**   246,145       132,069               24,162          89,914

  **\# of Proposals** 29,564        10,324                5,292           13,948

  **Reviews per       8.3           12.8                  4.6             6.4
  Proposal**                                                              
  ------------------- ------------- --------------------- --------------- ----------------

Note: FY 1999 numbers includes 1,570 reviewed preproposals, 1,554 were
reviewed by panel and 16 by mail. Source: Custom program by Special Data
Group, February 23, 2000.

## 

A growing number of reviews are submitted electronically through NSF's
FastLane system. Of the 246,145 reviews submitted in FY 1999, 74,891, or
30 percent, were submitted through FastLane. In the first three months
of FY 2000, 27,040 proposals were submitted through FastLane, compared
to 11,648 for the same period in FY 1999. FastLane promises considerable
workload reduction for both the reviewer community and the NSF staff.

Diversity of the reviewer pool is an important feature of the NSF merit
review system. Reviewers from diverse backgrounds help ensure that a
wide range of perspectives are taken into consideration in the review
process. NSF emphasizes reviewer diversity through a variety of
processes, including use of a large and expanding Foundation-wide
reviewer database, explicit policy guidance, mandatory training for all
program officers, and directorate-level initiatives.

NSF maintains a central electronic database of about 250,000 reviewers.
For proposal decisions in FY 1999, 47,300 of these reviewers were sent
one or more proposals for mail review, 30,700 reviewed at least one
proposal by mail, and 8,300, reviewers served as panelists. In all,
51,900 individuals either served on a panel, were sent a proposal for
mail review, or served in both functions. Of these reviewers, 16 percent
were female, 77 percent were male, and for 7 percent the gender was
unknown. The data for minorities is not of sufficient quality to
publish.

Potential reviewers are identified from a variety of sources including
applicant suggestions, references attached to proposals and published
papers, and input from mail reviewers, panelists, and visiting
scientists. During FY 1999, approximately 29,400 of the 250,000 records
now in the reviewer database were either added or updated.

Participation in the peer review process is voluntary. Panelists are
reimbursed for expenses; mail reviewers receive no financial
compensation. In FY 1999, 59 percent of requests for mail reviews
produced responses, which represents a decrease from the 62 percent
response rate in FY 1998 and the 64 percent response rate that had been
stable since 1991.

## Reviewer Proposal Ratings

The NSF merit review system emphasizes reviewer narratives over summary
ratings. Summary ratings are but one indicator of reviewer judgment of
the proposal quality. The written narratives provided by reviewers, the
deliberations by panel members, and the expert opinions provided by
program officers are all important components of the merit review
system. No one component is allowed to dominate over the others.

The distribution of average summary ratings of reviews for awarded and
declined proposals is provided in **Text Figure 7**. Only those ratings
provided for mail-only and mail-plus-panel reviewers have been included.
Panel-only reviewers often submit comments without a summary rating.
These data indicate considerable overlap among the average reviewer
ratings of successful and unsuccessful proposals, most notably in the
range of "very good" average ratings. The judgment of NSF staff is
essential to making this difficult separation between awards and
declines. The data also indicate that a large number of potentially
fundable proposals are declined each year.

![](media/image5.wmf)

## 

## NSF Program Officers

The narrative comments and summary ratings provided by external
reviewers are essential inputs in NSF's system of merit review. Once
received, these inputs inform the judgment of the program officers who
formulate award and decline recommendations to NSF's senior management.
These program officers are scientists, engineers, and educators to whom
NSF looks for expert judgment and program management. In making
recommendations to award or decline proposals, these highly qualified
individuals produce and manage a portfolio of awards addressing NSF's
strategic goals and related factors such as:

-   Contributions to human resource and institutional infrastructure
    development,

-   Support for "risky" proposals with potential for significant
    advances in a field,

-   Encouragement of interdisciplinary activities, and

-   Achievement of program-level objectives and initiatives.

The number of program officers employed by NSF has remained stable at
around 400 for the past five years, despite increases in proposal
complexity and general workload. Depending on their professional
experience, program officers are classified as assistant program
director, associate program director, or program director. They can be
permanent NSF employees or temporary employees. Some temporary program
officers are "on loan" as visiting scientists, engineers, and educators
(VSEEs) for up to three years from their host institutions. Others are
employed through grants to the home institutions under the terms of the
Intergovernmental Personnel Act. The distribution of these program
officers among these and other categories is presented in **Text Figure
8**.

## **Text Figure 8**

**Distribution of NSF Program Officers by Characteristics**

+------------------+--------------+------------+-------------+--------+
|                  | *            | **A        | **Program** | **T    |
|                  | *Assistant** | ssociate** |             | otal** |
|                  |              |            | **          |        |
|                  | **Program**  | *          | Directors** |        |
|                  |              | *Program** |             |        |
|                  | *            |            |             |        |
|                  | *Directors** | **D        |             |        |
|                  |              | irectors** |             |        |
+------------------+--------------+------------+-------------+--------+
| **Total**        | 13           | 29         | 371         | 413    |
+------------------+--------------+------------+-------------+--------+
| **Male**         | 6            | 21         | 236         | 263    |
|                  |              |            |             |        |
|                  | 46.2%        | 72.4%      | 63.6%       | 63.7%  |
+------------------+--------------+------------+-------------+--------+
| **Female**       | 7            | 8          | 135         | 150    |
|                  |              |            |             |        |
|                  | 53.8%        | 27.6%      | 36.4%       | 36.3%  |
+------------------+--------------+------------+-------------+--------+
| **Minority**     | 1            | 4          | 68          | 73     |
|                  |              |            |             |        |
|                  | 7.7%         | 13.8%      | 18.3%       | 17.7%  |
+------------------+--------------+------------+-------------+--------+
| **White,**       | 12           | 25         | 303         | 340    |
|                  |              |            |             |        |
| **Non-Hispanic** | 92.3%        | 86.2%      | 81.7%       | 82.3%  |
+------------------+--------------+------------+-------------+--------+
| **Permanent**    | 10           | 19         | 195         | 224    |
|                  |              |            |             |        |
|                  | 76.9%        | 65.5%      | 52.5%       | 54.2%  |
+------------------+--------------+------------+-------------+--------+
| **VSEE**         | 0            | 3          | 34          | 37     |
|                  |              |            |             |        |
|                  | 0.0%         | 10.4%      | 9.2%        | 9.0%   |
+------------------+--------------+------------+-------------+--------+
| **Temporary**    | 2            | 0          | 43          | 45     |
|                  |              |            |             |        |
|                  | 15.4%        | 0.0%       | 11.6%       | 10.9%  |
+------------------+--------------+------------+-------------+--------+
| **IPA**          | 1            | 7          | 99          | 107    |
|                  |              |            |             |        |
|                  | 7.7%         | 24.1%      | 26.7%       | 25.9%  |
+------------------+--------------+------------+-------------+--------+
| Source: NSF      |              |            |             |        |
| Division of      |              |            |             |        |
| Human Resource   |              |            |             |        |
| Management       |              |            |             |        |
|                  |              |            |             |        |
| Notes: VSEE:     |              |            |             |        |
| Individual       |              |            |             |        |
| employed as a    |              |            |             |        |
| Visiting         |              |            |             |        |
| Scientist,       |              |            |             |        |
| Engineer, or     |              |            |             |        |
|                  |              |            |             |        |
| Educator         |              |            |             |        |
| (formerly termed |              |            |             |        |
| "Rotator").      |              |            |             |        |
|                  |              |            |             |        |
| IPA: Individual  |              |            |             |        |
| employed under   |              |            |             |        |
| the              |              |            |             |        |
| I                |              |            |             |        |
| ntergovernmental |              |            |             |        |
| Personnel Act.   |              |            |             |        |
+------------------+--------------+------------+-------------+--------+

## 

## Assuring Objectivity in the Merit Review Process

NSF program officers carefully check all proposals for potential
conflict-of-interest and select expert outside reviewers with no
apparent potential conflicts. All reviewers are instructed to declare
potential conflicts. All program officers receive conflicts-of-interest
training annually.

NSF policy includes several mechanisms that provide proposers with
information on how the review process led to a recommendation, and on
procedures for obtaining additional explanations for declinations. These
policies help to ensure that NSF's review has been fair and reasonable,
and that NSF maintains the openness, quality, and integrity of the merit
review process.

Every proposer receives from the NSF program officer a description of
the context in which the proposal was reviewed, along with an anonymous
verbatim copy of each review that was considered in the review process.
A declined PI may ask the cognizant program officer for additional
clarification of the decision. If after considering this additional
information a PI is not satisfied that the proposal was fairly handled
and reasonably reviewed, he or she may request formal reconsideration
from the cognizant Assistant Director (AD). This request can be based on
the PI's perception of procedural errors or on disagreements over the
substantive issues dealt with by reviewers. If the AD upholds the
original action, the applicant's [institution]{.underline} may request a
second reconsideration from the Foundation's Deputy Director (O/DD).

On average, NSF annually declines over 20,000 proposals but receives, on
average, only 40-50 requests for formal reconsideration. Most
program-level decisions are upheld in the reconsideration process. Out
of the 240 requests for formal reconsideration of declined proposals
during the past six years, nine decisions have been reversed. The number
of requests for formal reconsideration and resulting decisions at both
the AD and O/DD levels from FY 1994 through FY 1999 are displayed in
**Appendix Table 7**.

Each program officer's recommendation to award or decline a proposal is
subject to a programmatic review by a higher level reviewing official
(usually the division director), and an administrative review by a
grants officer in the Office of Budget, Finance, and Award Management
(BFA). All award recommendations in excess of \$1.5 million in any one
project year or \$6 million over five years must be reviewed by the
Director's Review Board (DRB). In FY 1999, awards in excess of a \$3
million commitment during a project year, or \$15 million over five
years, required approval by the National Science Board.

Changes to the NSB threshold were approved by the NSB in July, 1999 and
implemented on September 30, 1999, effective Fiscal Year 2000. The new
threshold requires NSB approval of awards where the average annual award
amount is 1% or more of the awarding directorate\'s prior year current
plan.

# Other Issues Related to Merit Review

## 

## Committees of Visitors (COV)

NSF regularly assesses performance of all aspects of the merit review
system, comparing its efficiency, effectiveness, customer satisfaction
and integrity against similar processes run by other organizations. For
example, panels of external experts called Committees of Visitors (COVs)
are convened to review the technical and managerial stewardship of NSF
programs on a three-year cycle. COVs report on the integrity and
efficiency of the processes for proposal review and the quality of
results of NSF\'s programs in the form of outputs and outcomes that
appear over time. The recommendations of COVs are reviewed by management
and taken into consideration by NSF when evaluating existing programs
and future directions for the Foundation.

In FY 1999, NSF's activities were organized into nearly 200 programs. In
FY 1999, a total of 18 committees of visitors (COVs) met to conduct
reviews of 82 programs, producing a total of 43 COV reports assessing
the quality of program performance and outcome results. The number of
COV reports is greater than the number of COVs because the COVs were
organized into subgroups to produce reports covering more than one
program. A list of all programs subject to review by a Committee of
Visitors and the fiscal year of the most recent review is provided in
**Appendix Table 10**.

Each COV must operate in accordance with the Federal Advisory Committee
Act (FACA) of 1972. In compliance with FACA regulations, virtually all
COVs are established as subcommittees of an existing chartered
directorate advisory committee, and the COV report is reviewed and
approved by the parent advisory committee. The cognizant assistant
director (AD) provides the parent advisory committee with a written
response to each COV report. The COV's report and the AD's response are
public documents; some have been publicized in the professional
literature.

## Government Performance and Results Act (GPRA)

The merit review system is at the very heart of NSF's selection of the
projects through which its GPRA outcome goals are achieved. Hence, in
NSF's new GPRA Strategic Plan, "o*perating a credible, efficient merit
review system"* is cited as a critical factor for success of the agency.
Ensuring a credible, efficient system requires constant attention and
openness to change. Two implementation strategies for accomplishing this
are: (1) Regularly assess performance of all aspects of the merit review
system, comparing its efficiency, effectiveness, customer satisfaction
and integrity against similar processes run by other organizations; and
(2) Promote the use of both merit review criteria (i.e. *intellectual
merit* and *broader impacts*) in the evaluation of proposals.

In the FY 1999 GPRA Performance Plan, NSF's investment process goals
focus on the means and strategies the Foundation uses to make investment
decisions and shape its portfolio of awards in order to achieve its
mission and desired outcome goals. Two of these goals (goals 6&7,
described below) specifically addressed the use of merit review.

[Goal 6]{.underline}: At least 90% of NSF funds will be allocated to
projects reviewed by appropriate peers external to NSF and selected
through a merit-based competitive process. (This is a government-wide
goal for all federal science, space and technology funding agencies.)

*Results*: In FY 1999 NSF exceeded this goal, with 95% of project funds
allocated to projects subjected to merit review. This goal will be
maintained in FY 2000. NSF expects to exceed this government-wide goal
again.

[Goal 7:]{.underline} NSF performance in implementation of the new merit
review criteria

Background: In 1997 the NSB approved new NSF merit review criteria. The
two new review criteria are (1) What is the intellectual merit of the
proposed activity; and (2) What are the broader impacts of the proposed
activity? The guidance accompanying the new review criteria state that
[both]{.underline} criteria must be addressed in the evaluation of a
proposal.

NSF's performance goal for the implementation of the new merit review
criteria is stated in the alternative (narrative) GPRA format. NSF
performance is:

-   ***successful*** when reviewers address the elements of both generic
    review criteria appropriate to the proposal at hand and when program
    officers take the information provided into account in their
    decisions on awards; or

-   ***minimally effective*** when reviews consistently use only a few
    of the suggested elements of the generic review criteria although
    others might be applicable.

> *Results*: *Largely successful, needs some improvement*. For FY 1999,
> Committees of Visitors (COV's) and Advisory Committees (AC's) used the
> alternative format to judge how well NSF is implementing the new
> criteria. In FY 1999, a total of 38 COV reports and 6 AC reports rated
> NSF programs on their use of the new merit review criteria. NSF was
> rated successful in achieving this goal in 33 COV reports and 3 AC
> reports.
>
> One AC report gives NSF a qualified successful rating, and two AC
> reports rate NSF minimally effective in implementing this goal. In
> most cases where NSF was not fully successful, it was found that
> reviewers and applicants were not fully addressing both review
> criteria. This goal will be maintained for FY 2000.

*Recent Progress:* NSF has established guidelines in program
announcements requiring applicants and reviewers to address these
criteria in proposals and reviews. NSF has recently re-issued guidance
to the applicants and reviewers, stressing the importance of using both
criteria in the preparation and evaluation of proposals submitted to
NSF. NSF is also taking additional steps to ensure that applicants
address these criteria when reporting project results.

As part of the FY 1999 Performance Plan, the following language was
added to NSF program announcements and included in the Grant Proposal
Guide:

*PIs should address the following elements in their proposal to provide
reviewers with the information necessary to respond fully to the
above-described NSF merit review criteria. NSF staff will give these
elements careful consideration in making funding decisions.*

#### Integration of Research and Education 

One of the principal strategies in support of NSF's goals is to foster
integration of research and education through the programs, projects and
activities it supports at academic and research institutions. These
institutions provide abundant opportunities where individuals may
concurrently assume responsibilities as researchers, educators, and
students, and where all can engage in joint efforts that infuse
education with the excitement of discovery and enrich research through
the diversity of learning perspectives.

#### Integrating Diversity into NSF Programs, Projects, and Activities

Broadening opportunities and enabling the participation of all citizens
\-- women and men, underrepresented minorities, and persons with
disabilities \-- are essential to the health and vitality of science and
engineering. NSF is committed to this principle of diversity and deems
it central to the programs, projects, and activities it considers and
supports.

## Use of Preliminary Proposals

The use of preliminary proposals has increased in frequency over the
past several years in NSF programs. The intent of preliminary proposals
is to limit the burden imposed on proposers, reviewers and NSF staff.
Normally, preliminary proposals require only enough information to make
fair and reasonable decisions regarding encouragement/discouragement of
a full proposal. In FY 1999, NSF acted on 1,594 preliminary proposals
that were officially logged into the proposal processing system. Of
these, NSF encouraged the submission of full proposals in 221 cases and
discouraged submission in 1,373 cases. Until now, there have not been
NSF-wide policies for the use, review and tracking of preliminary
proposals. However, the establishment of such policies is currently
under discussion.

## Cost Sharing Policy Revision

In accordance with Congressional requirements, NSF requires that each
grantee share in the cost of NSF research projects resulting from
unsolicited proposals. In addition to the statutory requirements, NSF
can require cost sharing when it believes there is tangible benefit to
the award recipient(s) (normally beyond the immediate term or scope of
the NSF-supported activity).

In FY 1999, NSF revised its policy with respect to cost sharing. It took
effect on 7 May 1999, when approved by the National Science Board. The
revised policy set forth the following principle for how non-statutory
cost sharing may be used as an eligibility factor in the receipt of
proposals.

> *NSF cost sharing requirements beyond the statutory requirement will
> be clearly stated in the program announcement, solicitation or other
> mechanism, which generates proposals to the program. NSF-required cost
> sharing is considered an eligibility rather than review criterion.*

This obviously has important implications for the proposal solicitation
process and its implementation will be closely monitored. A new system
to automate proposal solicitation preparation has helped support the use
of clear, consistent language in this area.

## Small Grants for Exploratory Research (SGER)

Since the beginning of FY 1990, the Small Grants for Exploratory
Research (SGER) option has permitted program officers throughout the
Foundation to make short-term (one to two years), small-scale (less than
\$50 K) grants *without formal external review*. Characteristics of
activities that can be supported by an SGER award include:

-   Preliminary work on untested and novel ideas;

-   Application of new approaches to "old" topics;

-   Ventures into emerging research areas; and

-   Narrow windows of opportunity for data collection, such as natural
    disasters and infrequent phenomena.

The SGER funding rate is much higher than for regular, competitively
reviewed proposals in large part because potential SGER applicants are
encouraged to contact an NSF program officer before submitting an SGER
proposal to determine its appropriateness for the SGER funding option.
As potential SGER applicants have become familiar with this practice,
the SGER funding rate has increased from 55 percent in its first year
(FY 1990) to 81 percent in FY 1999. Additional details are shown in
**Text Figure 9**.

### Text Figure 9

**Small Grants for Exploratory Research (SGER)**

Fiscal Years

  ---------------- ------- ------ ------ ------ ------ ------ ------ ------
                   1992    1993   1994   1995   1996   1997   1998   1999

  Proposals        330     309    258    228    205    244    302    277

  Awards           194     213    185    168    144    194    250    223

  Funding Rate (%) 65      69     72     74     70     80     83     81
  ---------------- ------- ------ ------ ------ ------ ------ ------ ------

NSF management has been concerned by the decrease in SGER proposal
pressure since the activity's inception in 1990. In response, Staff
Memorandum O/D 97-06 (dated June 5, 1997) announced a three-year
experiment that increases the SGER award limit from \$50,000 to
\$100,000. Program officers were also given permission to grant
six-month extensions and supplements of up to \$50,000 for extant SGER
awards. (EHR and BIO elected not to offer these time extensions or
supplements.)

Coincident with these policy changes, the downward trends in SGER
proposal pressure and number of awards were reversed in FY 1998. NSF
received 244 SGER proposals in FY 1997 and made 194 awards. NSF received
302 SGER proposals in FY 1998 and made 250 awards. Directorates vary in
the degree to which their program officers discourage potential
proposers from submitting inappropriate ideas as formal SGER proposals.
As a result, FY 1998 funding rates for SGER proposals varied among
directorates from 47 to 94 percent. Directorate-level data on SGER
proposal pressure and funding rates are presented in **Appendix Table
8**.

The total amount awarded to SGERs in FY 1999 was \$12,293,477. The
average SGER award amount in FY 1999 was \$55,128, a 27 percent increase
relative to the FY 1997 average award amount of \$43,367. Despite these
increases, the total NSF investment in SGERs remains less than one half
of one percent of the operating budget for research and education, far
below the five percent that program officers may commit to SGER awards.
The history of SGER awards by directorate from FY 1997 to FY 1999 is
presented in **Appendix Table 9**.

## Accomplishment Based Renewals and Creativity Extensions

In addition to SGERs, NSF program officers may recommend *accomplishment
based renewals* and *creativity extensions*.

An accomplishment-based renewal is a method that can be used by PIs to
submit renewal proposals to NSF. In this type of renewal proposal, the
project description is replaced by copies of no more than six reprints
of publications resulting from the research supported by NSF (or
research supported by other sources that is closely related to the
NSF-supported research) during the preceding three- to five-year period.
Of the six publications, two preprints (accepted for publication) may be
included. In addition, a brief (not to exceed four pages) summary of
plans for the proposed support period must be submitted. All other
information required for NSF proposal submission remains the same.

A creativity extension is an extension of funding for up to two years
for certain research grants. The objective of such extensions is to
offer the most creative investigators an extended opportunity to attack
adventurous, \"high-risk\" opportunities in the same general research
area, but not necessarily covered by the original/current proposal.
Special Creativity Extensions are initiated by the NSF Program Officer
based on progress during the first two years of a three-year grant.

In 1999 there were 30 requests for accomplishment based renewals, 15 of
which were awarded. There were also 48 creativity based extensions made
to existing NSF grants.

## Exemptions to the Merit Review Process

Authorized exemptions to the peer review process are listed in NSF
Manual 10, Section 122 (**Attachment I**) and include routine award
actions such as continuing grant increments and no-cost extensions. In
special circumstances, the Director or designee may waive peer review
requirements. Such waivers of peer review were granted 7 times during FY
1999; 5 for OIA, 1 for SBE and 1 for CISE.

**Appendix Table 1**

**Competitively Reviewed Proposals, Awards and Funding Rates**

######### By Directorate, FY 1995 -- 1999

+------+----------+------+------+------+------+------+------+------+
|      |          | Fi   |      |      |      |      | *Fi  | ##   |
|      |          | scal |      |      |      |      | ve-y | #### |
|      |          | Year |      |      |      |      | ear* | ## F |
|      |          |      |      |      |      |      |      | ive- |
|      |          |      |      |      |      |      |      | year |
+------+----------+------+------+------+------+------+------+------+
|      |          | 1995 | 1996 | 1997 | 1998 | 1999 | *To  | *    |
|      |          |      |      |      |      |      | tal* | Aver |
|      |          |      |      |      |      |      |      | age* |
+------+----------+------+------+------+------+------+------+------+
| NSF  | P        | 30   | 30   | 30   | 28   | 28   | 147  | 29   |
|      | roposals | ,700 | ,231 | ,189 | ,318 | ,504 | ,942 | ,588 |
+------+----------+------+------+------+------+------+------+------+
|      | Awards   | 9    | 9    | 9    | 9    | 9    | 46   | 9    |
|      |          | ,563 | ,071 | ,864 | ,279 | ,112 | ,889 | ,378 |
+------+----------+------+------+------+------+------+------+------+
|      | Funding  | 31%  | 30%  | 33%  | 33%  | 32%  | 32%  | 32%  |
|      | Rate     |      |      |      |      |      |      |      |
+------+----------+------+------+------+------+------+------+------+
| BIO  | P        | 5    | 5    | 5    | 4    | 4    | 25   | 5    |
|      | roposals | ,255 | ,676 | ,209 | ,857 | ,711 | ,708 | ,142 |
+------+----------+------+------+------+------+------+------+------+
|      | Awards   | 1    | 1    | 1    | 1    | 1    | 6    | 1    |
|      |          | ,383 | ,328 | ,416 | ,404 | ,370 | ,901 | ,380 |
+------+----------+------+------+------+------+------+------+------+
|      | Funding  | 26%  | 23%  | 27%  | 29%  | 29%  | 27%  | 27%  |
|      | Rate     |      |      |      |      |      |      |      |
+------+----------+------+------+------+------+------+------+------+
| CSE  | P        | 2    | 1    | 2    | 2    | 2    | 10   | 2    |
|      | roposals | ,067 | ,931 | ,010 | ,035 | ,255 | ,298 | ,060 |
+------+----------+------+------+------+------+------+------+------+
|      | Awards   | 722  | 647  | 731  | 706  | 759  | 3    | 713  |
|      |          |      |      |      |      |      | ,565 |      |
+------+----------+------+------+------+------+------+------+------+
|      | Funding  | 35%  | 34%  | 36%  | 35%  | 34%  | 35%  | 35%  |
|      | Rate     |      |      |      |      |      |      |      |
+------+----------+------+------+------+------+------+------+------+
| EHR  | P        | 4    | 3    | 3    | 3    | 2    | 18   | 3    |
|      | roposals | ,979 | ,732 | ,369 | ,508 | ,827 | ,415 | ,683 |
+------+----------+------+------+------+------+------+------+------+
|      | Awards   | 1    | 1    | 1    | 1    | 809  | 6    | 1    |
|      |          | ,475 | ,326 | ,191 | ,212 |      | ,013 | ,203 |
+------+----------+------+------+------+------+------+------+------+
|      | Funding  | 30%  | 36%  | 35%  | 35%  | 29%  | 33%  | 33%  |
|      | Rate     |      |      |      |      |      |      |      |
+------+----------+------+------+------+------+------+------+------+
| ENG  | P        | 5    | 5    | 6    | 5    | 5    | 28   | 5    |
|      | roposals | ,740 | ,956 | ,076 | ,589 | ,475 | ,836 | ,767 |
+------+----------+------+------+------+------+------+------+------+
|      | Awards   | 1    | 1    | 1    | 1    | 1    | 7    | 1    |
|      |          | ,473 | ,383 | ,573 | ,390 | ,483 | ,302 | ,460 |
+------+----------+------+------+------+------+------+------+------+
|      | Funding  | 26%  | 23%  | 26%  | 25%  | 27%  | 25%  | 25%  |
|      | Rate     |      |      |      |      |      |      |      |
+------+----------+------+------+------+------+------+------+------+
| GEO  | P        | 3    | 3    | 3    | 3    | 3    | 17   | 3    |
|      | roposals | ,421 | ,723 | ,950 | ,317 | ,435 | ,846 | ,569 |
+------+----------+------+------+------+------+------+------+------+
|      | Awards   | 1    | 1    | 1    | 1    | 1    | 6    | 1    |
|      |          | ,199 | ,161 | ,337 | ,227 | ,312 | ,236 | ,247 |
+------+----------+------+------+------+------+------+------+------+
|      | Funding  | 35%  | 31%  | 34%  | 37%  | 38%  | 35%  | 35%  |
|      | Rate     |      |      |      |      |      |      |      |
+------+----------+------+------+------+------+------+------+------+
| MPS  | P        | 5    | 4    | 5    | 5    | 5    | 26   | 5    |
|      | roposals | ,203 | ,958 | ,536 | ,265 | ,177 | ,139 | ,228 |
+------+----------+------+------+------+------+------+------+------+
|      | Awards   | 1    | 1    | 1    | 1    | 1    | 9    | 1    |
|      |          | ,864 | ,817 | ,993 | ,835 | ,891 | ,400 | ,880 |
+------+----------+------+------+------+------+------+------+------+
|      | Funding  | 36%  | 37%  | 36%  | 35%  | 37%  | 36%  | 36%  |
|      | Rate     |      |      |      |      |      |      |      |
+------+----------+------+------+------+------+------+------+------+
| SBE  | P        | 3    | 3    | 3    | 3    | 3    | 17   | 3    |
|      | roposals | ,490 | ,453 | ,286 | ,091 | ,909 | ,229 | ,446 |
+------+----------+------+------+------+------+------+------+------+
|      | Awards   | 1    | 1    | 1    | 1    | 1    | 5    | 1    |
|      |          | ,149 | ,137 | ,223 | ,262 | ,190 | ,961 | ,192 |
+------+----------+------+------+------+------+------+------+------+
|      | Funding  | 33%  | 33%  | 37%  | 41%  | 30%  | 35%  | 35%  |
|      | Rate     |      |      |      |      |      |      |      |
+------+----------+------+------+------+------+------+------+------+
| O    | P        | 545  | 802  | 753  | 656  | 715  | 3    | 694  |
| ther | roposals |      |      |      |      |      | ,471 |      |
+------+----------+------+------+------+------+------+------+------+
|      | Awards   | 298  | 272  | 400  | 243  | 298  | 1    | 302  |
|      |          |      |      |      |      |      | ,511 |      |
+------+----------+------+------+------+------+------+------+------+
|      | Funding  | 55%  | 34%  | 53%  | 37%  | 42%  | 44%  | 44%  |
|      | Rate     |      |      |      |      |      |      |      |
+------+----------+------+------+------+------+------+------+------+

Source: NSF Enterprise Information System, as of December 15, 1999.

**Appendix Table 2**

**Competitively Reviewed Proposals, Awards and Funding Rates**

**By PI Characteristics, FY 1992 - 1999**

  ---------- ----------- -------- -------- -------- -------- -------- -------- -------- --------
                         Fiscal                                                         
                         Year                                                           

                         1992     1993     1994     1995     1996     1997     1998     1999

  All PIs    Proposals   30,320   30,003   30,399   30,700   30,231   30,189   28,318   28,504

             Awards      10,356   9,148    9,976    9,563    9,071    9,864    9,279    9,112

             Funding     34%      30%      33%      31%      30%      33%      33%      32%
             Rate                                                                       

  Female PIs Proposals   4,448    4,468    4,833    4,945    5,170    5,382    5,603    5,292

             Awards      1,519    1,460    1,641    1,582    1,666    1,938    1,920    1,658

             Funding     34%      33%      34%      32%      32%      36%      34%      31%
             Rate                                                                       

  Male PIs   Proposals   25,334   25,137   25,023   25,151   24,712   24,448   22,379   22,935

             Awards      8,503    7,563    8,018    7,633    7,292    7,793    7,219    7,365

             Funding     34%      30%      32%      30%      30%      32%      32%      32%
             Rate                                                                       

  Minority   Proposals   1,481    1,408    1,449    1,521    1,527    1,452    1,377    1,418
  PIs                                                                                   

             Awards      469      391      422      422      472      459      408      430

             Funding     32%      28%      29%      28%      31%      32%      30%      30%
             Rate                                                                       

  New PIs    Proposals   14,988   14,284   14,566   14,192   13,630   13,267   12,204   11,831

             Awards      3,735    3,025    3,598    3,367    3,021    3,264    3,041    2,675

             Funding     25%      21%      25%      24%      22%      25%      25%      23%
             Rate                                                                       

  Prior PIs  Proposals   15,332   15,719   15,833   16,508   16,601   16,922   16,114   16,673

             Awards      6,621    6,123    6,378    6,196    6,050    6,600    6,238    6,437

             Funding     43%      39%      40%      38%      36%      39%      39%      39%
             Rate                                                                       
  ---------- ----------- -------- -------- -------- -------- -------- -------- -------- --------

Notes:

"Competitively reviewed" proposals and awards are actions for research,
education and training processed through NSF's merit review system each
year.

"Gender" is based on self-reported information from the PI's most recent
proposal.

"Minority" is based on the PI's ethnic/racial status as reported to NSF
on the most recent proposal.

PIs can decline to report their ethnic/racial status. Includes American
Indian, Alaskan Native, Black, Hispanic, and Pacific Islander and
excludes Asian and White-Not of Hispanic Origin.

Source: NSF Enterprise Information System, as of December 15, 1999

**Appendix Table 3**

**Median and Average Award Amounts by Directorate**

**Research Grants**

**FY 1995 -- 1999**

+--------+--------+---------+---------+---------+---------+---------+
|        |        | Fiscal  |         |         |         |         |
|        |        | Year    |         |         |         |         |
+--------+--------+---------+---------+---------+---------+---------+
|        |        | 1995    | 1996    | 1997    | 1998    | 1999    |
+--------+--------+---------+---------+---------+---------+---------+
| NSF    | Median | \       | \       | \       | \       | \       |
|        |        | $59,468 | $61,744 | $67,000 | $70,854 | $73,335 |
+--------+--------+---------+---------+---------+---------+---------+
|        | A      | \       | \       | \       | \       | \       |
|        | verage | $72,798 | $75,613 | $82,256 | $89,854 | $92,788 |
+--------+--------+---------+---------+---------+---------+---------+
| BIO    | Median | \       | \       | \       | \       | \       |
|        |        | $76,667 | $79,604 | $85,819 | $90,000 | $94,078 |
+--------+--------+---------+---------+---------+---------+---------+
|        | A      | \       | \       | \       | \$      | \$      |
|        | verage | $82,960 | $84,415 | $89,670 | 103,327 | 125,901 |
+--------+--------+---------+---------+---------+---------+---------+
| CSE    | Median | \       | \       | \       | \       | \       |
|        |        | $55,395 | $57,788 | $74,450 | $75,000 | $82,850 |
+--------+--------+---------+---------+---------+---------+---------+
|        | A      | \       | \       | \       | \       | \$      |
|        | verage | $76,283 | $86,721 | $93,634 | $99,014 | 108,661 |
+--------+--------+---------+---------+---------+---------+---------+
| ENG    | Median | \       | \       | \       | 473,122 | \       |
|        |        | $62,087 | $68,870 | $70,000 |         | $73,547 |
|        | A      |         |         |         | \       |         |
|        | verage | \       | \       | \       | $86,879 | \       |
|        |        | $69,433 | $73,168 | $77,864 |         | $84,594 |
+--------+--------+---------+---------+---------+---------+---------+
|        |        |         |         |         |         |         |
+--------+--------+---------+---------+---------+---------+---------+
| GEO    | Median | \       | \       | \       | \       | \       |
|        |        | $61,821 | $62,635 | $66,516 | $69,465 | $66,796 |
+--------+--------+---------+---------+---------+---------+---------+
|        | A      | \       | \       | \       | \       | \       |
|        | verage | $75,859 | $80,467 | $80,182 | $86,747 | $85,609 |
+--------+--------+---------+---------+---------+---------+---------+
| MPS    | Median | \       | \       | \       | \       | \       |
|        |        | $55,833 | $60,000 | $65,000 | $73,670 | $77,139 |
+--------+--------+---------+---------+---------+---------+---------+
|        | A      | \       | \       | \       | \       | \       |
|        | verage | $80,487 | $82,060 | $91,409 | $94,559 | $93,591 |
+--------+--------+---------+---------+---------+---------+---------+
| SBE    | Median | \       | \       | \       | \       | \       |
|        |        | $33,973 | $27,471 | $32,278 | $37,387 | $39,493 |
+--------+--------+---------+---------+---------+---------+---------+
|        | A      | \       | \       | \       | \       | \       |
|        | verage | $42,068 | $37,404 | $46,673 | $51,807 | $49,481 |
+--------+--------+---------+---------+---------+---------+---------+
| O/D    | Median | \       | \       | \       | \       | \       |
|        |        | $61,466 | $76,358 | $75,634 | $85,575 | $81,096 |
+--------+--------+---------+---------+---------+---------+---------+
|        | A      | \       | \       | \$      | \$      | \$      |
|        | verage | $80,826 | $95,434 | 103,259 | 111,461 | 112,671 |
+--------+--------+---------+---------+---------+---------+---------+

> Note: Median and average are based on competitively reviewed research
>
> awards.
>
> Source: NSF Enterprise Information System, as of February 7, 2000.

**Appendix Table 4**

**Methods of NSF Proposal Review**

**FY 1990-1999**

  -------- ----------- ----------- --------- ----------- --------- ------------ ---------
           Total       Mail +                Mail-Only             Panel-Only   
                       Panel                                                    

  FY       Proposals   Proposals   Percent   Proposals   Percent   Proposals    Percent

  1999     29,564      10,324      35%       5,292       18%       13,948       47%

  1998     28,492      9,883       35%       5,890       21%       12,719       45%

  1997     29,467      10,334      35%       6,718       23%       12,415       42%

  1996     29,595      9,853       33%       6,853       23%       12,889       44%

  1995     30,096      9,844       33%       7,540       25%       12,712       42%

  1994     29,869      8,499       28%       7,582       25%       13,788       46%

  1993     28,956      8,401       29%       7,949       27%       12,606       44%

  1992     28,938      8,484       29%       8,970       31%       11,484       40%

  1991     27,945      8,602       31%       8,307       30%       11,036       39%

  1990     27,987      8,834       32%       9,099       33%       10,054       36%
  -------- ----------- ----------- --------- ----------- --------- ------------ ---------

> Note: "O/D" includes the Office of Polar Programs and the Office of
> Integrative Activities.
>
> Panel-Only includes cases where panelist was mailed proposal for
> review prior to panel.
>
> FY 1999 numbers includes 1,570 reviewed preproposals.
>
> Source: NSF Enterprise Information System, as of December 15, 1999.

**Appendix Table 5**

**Methods of NSF Proposal Review**

**By Directorate, FY 1999**

  ------------- ----------- ----------- --------- ----------- --------- ------------ ---------
                Total       Mail +                Mail-Only             Panel-Only   
                            Panel                                                    

  Directorate   Proposals   Proposals   Percent   Proposals   Percent   Proposals    Percent

  NSF           29,564      10,324      35%       5,292       18%       13,948       47%

  BIO           4,622       3,398       74%       99          2%        1,125        24%

  CSE           2,287       412         18%       158         7%        1,717        75%

  EHR           3,364       111         3%        103         3%        3,150        94%

  ENG           5,626       456         8%        813         14%       4,357        77%

  GEO           3,401       2,457       72%       649         19%       295          9%

  MPS           5,136       1,230       24%       2,521       49%       1,385        27%

  SBE           3,880       1,853       48%       730         19%       1,297        33%

  Other         1,248       407         33%       219         18%       622          50%
  ------------- ----------- ----------- --------- ----------- --------- ------------ ---------

Note: "Other" includes the Office of Polar Programs and the Office of
Integrative Activities.

> Panel-Only includes cases where panelist was mailed proposal for
> review prior to panel. FY 1999 numbers includes 1,570 reviewed
> preproposals.

Source: NSF Enterprise Information System, as of December 15, 1999

**Appendix Table 6**

**Average Number of Reviews per Proposal**

**By Method and Directorate, FY 1999**

  ------------- ----------- ------------ ------------- ----------- ------------
                            Methods Of                             
                            Review                                 

  Directorate               All Methods  Mail + Panel  Mail-Only   Panel-Only

  BIO           Reviews     72,736       57,157        474         15,105

                Proposals   4,622        3,398         99          1,125

                Rev/Prop    15.7         16.8          4.8         13.4

                                                                   

  CSE           Reviews     11,727       2,635         645         8,447

                Proposals   2,287        412           158         1,717

                Rev/Prop    5.1          6.4           4.1         4.9

                                                                   

  EHR           Reviews     19,664       777           428         18,459

                Proposals   3,364        111           103         3,150

                Rev/Prop    5.8          7.0           4.2         5.9

                                                                   

  ENG           Reviews     22,521       2,506         3,439       16,576

                Proposals   5,626        456           813         4,357

                Rev/Prop    4.0          5.5           4.2         3.8

                                                                   

  GEO           Reviews     36,709       31,042        3,410       2,257

                Proposals   3,401        2,457         649         295

                Rev/Prop    10.8         12.6          5.3         7.7

                                                                   

  MPS           Reviews     37,306       12,266        12,052      12,988

                Proposals   5,136        1,230         2,521       1,385

                Rev/Prop    7.3          10.0          4.8         9.4

                                                                   

  SBE           Reviews     37,883       21,865        2,722       13,296

                Proposals   3,880        1,853         730         1,297

                Rev/Prop    9.8          11.8          3.7         10.3

                                                                   

  O/D           Reviews     7,599        3,821         992         2,786

                Proposals   1,248        407           219         622

                Rev/Prop    6.1          9.4           4.5         4.5
  ------------- ----------- ------------ ------------- ----------- ------------

> Notes:
>
> Peers participating as both a mail and panel reviewer for the same
> proposal are counted as one review in this table.
>
> Source: NSF Enterprise Information System, as of December 15, 1999.
>
> FY 1999 numbers includes 1,570 reviewed preproposals, 1,554 were
> reviewed by panel and 16 by mail.

**Appendix Table 7**

**Requests for Formal Reconsideration of Declined Proposals**

**By Directorate, FY 1994 - 1999**

+-------+-------------+-------+-------+-------+-------+-------+-------+
|       |             | #     |       |       |       |       |       |
|       |             | ##### |       |       |       |       |       |
|       |             | ### F |       |       |       |       |       |
|       |             | iscal |       |       |       |       |       |
|       |             |  Year |       |       |       |       |       |
+-------+-------------+-------+-------+-------+-------+-------+-------+
|       |             | **1   | **1   | **1   | **1   | **1   | **1   |
|       |             | 994** | 995** | 996** | 997** | 998** | 999** |
+-------+-------------+-------+-------+-------+-------+-------+-------+
| First |             |       |       |       |       |       |       |
| Level |             |       |       |       |       |       |       |
| Re    |             |       |       |       |       |       |       |
| views |             |       |       |       |       |       |       |
| (by   |             |       |       |       |       |       |       |
| Assi  |             |       |       |       |       |       |       |
| stant |             |       |       |       |       |       |       |
| D     |             |       |       |       |       |       |       |
| irect |             |       |       |       |       |       |       |
| ors): |             |       |       |       |       |       |       |
+-------+-------------+-------+-------+-------+-------+-------+-------+
| **    | **Request** | *     | *     | *     | *     | *     | *     |
| NSF** |             | *28** | *38** | *46** | *39** | *53** | *36** |
|       | **-         |       |       |       |       |       |       |
|       | Upheld**    | *     | *     | *     | *     | *     | *     |
|       |             | *30** | *37** | *45** | *34** | *48** | *34** |
|       | **-         |       |       |       |       |       |       |
|       | Reversed**  | **0** | **1** | **1** | **4** | **3** | **0** |
+-------+-------------+-------+-------+-------+-------+-------+-------+
| BIO   | Request     | 5     | 4     | 3     | 4     | 6     | 4     |
|       |             |       |       |       |       |       |       |
|       | \- Upheld   | 5     | 4     | 3     | 2     | 6     | 4     |
|       |             |       |       |       |       |       |       |
|       | \- Reversed | 0     | 0     | 0     | 2     | 0     | 0     |
+-------+-------------+-------+-------+-------+-------+-------+-------+
| CISE  | Request     | 0     | 3     | 1     | 2     | 3     | 1     |
|       |             |       |       |       |       |       |       |
|       | \- Upheld   | 0     | 3     | 1     | 2     | 3     | 1     |
|       |             |       |       |       |       |       |       |
|       | \- Reversed | 0     | 0     | 0     | 0     | 0     | 0     |
+-------+-------------+-------+-------+-------+-------+-------+-------+
| EHR   | Request     | 4     | 3     | 8     | 4     | 6     | 3     |
|       |             |       |       |       |       |       |       |
|       | \- Upheld   | 6     | 3     | 8     | 4     | 5     | 3     |
|       |             |       |       |       |       |       |       |
|       | \- Reversed | 0     | 0     | 0     | 0     | 1     | 0     |
+-------+-------------+-------+-------+-------+-------+-------+-------+
| ENG   | Request     | 6     | 3     | 5     | 9     | 5     | 4     |
|       |             |       |       |       |       |       |       |
|       | \- Upheld   | 6     | 3     | 5     | 9     | 4     | 4     |
|       |             |       |       |       |       |       |       |
|       | \- Reversed | 0     | 0     | 0     | 0     | 0     | 0     |
+-------+-------------+-------+-------+-------+-------+-------+-------+
| GEO   | Request     | 5     | 5     | 4     | 2     | 2     | 2     |
|       |             |       |       |       |       |       |       |
|       | \- Upheld   | 5     | 5     | 4     | 2     | 2     | 1     |
|       |             |       |       |       |       |       |       |
|       | \- Reversed | 0     | 0     | 0     | 0     | 0     | 0     |
+-------+-------------+-------+-------+-------+-------+-------+-------+
| MPS   | Request     | 8     | 18    | 20    | 17    | 25    | 20    |
|       |             |       |       |       |       |       |       |
|       | \- Upheld   | 8     | 17    | 19    | 15    | 22    | 19    |
|       |             |       |       |       |       |       |       |
|       | \- Reversed | 0     | 1     | 1     | 2     | 2     | 0     |
+-------+-------------+-------+-------+-------+-------+-------+-------+
| SBE   | Request     | 0     | 2     | 1     | 2     | 3     | 0     |
|       |             |       |       |       |       |       |       |
|       | \- Upheld   | 0     | 2     | 1     | 1     | 3     | 1     |
|       |             |       |       |       |       |       |       |
|       | \- Reversed | 0     | 0     | 0     | 0     | 0     | 0     |
+-------+-------------+-------+-------+-------+-------+-------+-------+
| Other | Request     | 0     | 0     | 4     | 0     | 0     | 0     |
|       |             |       |       |       |       |       |       |
|       | \- Upheld   | 0     | 0     | 4     | 0     | 0     | 0     |
|       |             |       |       |       |       |       |       |
|       | \- Reversed | 0     | 0     | 0     | 0     | 0     | 0     |
+-------+-------------+-------+-------+-------+-------+-------+-------+
| S     |             |       |       |       |       |       |       |
| econd |             |       |       |       |       |       |       |
| Level |             |       |       |       |       |       |       |
| Re    |             |       |       |       |       |       |       |
| views |             |       |       |       |       |       |       |
| (by   |             |       |       |       |       |       |       |
| D     |             |       |       |       |       |       |       |
| eputy |             |       |       |       |       |       |       |
| Direc |             |       |       |       |       |       |       |
| tor): |             |       |       |       |       |       |       |
+-------+-------------+-------+-------+-------+-------+-------+-------+
| O/DD  | Request     | 8     | 11    | 7     | 4     | 3     | 2     |
|       |             |       |       |       |       |       |       |
|       | \- Upheld   | 9     | 10    | 7     | 4     | 3     | 1     |
|       |             |       |       |       |       |       |       |
|       | \- Reversed | 0     | 0     | 0     | 0     | 0     | 0     |
+-------+-------------+-------+-------+-------+-------+-------+-------+

> Notes: The number of decisions (upheld or reversed) may not equal the
> number of requests in each year due to carryover of pending
> reconsideration request.
>
> Source: Office of the Director

**Appendix Table 8**

**Small Grants for Exploratory Research (SGER)**

**Proposals, Awards and Funding Rates**

**By Directorate, FY 1992 -- 1999**

  ----- ----------- -------- ------ ------ ------ ------ ------ ------ ------ -------------- --------------
                    Fiscal                                                    *Eight-year*   *Eight-year*
                    Year                                                                     

                    1992     1993   1994   1995   1996   1997   1998   1999   *Total*        *Average*

  NSF   Proposals   300      309    258    228    205    244    302    277    2,123          265

        Awards      194      213    185    168    144    194    250    223    1,571          196

        Funding     65%      69%    72%    74%    70%    80%    83%    81%    74%            74%
        Rate                                                                                 

  BIO   Proposals   81       80     63     56     50     44     59     48     481            60

        Awards      39       57     39     40     28     29     48     36     316            40

        Funding     48%      71%    62%    71%    56%    66%    81%    75%    66%            66%
        Rate                                                                                 

  CSE   Proposals   14       15     11     18     22     23     21     24     148            19

        Awards      9        11     9      18     19     23     20     22     131            16

        Funding     64%      73%    82%    100%   86%    100%   95%    92%    89%            89%
        Rate                                                                                 

  EHR   Proposals   1        9      5      5      1      7      9      15     52             7

        Awards      1        9      5      5      1      6      8      14     49             6

        Funding     100%     100%   100%   100%   100%   86%    89%    93%    94%            94%
        Rate                                                                                 

  ENG   Proposals   91       94     83     65     59     68     95     89     644            81

        Awards      69       67     61     52     45     57     72     75     498            62

        Funding     76%      71%    73%    80%    76%    84%    76%    84%    77%            77%
        Rate                                                                                 

  GEO   Proposals   44       41     36     28     27     40     56     44     316            40

        Awards      37       37     33     22     23     38     54     40     284            36

        Funding     84%      90%    92%    79%    85%    95%    96%    91%    90%            90%
        Rate                                                                                 

  MPS   Proposals   46       44     42     35     27     32     17     33     276            35

        Awards      21       17     25     16     12     13     10     16     130            16

        Funding     46%      39%    60%    46%    44%    41%    59%    48%    47%            47%
        Rate                                                                                 

  SBE   Proposals   21       28     12     15     14     19     30     16     155            19

        Awards      16       17     8      9      11     18     25     12     116            15

        Funding     76%      61%    67%    60%    79%    95%    83%    75%    75%            75%
        Rate                                                                                 

  OPP   Proposals   2        7      11     11     6      11     15     8      71             9

        Awards      2        7      10     11     6      10     13     8      67             8

        Funding     100%     100%   91%    100%   100%   91%    87%    100%   94%            94%
        Rate                                                                                 
  ----- ----------- -------- ------ ------ ------ ------ ------ ------ ------ -------------- --------------

Source: NSF Enterprise Information System, as of January 14, 2000.

**Appendix Table 9**

**Small Grants for Exploratory Research (SGER)**

**Funding Trends by Directorate, FY 1997 -- 1999**

  -------- ---------- ------------- -------------- -------------- --------------
                      Fiscal Year                                 *Three-year*

                      1997          1998           1999           *Total*

  NSF      Total \$   \$8,413,152   \$12,320,200   \$12,293,477   \$33,026,829

           Awards     194           250            223            667

           Average \$ \$ 43,367     \$ 49,281      \$ 55,128      \$ 49,515

  BIO      Total \$   \$1,335,223   \$2,496,514    \$1,984,457    \$ 5,816,194

           Awards     29            48             36             113

           Average \$ \$ 46,042     \$ 52,011      \$ 55,124      \$ 51,471

  CSE      Total \$   \$1,173,626   \$1,330,556    \$1,739,513    \$ 4,243,695

           Awards     23            20             22             65

           Average \$ \$ 51,027     \$ 66,528      \$ 79,069      \$ 65,288

  EHR      Total \$   \$263,762     \$597,469      \$971,346      \$ 1,832,577

           Awards     6             8              14             28

           Average \$ \$ 43,960     \$ 74,684      \$ 69,382      \$ 65,449

  ENG      Total \$   \$2,867,796   \$4,080,144    \$4,371,965    \$11,319,905

           Awards     57            72             75             204

           Average \$ \$ 50,312     \$ 56,669      \$ 58,293      \$ 55,490

  GEO      Total \$   \$1,183,592   \$2,143,438    \$1,464,750    \$ 4,791,780

           Awards     38            54             40             132

           Average \$ \$ 31,147     \$ 39,693      \$ 36,619      \$ 36,301

  MPS      Total \$   \$650,350     \$497,735      \$908,436      \$ 2,056,521

           Awards     13            10             16             39

           Average \$ \$ 50,027     \$ 49,774      \$ 56,777      \$ 52,731

  SBE      Total \$   \$625,708     \$661,043      \$534,126      \$ 1,820,877

           Awards     18            25             12             55

           Average \$ \$ 34,762     \$ 26,442      \$ 44,511      \$ 33,107

  O/D      Total \$   \$313,095     \$513,301      \$318,884      \$ 1,145,280

           Awards     10            13             8              31

           Average \$ \$ 31,310     \$ 39,485      \$ 39,861      \$ 36,945
  -------- ---------- ------------- -------------- -------------- --------------

> Source: NSF Enterprise Information System, as of January 14, 2000.

[^1]: ^^ The term "directorates" as used in this report, refers to
    NSF's seven programmatic directorates and the Office of Polar
    Programs.