![](media/image1.wmf){width="6.499305555555556in"
height="8.403472222222222in"}

## Enabling the Nation's future through 

## discovery, learning and innovation.

> [Realizing the promise of the 21^st^ century depends in large measure
> on today's investments]{.smallcaps}
>
> [in science, engineering and]{.smallcaps}
>
> [mathematics research and education.]{.smallcaps}
>
> [NSF investments -- in people, in their ideas, and in the tools they
> use -- will catalyze the strong progress in science and engineering
> needed to secure the Nation's future.]{.smallcaps}

![](media/image2.png)

## NSF is a Source of Human Capital 

> [**for People** who win Nobel Prizes-]{.smallcaps}
>
> [And who become Teachers,]{.smallcaps}
>
> [And CEOs, and Politicians.]{.smallcaps}
>
> [We foster and encourage creative thinking; we select for excellence;
> and we fund the best and the brightest based on merit. We all
> benefit-and the American people most of all.]{.smallcaps}
>
> Rita Colwell, Director, NSF

## 

## 

## Message from the Director

*I am pleased to present the National Science Foundation's Annual
Program Performance Report for fiscal year 2000, as required by the
Government Performance and Results Act of 1993 (GPRA). Here you will
learn about NSF's progress in meeting a broad range of challenging goals
that aim to improve our ability to invest in the nation's future.*

*The year 2000 marked NSF's 50^th^ Anniversary. During this half
century, knowledge about the world has exploded and the pace of
scientific discovery and technological innovation has accelerated
unabatedly. By spurring innovation, advances in science and engineering
knowledge lay the foundation for new jobs and fuel economic growth,
raising U.S. living standards and improving the quality of life.*

*Through its public investments in fundamental research and education,
the Foundation has played an important role in keeping the U.S. at the
forefront of these developments. They have brought us advances in
biomedicine, new modes of communication, and led to improvements in
public safety, agriculture, and industry. They have enabled the U.S. to
sustain a half-century of world leadership in science, engineering and
technology.*

*Just consider the following achievements highlighted in the report:*

-   *New evidence of life in extreme environments.*

-   *The Boomerang project, which brought new insights into the
    "geometry" of the universe.*

-   *New nanowires: self assembling structures that hold the potential
    to increase the capacity of computer memory chips by a factor of
    200.*

-   *Substantial increases in student achievement -- and a narrowing of
    the gap between minority and majority students -- through
    investments in educational systemic reform.*

*It is also noteworthy that of the 11 Nobel laureates announced in 2000,
six have been supported by NSF.*

*NSF's mission is to strengthen the nation's capabilities across the
entire spectrum of research and education in the sciences, engineering,
and mathematics. Each year the Foundation invests in the creative
people, the innovative ideas, and the cutting-edge technologies that
will have the highest returns in advancing discovery at the frontiers of
knowledge. It works to promote science, engineering, and mathematics
learning -- from pre-school through post-doctoral -- to prepare the next
generation of scientific talent and foster a U.S. workforce that is
second to none in the world.*

*Because the results of fundamental research are often realized only
years later as they are transformed into the products and social
benefits that improve our lives, NSF faces unique challenges in
measuring and evaluating performance. NSF has developed an evaluation
process that reflects these challenges. Each year, NSF uses the GPRA
review process as an opportunity to improve and refine the way it
measures and rates its success.*

*In conducting its FY 2000 performance review, NSF raised the bar on
performance. We increased the stringency of evaluation criteria, set
goals that were significantly beyond past performance, added new
categories of achievement, and submitted NSF activities to unprecedented
levels of internal and external review. In addition to Foundation staff,
about 400 external evaluators participated in the performance
assessment, and generated 64 reports covering 78 of the NSF's 200
programs.*

*Using these high standards, NSF met two-thirds of its 28 revised goals.
The agency met six of its eight goals relating directly to the results
of research and education, and outside evaluators noted progress toward
realizing the other two. Among the management and investment process
goals that were not met, NSF achieved substantial improvements in many
areas. Examples are increased capacity to process proposals
electronically, and an increase in the percent of awards involving new
investigators -- a key measure of the "openness of NSF's system."*

*In other areas, more stringent ratings produced results that were
actually lower than in FY 1999, despite the fact that outside evaluators
cited progress in those areas. One goal not met in FY 2000 -- NSF's
successful implementation of the new merit review criteria -- is the
result of the establishment of two new merit review criteria in FY 1998.
For this performance report, the baseline for the agency's evaluation (a
period of three years) includes one fiscal year of merit review that did
not incorporate the new criteria.*

*NSF's FY 2000 Report describes in detail the criteria NSF uses to
assure the credibility of the data used to verify and validate progress
in accomplishing its goals. This year, NSF submitted its data and
methods to independent, external review by PricewaterhouseCoopers LLP.*

*It is our aim at NSF to adhere to the highest standards of management
efficiency and integrity, and to produce outcomes of substantial benefit
to the nation. I am therefore pleased to report that the data measuring
NSF's performance that are contained in this report are complete and
reliable.*

*Sincerely,*

*Rita R. Colwell*

*Director*

# Executive Summary

This report, made pursuant to the Government Performance and Results Act
(1993), covers activities of the National Science Foundation during
Fiscal Year 2000. It is substantially more comprehensive than its
predecessor, and records an unprecedented level of effort and
achievement.

In conducting the FY 2000 assessment, NSF undertook the most rigorous
and challenging performance review in its history. Several goals and
indicators were added or revised, and evaluation criteria were made much
more stringent.

Previously used ratings such as "partially successful" or "minimally
effective" were eliminated; all outcomes were judged either "successful"
or "not successful." As a result, some of NSF's performance scores were
actually lower than in FY 1999, despite the fact that outside evaluators
cited progress in those particular areas since 1999.

In addition to Foundation staff, about 400 external evaluators
participated in the performance assessment, and generated 64 reports
covering 78 of the NSF's 200 programs. NSF engaged an independent
outside examiner, PricewaterhouseCoopers LLP, to verify data
compilations.

The Foundation aimed extremely high. For example, NSF set itself the
goal of processing 70 percent of proposals within six months of
submission. That would have been unmatched in the Foundation's history,
and would far exceed the norm at comparable granting agencies. In fact,
during FY 2000 well over half of all NSF proposals were fully processed
within six months and 71 percent were fully processed within seven
months. By most measures, that would be considered extraordinary; yet it
was not successful by NSF's high standards.

Similarly, NSF attempted to develop the technological capability that
would permit the electronic review and processing of tens of thousands
of competitive proposals each year - making it possible to do so without
generating any paperwork within the Foundation. No other research and
education funding organization in government has attempted such a feat.
NSF did encounter significant technological challenges in trying to
realize this goal, but robust progress was made and the agency will
initiate pilot projects to demonstrate its electronic review process
capability during FY 2001.

Other goals in which NSF was rated "not successful" simply may not be
achievable within a short time, such as "improved achievement in
mathematics and science skills needed by all Americans". Performance
assessment activities in FY 2000 clearly demonstrated that a goal of
reaching *all* Americans was unrealistic. Consequently, NSF will revise
its performance indicators in future years to focus on related aspects
more directly within the agency\'s responsibility and control.

However, even with new and revised goals and more exacting definitions
of success, the Foundation met two-thirds of its 28 goals, which are
divided into three broad areas: outcome goals, management goals, and
investment process goals.

## Outcomes

Outcome goals concern the practical, concrete results of NSF grants and
programs, as opposed to the procedures and methods whereby the
Foundation carries out its work.

NSF achieved 75 percent (6 out of 8) of its goals in the Outcomes
category, notably including:

-   production of "discoveries at and across the frontiers of science
    and engineering";

-   rapid and widespread connections between those discoveries and
    society as a whole;

-   ensuring that more than 80 percent of schools participating in
    education-improvement projects called "systemic initiatives" make
    substantial progress;

-   providing intensive professional development programs for at least
    65,000 teachers in grades K-12;

-   prompt compilation and electronic dissemination of essential
    national data sets; and

-   development of ways to determine and assure the quality of survey
    materials.

Two ambitious goals were rated as "not fully successful" (that is, not
successful) by reviewers. One was the objective of ensuring that NSF
grants contributed to improved scores in "mathematics and science skills
needed by all Americans," as judged by independent external evaluators.

The other was the attempt to produce a "diverse, globally oriented
workforce of scientists and engineers." Both outcomes are extremely
difficult to achieve, and neither is completely within the Foundation's
responsibility and control. In NSF\'s pursuit of both goals, however,
evaluators noted progress since FY 1999.

## Management

Management performance goals concern the effectiveness and efficiency of
the way NSF handles its workload. For FY 2000, the Foundation identified
6 specific management goals, and achieved 5 of them (83 percent).

Goals that were met included:

-   ensuring that at least 60 percent of full proposals are submitted
    electronically through the computer-based "FastLane" system;

-   increasing the total number of science and engineering hires from
    under-represented groups, as judged against the FY 1997 baseline
    (NSF achieved a 120 percent increase in female hires and a 27
    percent increase in minority hires);

-   providing FastLane orientation for all NSF staff and insisting on
    practice in key modules for at least 80 percent of program and
    support staff;

-   completing work on all "Y2K" information-technology problems as
    planned, on schedule and within budget; and

-   ensuring that at least 85 percent of all project reports are
    submitted electronically through a new, computer-based Project
    Reporting System.

Only one management performance goal was not met. NSF had set itself the
objective of having the technological capability to move competitive
proposals submitted electronically through the entire review and
award/decline process without generating any paperwork.

While this may have been extremely ambitious, significant progress was
made. By the end of FY 2000, the only significant impediment to
attaining a full electronic review capability was the development of a
secure "electronic signature". This issue will continue to be addressed
in FY 2001 when the agency will pilot ten all-electronic review
projects.

## Investment Process

Investment process goals involve the specific procedures whereby NSF
makes grants, funds and manages capital projects, and serves its
customers in general.

For FY 2000, NSF identified 15 such goals. One was found to be
inapplicable. Of the remaining 14, the Foundation clearly achieved 7, or
50 percent. Those included:

-   allocating at least 90 percent of funds to projects reviewed by
    external peer groups and selected through merit-based competition;

-   identifying possible reasons for any customer dissatisfaction with
    NSF merit review and complaint-management systems;

-   improving NSF's overall American Customer Satisfaction Index;

-   devising systems that require Principal Investigators to integrate
    educational components into their research proposals, and verifying
    the outcome;

-   developing methods of requesting and tracking reviewer answers to
    NSF merit review criterion - "what are the broader impacts of the
    proposed activity";

-   find ways to increase the number of women and under-represented
    minorities in the pool of applicants for grants; and

-   keep costs of construction and upgrades on facilities within 110
    percent of estimates.

NSF did not meet 7 of its Investment Process Goals, even though efforts
during FY 2000 often produced remarkable, measurable progress toward
achieving those goals.

For example, NSF attempted to ensure that 95 percent of program
announcements and solicitations would be available at least three months
prior to proposal deadlines or target dates. In fact, 89 percent of
announcements and solicitations met that standard - up from 75 percent
in the preceding year - and 97 percent of program announcements and
solicitations were available within 5 days of the three-month goal.

Another important area involves the goal of making at least 30 percent
of competitive research grants to new investigators. In determining its
performance on this goal, NSF counted only awards made to new Principal
Investigators. During FY 2000, 28 percent of awards were made to new
Principal Investigators -- up from 27 percent in FY 1999. However, more
than 33 percent of FY 2000 awards were made to teams of Investigators
where at least one Investigator was new -- up from 31 percent in FY
1999. So although the goal was not achieved in its strictest
interpretation, the results reported clearly demonstrate an increasing
\"openness in the system\" that the agency is committed to maintain.

Finally, NSF has committed to ensure that external merit reviewers take
*both* NSF criteria fully into account when evaluating proposals. The
two generic criteria are, simply put, scientific importance to the
individual field, and broader significance to science and society as a
whole. Although evaluators noted considerable improvement in this area,
they rated NSF's overall performance as "not fully successful\", that
is, not successful.

That outcome was not unexpected since the new merit review criteria were
implemented only in early FY 1998. In assessing the agency\'s FY 2000
performance, external evaluators examined proposals considered for
funding in FY 1997, FY 1998, FY 1999 and in a few cases, FY 2000.
Therefore, only about two thirds of proposals examined actually could
have been measured against the new merit review criteria. However,
evaluators did note that for those proposals subject to the new review
criteria (i.e. those submitted for funding consideration in FY 1998, FY
1999 and FY 2000), NSF staff had been generally successful in employing
both criteria in making funding decisions. They also noted that NSF
needs to increase its outreach efforts to the research and education
community to ensure that proposers and reviewers alike adequately
address both criteria in proposals and in the proposal review process.

Realization of this goal is increasingly likely over the next few years
as NSF effectively communicates to proposers and reviewers the
importance of addressing both criteria in proposals and reviews, and as
evaluators examine a full complement of proposals subject to these
criteria.

In four other goal areas, NSF did not successfully realize the high
standards it set.

In one, NSF had attempted to process 70 percent of proposals within six
months of receipt. Although only 54 percent of proposals were processed
within that time frame, 71 percent of proposals were processed within
seven months. This represents an accomplishment, considering the
increasing complexity and multidisciplinarity of proposals and NSF\'s
continuing commitment to external merit review. Nonetheless, the
Foundation will take steps to improve its performance in this area in FY
2001 and will again set its sights on processing 70 percent of proposals
within six months.

Another goal was to identify the best practices with which NSF staff
could explain the merit review process, answer questions, and handle
complaints.

During FY 2000, NSF conducted several customer-service surveys in an
attempt to understand and improve its performance in these areas. Three
concerns identified were: the quality and fairness of proposal review;
ease of submitting proposals via FastLane; and trouble finding the right
person to contact when questions arise. Models of best practice are
still being developed, and staff training is underway.

A third goal involved keeping facilities\' construction and upgrades
within annual schedules and assuring that total time for completion of
each phase did not exceed 110 percent of estimates. This goal is
ambitious, although it is one the agency strives to realize.
Nonetheless, in many scientific construction projects, unforeseen (and
perhaps unforeseeable) delays occur as a consequence of rapidly changing
technology and as a result of the fact that many such projects are
unique. In those circumstances, it is very difficult to produce exact
guidelines or timetables. Even so, of eleven construction and upgrade
projects supported by NSF, seven (or 64 percent) of them met the goal.

Finally, NSF had set a goal of holding operating time lost due to
unscheduled downtime at NSF-funded facilities to less than 10 percent of
total scheduled operating time. Again, that target proved difficult to
achieve because of the special problems that crop up in such projects.
Nonetheless, of 26 reporting facilities, 22 (85 percent) met the
downtime goal, and only four did not.

Based on what was learned from the FY 2000 performance assessment, NSF
will pay particular attention to certain areas in FY 2001; these include
improving customer service by increasing the percentage of proposals
processed within six months and focusing on the effective implementation
of the merit review criteria.

Using the experience gained in measuring and assessing performance in FY
2000, the agency will revise its FY 2002 performance indicators to focus
more closely on achievements for which it has responsibility and
control. The Foundation will continue to set the highest standards for
itself and to employ the most stringent criteria for review of its
accomplishments.

# Performance Reporting Requirements

**The Government Performance and Results Act of 1993** requires each
Federal agency to report no later than 180 days following the close of
each fiscal year to the President and the Congress on it's performance
for the previous fiscal year.

Each performance report must set forth performance indicators as
established in the performance plan, along with actual performance
achieved compared with the performance goals for that fiscal year.

If performance goals are specified in the alternative form, the results
shall be described in relation to the specifications, including whether
the program failed to meet the criteria of a minimally effective or
successful program.

The report for fiscal year 2000 must include actual results for the
preceding fiscal year.

According to the OMB Circular No. A-11 (2000) revised draft dated
December 22, 2000, each report must include the following elements:

1.  A comparison of actual performance with projected levels of
    performance as set out in the performance goals in the annual
    performance plan;

2.  An explanation, where a performance goal was not achieved, for why
    the goal was not met;

3.  A description of the plans and schedules to meet an unmet goal in
    the future, or alternatively, the recommended action regarding an
    unmet goal where it is concluded that it is impractical or
    unfeasible to achieve the goal;

4.  An evaluation of the performance plan for the current fiscal year,
    taking into account the actual performance achieved in the fiscal
    year covered by the report;

5.  An assessment of the reliability and completeness of the performance
    data included in the report; and

6.  Eventually, actual performance information for at least four fiscal
    years.

Other features as they apply to the agency:

a.  Program evaluations;

b.  Information on use of non-Federal parties;

c.  Classified appendices not available to the public;

d.  Description of the quality of the reported performance information;

e.  Budget information;

f.  Analysis of tax expenditures; and

g.  Waivers of administrative requirements.

# Table of Contents

Director's Statement

PAGE

Executive Summary i

Performance Reporting Requirements vi

Table of Contents vii

I.  Agency Profile and Goals 1

II. Examples of Achievements Reported in FY 2000 6

III. Assessment and Evaluation Processes 13

> A. Types of Goals 13
>
> B. Types of Assessments 14

C.  Assessment Processes 15

D.  How Assessments and Evaluations are Utilized 19

```{=html}
<!-- -->
```
IV. Summary Table of FY 2000 Performance Goals and Results 21

```{=html}
<!-- -->
```
A.  Agency Results for FY 2000 21

> B. Table of FY 2000 Performance Goals and Results 23

V.  Findings from Program Assessments and Evaluations 31

```{=html}
<!-- -->
```
A.  Outcome Goals and Results 32

B.  Management Goals and Results 66

C.  Investment Process Goals and Results 77

D.  Table of Evaluations 100

```{=html}
<!-- -->
```
VI. Verification and Validation 105

> A. Quality of the Reported Performance Information 105

B.  Data Verification and Validation Activities 106

C.  Types and Sources of Performance Data and Information 107

D.  Data Limitations 108

E.  Other Issues -- Timing 110

```{=html}
<!-- -->
```
VII. Major Management Challenges 111

VIII. Transition to FY 2001 and Beyond 113

IX\. Information on Use of Non-Federal Parties 114

X.  Budget Information 114

XI. Classified Appendices Not Available to the Public 115

XII. Analysis of Tax Expenditures 115

XIII. Waivers of Administrative Requirements 115

XIV. Appendix of Additional Examples Illustrating Outcomes of NSF
     Investments 116

XV. Schedule of Program Evaluations (FY 99, 00, 01) 131

XVI. Table of Acronyms 138

# 

# I. Agency Profile and Goals

As the world marked the beginning of a new millennium, the National
Science Foundation (NSF) celebrated its 50th year as a leader and
steward of the nation's scientific and engineering research and
education enterprise. NSF is the only Federal agency dedicated to the
support of fundamental non-medical research and education across all
science, mathematics, and engineering disciplines and for all levels of
education.

The Foundation's responsibility is in contrast to other Federal
agencies, which provide specific services or support mission-oriented
research objectives related to energy, biomedicine, or space. NSF
supports research and education via grants, contracts, and cooperative
agreements to about 1,800 colleges, universities, K-12 schools, academic
consortia, non-profit organizations, small businesses and other research
institutions -- public, private, state, local, and Federal -- throughout
the United States.

In FY 2000, approximately 95% of NSF\'s \$3.9 billion budget supported
research and education activities carried out by awardees at their home
institutions. These programs and activities directly engaged nearly
184,000 people, including researchers, educators, students, and other
professionals. To conduct the administrative work of the agency, NSF
employed a scientific and engineering staff of approximately 1250
government employees, more than 120 visiting scientists and engineers,
and over 190 contractors. NSF staff administer the merit review and
award process: they do not conduct the research nor do they operate the
laboratories supported by NSF awards.

As part of their administrative responsibility, NSF staff processed
approximately 240,000 merit-based reviews and made funding decisions on
nearly 30,000 competitive proposals submitted by applicants in FY 2000.
NSF staff processed about 9,760 new awards and 6,680 continuing awards.
New awards were selected by merit review conducted by about 50,000
external reviewers who donate tens of thousands of hours each year to
the review of proposals for research and education.

NSF provides national leadership in improving science, mathematics,
engineering and technology (SMET) education, and in broadening
participation in the SMET enterprise to prepare a diverse, globally
oriented workforce. NSF plays a major role in the development of our
nation's future scientists and engineers.

Throughout the last fifty years, NSF has worked diligently to identify
and enable the best science, mathematics, and engineering research and
education possible for the entire country. These fifty years have been
marked by path-breaking advances in science and engineering knowledge
that have spurred innovation, fueled economic growth, and led to the
highest standard of living in U.S. history. Discoveries at the frontiers
of knowledge have transformed agriculture, communications,
transportation, and industry for the benefit of the American public.
NSF-supported projects have contributed to significant improvements in a
broad array of areas -- among them education, public safety, national
defense, health, and the environment. In this report, NSF shares its
enthusiasm for its work, and hope to engage readers in a better
understanding of the NSF mission and role in the federal system. NSF's
hope is to continue to be a leader in enabling scientific discoveries,
developing people, and providing the necessary tools to advance
fundamental research and learning in all fields of science, mathematics,
engineering, and education over the next fifty years. NSF's purpose is
to ensure that future generations will enjoy sustained health,
prosperity, and a higher quality of life.

## A. Organization of Performance Goals

The Foundation's primary mission is to promote the progress of science
by ensuring that the United States maintains leadership in discovery,
learning and innovation across science, mathematics, and engineering.
NSF carries out its mission by making merit-based awards to support the
work of outstanding individuals and groups, in partnership with
colleges, universities, and other institutions -- public, private,
state, local, and Federal -- throughout the U.S. NSF awards are the
Nation's investment in individuals and organizations who ultimately
develop and produce the outcomes of the investment process that NSF
manages.

To create a high quality balanced portfolio of awards to best serve the
Nation, NSF developed the FY 2000 Annual Performance Plan and goals
based on *the NSF Strategic Plan, FY 1997-2003*. The NSF FY 2000 Annual
Performance Plan is organized into three areas of mutually supportive
goals for the agency to pursue. The FY 2000 goal areas are Outcomes,
Management, and Investment Process. NSF Strategic Plans and Annual
Performance Plans may be found on the NSF web site at
http://www.nsf.gov/od/gpra/start.htm.

The Outcome Goals address the Foundation's programmatic investments as
they relate to the agency's mission and are intended to cover the full
range of activities supported by NSF awards. The Management and
Investment Process Goals are important for the success of the Outcome
Goals. NSF's Outcome Goals are long-term goals which are difficult to
measure annually or directly, and thus are evaluated qualitatively and
by looking for progress and trends over many years. To determine
progress in achieving the Outcome Goals, NSF aggregates performance
across the agency annually and relies upon the qualitative judgement of
external experts. All goals and results achieved for FY 2000 are
described in detail in Section V of this report.

## B. Focus on Outcomes

NSF focuses its goals on long-term outcomes because they ultimately
convey the value and demonstrate the impact of what NSF does for the
American public. Each year, NSF receives nearly 30,000 proposals for
research and education projects. NSF staff make use of the merit review
process to select the most promising activities that will lead to the
best results or outcomes in the future. The total amount requested in
these proposals exceeds, by many multiples, the annual NSF budget. Given
this intense competition, NSF is able to support only one in three new
proposals each year.

Each year, NSF annually reviews the collection of results reported by
awardees during the year. However, the outcomes of NSF investments are
the long-term impacts of awards, rather than the incremental annual
progress of individual projects. Outcomes are the results which provide
the evidence of NSF's success as an investment agent for the country.
NSF believes strong performance in achievement of the Outcome Goals is
vital to the nation's future economic strength, security, and quality of
life.

## C. How NSF is Structured

The Foundation is comprised of the National Science Board (NSB) and a
Director who also serves as an *ex officio* NSB member. The NSB is
composed of 24 part-time members who are appointed by the President and
confirmed by the U. S. Senate. Members are selected on the basis of
their eminence to represent all areas of science, mathematics,
engineering and education, including basic, medical, or social sciences,
engineering, agriculture, education, research management, industry, or
public affairs. Members are expected to represent the views of the
scientific and engineering communities nationwide.

Terms of service on the NSB are six years, with no member serving more
than two consecutive terms. The National Science Board has dual
responsibilities as national science policy advisor to the President and
the Congress and as the governing body for NSF.

Other senior officials include a Deputy Director who is appointed by the
President with the advice and consent of the U.S. Senate, seven
Assistant Directors and two Office Directors. Proposals and awards are
managed by nine programmatic organizations comprised of seven
disciplinary directorates (Biological Sciences; Computer and Information
Science and Engineering; Engineering; Geosciences; Mathematical and
Physical Sciences; Social, Behavioral and Economic Sciences; and
Education and Human Resources), and two offices: the Office of Polar
Programs, and the Office of Integrative Activities. The more than 1250
NSF staff members work to coordinate NSF\'s investments with those of
other organizations, agencies and countries to provide synergy and
integration to the science and engineering enterprise of the Nation.

NSF carries out its responsibility to the nation by supporting a broad
range of activities. A few are described below:

+-----------------------------------------------+----------------------+
| -   NSF has major programs for research and   |                      |
|     education in information technology,      |                      |
|     computing, and communications.            |                      |
+-----------------------------------------------+----------------------+
| -   NSF has a unique geographic               |                      |
|     responsibility for the north and south    |                      |
|     polar regions - and plays a central role  |                      |
|     in environmental issues related to global |                      |
|     climate studies and extreme environment   |                      |
|     studies.                                  |                      |
+-----------------------------------------------+----------------------+
| -   NSF supports fundamental research in the  | NSF investments      |
|     earth, atmospheric, and ocean sciences to | provide state-of-the |
|     advance our understanding of the behavior | art                  |
|     of the Earth's atmosphere and its         | [tools]{.smallcaps}  |
|     interactions with the sun.                | for research and     |
|                                               | education, such as   |
|                                               | instrumentation,     |
|                                               | equipment, and       |
|                                               | multi-user           |
|                                               | facilities like      |
|                                               | accelerators,        |
|                                               | telescopes, research |
|                                               | vessels and          |
|                                               | aircraft, and        |
|                                               | earthquake           |
|                                               | simulators.          |
+-----------------------------------------------+----------------------+
| -   NSF focuses on strengthening the nation's |                      |
|     engineering science base in the areas of  |                      |
|     engineering systems, devices and          |                      |
|     materials, and associated engineering     |                      |
|     processes and methodologies.              |                      |
+-----------------------------------------------+----------------------+
| -   NSF supports research to advance the      |                      |
|     understanding of the underlying           |                      |
|     principles and mechanisms governing life. |                      |
|     NSF leads in supporting plant research    |                      |
|     from detailed genetic characterization    |                      |
|     and sequencing work to biochemistry, cell |                      |
|     biology, plant development, and ecology.  |                      |
+-----------------------------------------------+----------------------+
| -   NSF provides leadership in supporting     |                      |
|     researching on learning and education,    |                      |
|     improving science, mathematics,           |                      |
|     engineering and technology (SMET)         |                      |
|     education, and in broadening              |                      |
|     participation in the SMET enterprise at   |                      |
|     every educational level.                  |                      |
+-----------------------------------------------+----------------------+
| -   NSF supports a diverse portfolio of       |                      |
|     research and education in mathematics,    |                      |
|     astronomy, physics, chemistry and         |                      |
|     materials research to deepen our          |                      |
|     understanding of the physical universe    |                      |
|     and to use that understanding in service  |                      |
|     to society through training the future    |                      |
|     workforce.                                |                      |
+-----------------------------------------------+----------------------+
| -   NSF supports research to advance the      |                      |
|     understanding of the behavior of human    |                      |
|     beings and the economic, political, and   |                      |
|     social consequences of their behavior;    |                      |
|     one of the highest priorities is to       |                      |
|     improve the quality of life in the U.S.   |                      |
|     by enriching understanding of the         |                      |
|     knowledge base about people.              |                      |
+-----------------------------------------------+----------------------+
| -   NSF promotes partnerships between U.S.    |                      |
|     and foreign researchers and enhances      |                      |
|     access to critical research conducted     |                      |
|     outside the US.                           |                      |
+-----------------------------------------------+----------------------+
| -   NSF provides data and analysis on the     |                      |
|     science and engineering enterprise in the |                      |
|     U.S. from an international perspective    |                      |
|     for policy-makers, researchers, and       |                      |
|     others. NSF works with other Federal      |                      |
|     agencies, academic institutions,          |                      |
|     industry, foreign, and multi-national     |                      |
|     organizations to identify and meet key    |                      |
|     data needs for policy decision making.    |                      |
|     Key products include                      |                      |
|     Congressionally-mandated reports,         |                      |
|     statistical reports from national         |                      |
|     surveys, special topic reports, and       |                      |
|     public use data bases.                    |                      |
+-----------------------------------------------+----------------------+

# 

# [Examples of Achievements Reported in FY 2000]{.smallcaps}

NSF is proud to share just a few of the many highlights of the research
and education results reported by awardees in FY 2000. These examples
were selected as some of the most exciting discoveries and results
reported this past year. They reflect the broad range of achievements
important to NSF's mission. Each result was obtained in part or entirely
through NSF support. Highlights of research and education projects
reported by NSF's Office of Legislative and Public Affairs (OLPA) are
presented first, followed by examples reported by awardees and
recognized by external committees as noteworthy achievements in FY 2000.
Additional examples relevant to each of NSF's outcome goals are
presented in Section V.A*, "Outcome Goals and Results for FY 2000*," and
also in Section XIV, "*Appendix of Additional Examples Illustrating
Outcomes of NSF Investment*s."

## FY 2000 RESEARCH AND EDUCATION HIGHLIGHTS --

## Examples Reported by OLPA (by Title)

Below are titles for NSF supported research discoveries and results that
were reported as highlights by NSF's Office of Legislative and Public
Affairs (OLPA). To look at the stories associated with these titles, and
to look at more examples, search the NSF highlights reported at
http://www.nsf.gov/home/news.html. This web site will present more
detail, and is a source of additional stories of recent newsworthy
highlights and exciting findings.

-   Astronomers Find Evidence for the First Planet Seen Orbiting a Pair
    of Stars

-   Scientists Report First Complete DNA Sequence of Plant Chromosomes

-   Earthquake Network Intended to Help Save Lives and Money

-   Bacteria May Thrive in Antarctic Lake

-   Report Shows Students Improving in Math and Science Preparation

-   Global Seismographic Network Establishes Internet Connection to
    Remote Africa

-   Solar \"Heartbeat\" Discovered

-   Astronomers Sight an Asteroid\'s Moon

-   Exploring the Far Frontiers of Sea and Space

-   Membrane Protein Research Yields New Insights into Inner Workings of
    the Cell

-   New England Experienced \"Ice Age\" El Niño

-   Human-Computer Interaction Gets a Helping Hand, Eye, and Voice

-   Researchers Discover Evidence of Microscopic Life at the South Pole

Three examples of NSF Highlights from the OLPA Exciting Findings are:

#### Reported January 14, 2000

-   **[Scientists Report First Complete DNA Sequence of Plant
    Chromosomes]{.smallcaps}**

> Scientists involved in an international effort to sequence the entire
> genome of Arabidopsis thaliana reported the first complete DNA
> sequence of a plant chromosome in the December 16, 1999, issue of the
> journal *Nature*. The results provide new information about chromosome
> structure, evolution, intracellular signaling, and disease resistance
> in plants. The research conducted by U.S. participants was funded in
> large part by the National Science Foundation, as well as the U.S.
> Department of Agriculture and U.S. Department of Energy.

#### Reported June 22, 2000 

-   **[New England experienced "Ice-Age" el Niño]{.smallcaps}**

> The New England region underwent El Niño-like climate changes during
> the Ice Age, NSF-supported researchers have found. Scientists define
> El Niño as a disruption of the ocean-atmosphere system in the tropical
> Pacific, which has important consequences for weather around the
> globe. The team\'s findings show a strong three-to-five-year cycle of
> El Niño activity during the latter part of the last Ice Age--the same
> frequency with which El Niño occurs today.

#### Reported February 7, 2000

-   **[Twelve Pioneering Researchers Receive 1999 National Medal of
    Science]{.smallcaps}**

> On January 31, 2000, President Clinton named 12 of the nation\'s most
> respected researchers, three of them Nobel Prize winners, to receive
> the 1999 National Medal of Science. Honoring the discoveries and
> lifetime achievements of the nation\'s top scientists, the Medal of
> Science recipients named by the president represent a widely diverse
> group that: created wholly new scientific fields, such as conservation
> biology and speech sciences; led to discoveries that determined why
> the ozone \"hole\" exists; and legitimized theories about
> technological progress on economic growth, among others.

## FY 2000 RESEARCH AND EDUCATION HIGHLIGHTS

## [Examples Cited by External Evaluators]{.smallcaps}

Examples of results cited by committees of external evaluators in the
performance assessment process are presented below. These examples are
but a few of the many that were recognized as contributing to the
successful performance of NSF in FY 2000. Examples are presented here to
give the reader a more tangible feeling for the value and impact of NSF
investments. Many of these examples illustrate the broader impacts of
NSF-supported research and education activities on the Nation and its
citizens. Each example has been recognized as contributing toward the
achievement of one or more of NSF's Outcome Goals. Additional examples
illustrating outcome achievements appear in Section V.A.. "*Findings
from Program Assessments and Evaluations: Outcome Goals and Results for
FY 2000*," and also in Section XIV, as an Appendix to this report.

-   **[Nanoscience and Engineering - Nanomotors]{.smallcaps}** The
    merger of molecular biology and nano-fabrication in engineering
    research has led to the assembly of a spinning molecular motor
    through the nanoscale assembly of protein flagella onto an array of
    nickel posts. The researchers grafted these bacterial motors to an
    ordered array of nano-scale metal posts and measured the revolutions
    per second, horsepower, and motor efficiency. This is a critical
    first step in integrating biological-mechanical components with
    deliberately patterned inorganic nanostructures that will produce
    entirely new classes of more powerful nanostructured devices.
    Ultimately, the researchers envision these nanomotors powering
    nanofactories that synthesize and deliver drugs directly to the
    tissues that need them, reducing toxicity to other tissues and
    increasing the effectiveness of drug therapies.

-   **[A Flat Universe]{.smallcaps}** A spectacular burst of new
    information about the Early Universe --Cosmic Microwave Background,
    or CMB -- is transforming the field of cosmology. The CMB radiation
    is considered to be a residue from the Big Bang origin of our
    universe some 12-15 billion years ago. Using balloon-borne microwave
    detectors as a telescope, the Boomerang project is an experiment
    that maps the CMB radiation using highly sensitive arrays of
    microwave detectors. The balloon-borne telescope circumnavigates the
    Antarctic continent suspended at an altitude of 120,000 feet (36,576
    meters). The high-resolution maps of the primeval cosmic microwave
    background showed that the overall large-scale geometry of the
    universe is surprisingly flat to unprecedented accuracy -- a truly
    fundamental discovery.

-   [**Alive After 250 Million Years: Isolation of Live Permian
    Microorganisms**]{.smallcaps} Recent interdisciplinary experiments
    conducted by NSF- supported researchers on salt crystals taken from
    the Permian Salado Formation in Southeastern New Mexico, have shown
    that some ancient crystals still contain viable micro-organisms
    trapped within tiny fluid inclusions. The careful use of stringent
    geological and microbiological selection criteria support the
    hypothesis that the bacteria are at least 250 million years old. The
    salt crystal that contained the organisms was taken from an ancient
    dissolution pipe located within primary sedimentary beds 564 meters
    below ground surface. The entire sedimentary layer was examined
    before removing the crystal, to be sure that the sample was taken
    from a primary bed. Two of these inclusions held trapped,
    microorganisms that were still viable. The isolated microbes are
    salt tolerant and respond to concentrated brines by forming spores.
    One of the organisms is related to several modern day bacilli but
    does have several unique characteristics.

-   **[Life in Extreme environments]{.smallcaps}** The discovery of
    viable microbes in Lake Vostok accretion ice provided the first
    evidence that Lake Vostok, a large subglacial lake located 4 km
    (\~2.5 miles) beneath the East Antarctic ice sheet, supports a
    microbial community. Results from the Vostok work was reported
    widely in the popular media and resulted in a BBC documentary
    entitled "The Lost World". Studies on the accretionary ice of Lake
    Vostok have led to predictions of a large and diverse population of
    bacteria within the lake itself, and large interdisciplinary studies
    of Lake Vostok are planned for the near future. These investigations
    have implications for life on Earth and serve as models for future
    interplanetary investigations.

-   **[Technology spin-offs from Gravity]{.smallcaps}** Fourteen billion
    years after the Big Bang, gravity is such a weak force that
    experiments to test gravity push the frontier of technology. Gravity
    is the least tested of all known forces in nature. Thus, any advance
    in our knowledge of gravity from laboratory experiments is of key
    importance. Recent laboratory tests have pushed measurements to new
    levels of accuracy, resulting in new technology spin-offs, such as:

```{=html}
<!-- -->
```
-   Development of high power solid-state lasers, up to 120 watts;

-   Development of pre-stabilized laser power amplifiers, in
    collaboration with industry;

-   Advances in large optics and metrology with sub-Angstrom smoothness
    and losses approaching one part per million;

-   Innovative software originally designed for huge numerical
    calculations in relativity was applied to oil exploration analysis.

```{=html}
<!-- -->
```
-   **[Understanding fundamental economic phenomena]{.smallcaps}**
    Research on risk sharing and financial markets has brought new and
    deeper understanding of fundamental economic phenomena. One line of
    discovery has dealt with individual behavior. How and why
    individuals fail to fully use available financial markets to buffer
    themselves against variations in their income has had important
    implications for economic policy. Another area of research concerns
    currency crises in the 1990s, where it has been found that the usual
    cause of currency crises -- too rapid expansion of the money
    supply-- was not a factor in the Asian crisis of 1998. Perhaps the
    most important result in this field stems from a recent discovery
    known as "Taylor's Rule," which has become a powerful and effective
    monetary policy guideline. What is the optimal Federal funds rate?
    According to Taylor\'s rule, the Fed should adjust the federal funds
    rate to respond to differences between actual and desired
    performance on the Fed\'s dual objectives of price stability and
    full employment. This is done by setting the real federal funds rate
    equal to 2% plus one half the difference between actual and targeted
    inflation and one half the percentage difference between actual and
    potential GDP (assumes potential real GDP growth of 3.5%). The
    nominal funds rate should be set equal to the targeted real funds
    rate plus actual inflation. The Federal Reserve and a growing number
    of central banks use the results of this NSF project to achieve
    sustained economic growth without high rates of inflation.

-   **[Education - Improved Achievement in Mathematics and Science
    Skills]{.smallcaps}** This year NSF's continuing emphasis on
    systemic reform, teacher education and professional development has
    yielded exciting advances. In general, investments in educational
    systemic reform have led to increased achievement for a diverse
    student population and substantial narrowing of the gaps between
    minority and majority students. For example,

```{=html}
<!-- -->
```
-   Over the first six years of the Miami-Dade Urban Systemic Initiative
    (USI), the median percentile scores on the Stanford-8 test for grade
    4 students increased from 26 to 40 for African-Americans, from 26 to
    59 for Hispanics, and from 74 to 77 for Whites, showing substantial
    progress toward closing the achievement gap.

> The Systemic Initiatives have also brought about substantial increases
> in the number of students taking more challenging science and
> mathematics courses in high school. For example,

-   Over a five-year period advanced placement science enrollment in Los
    Angeles USI schools increased by 53%, compared to 17% for non-USI
    schools in the city, with remarkable increases of 196% and 146%
    respectively for African-American and Hispanic students.

```{=html}
<!-- -->
```
-   **[Training World-class Scientists in Modern
    Technologies]{.smallcaps}** NSF support provides a unique
    opportunity for undergraduates to have a \"hands-on\" exposure to
    science by working in NSF-funded laboratories. At Massachusetts Bay
    Community College, a program in biotechnology for minority students
    reaches a pool of under-served students, notably those from an urban
    community college, and provides them with opportunities for research
    at Boston University and other institutions, including field
    stations such as the Savannah River Ecology laboratory and Skidway
    Institute of Oceanography. A measure of the success of this program
    is that in the last five years it has produced eight recipients of
    the prestigious Barry M. Goldwater Scholarships for students
    planning to pursue a Ph.D. in science, mathematics or engineering.

-   **[Innovative Research and Education Experiences]{.smallcaps}** An
    important role for NSF is to catalyze innovation in the ways that we
    apply science, mathematics, and engineering. Undergraduate students
    participating in the Research Experiences for Undergraduates site at
    the Milwaukee School of Engineering helped solve a local murder case
    that had remained unsolved for 2 years. The students developed a
    technique for creating a facial image from a skull, which allowed
    police to determine the race of the victim. After this image was
    published in local newspapers, someone came forward to identify the
    victim as an immigrant from Africa, the clue that broke the case.
    The FBI is now interested in working with the Milwaukee School of
    Engineering to develop advanced forensic methods based on the
    modeling technique developed by the undergraduate students.

-   **[The Internet]{.smallcaps}** was based on experience gained from
    the Defense Advance Research Projects Agency-funded ARPANET that
    connected a few military labs and universities and established
    TCP/IP as the "language" of internetworking. Building on that
    experience, NSF funded a series of civilian network projects that
    led to the Internet of today. The CSNET of 1980 provided networking
    capability to computer science and engineering academic researchers
    and educators and encouraged this wider group of researchers and
    educators to engage in networking research and training. During the
    1980's, CSNET was expanded as the NSFNET to connect university
    researchers to the NSF funded supercomputer sites and to NCAR. This
    increased the demand and uses for packet-based networks and drew the
    private sector into developing the expertise to further expand the
    technology. Major accomplishments in the 1980's include the design
    of the architecture for Internet routers that provided the
    foundations for companies such as Cisco and Bay Networks, the domain
    name system that gives us World Wide Web addresses such as
    www.nsf.gov or www.cisco.com, and the structure for connecting
    networks of different owners. In 1992, Congress asked NSF to open
    the Internet to competition and plan for its privatization. The
    result led to the system of Internet Service Providers and backbone
    providers that is now the acclaimed Internet.[^1] Continuing
    development for the Internet includes very high-speed services, vBNS
    by NSF and Internet2 by the private sector, as well as applications
    support from NSF for scientific visualization, the sharing of
    scientific data sets, and distributed computing on "grids" of
    computers. The Internet is surely one of NSF's premier long-range
    impacts of the last decade.

-   [**Coping with Internet Traffic Growth**]{.smallcaps} Internet
    traffic is growing at an incredible pace. While optical
    communications technologies are well able to accommodate these
    increases, a severe bottleneck is the electronics implementing the
    packet routing functions. NSF-supported research in how Internet
    routers look up addresses rapidly to achieve high speed throughput
    is leading to new information to cope with speed issues. New
    techniques have decreased the address lookup time by a factor of
    eight, without having to add new hardware. The ideas have been
    patented and licensed to several routing-equipment manufacturers,
    including Lucent, GTE, NEC, Microsoft, Onex and Quary Systems. NSF's
    research support has created an entirely new approach to designing
    high speed internet routers, estimated to comprise a several billion
    dollar per year market segment.

## 

## FY 2000 Nobel Prize Winners

Of the 11 Nobelists announced in 2000, six have been supported by NSF at
some time in their careers. This is consistent with an historical
connection: of about 400 Nobelists named since 1960, 109 have received
NSF funding. This year\'s NSF-funded Nobelists - neuroscientist Paul
Greengard, physicist Herbert Kroemer, chemists Alan Heeger and Alan
MacDiarmid, and economists James Heckman and Daniel McFadden --
demonstrate and reflect the multi-disciplinary influence of NSF support.
Some have been funded by NSF for decades, and together they have
received dozens of merit-reviewed NSF grants.

James Heckman and Daniel McFadden, who share the 2000 Nobel Prize in
economics for their development of statistical methods that are widely
used in the social sciences for predicting group behavior and evaluating
the impact of public programs, have long been supported by the National
Science Foundation. Dr. Heckman has been principal investigator on 21
NSF grants and Dr. McFadden on 33 grants since the 1970s. Dr. McFadden
used his economic theory methods to help design the BART transportation
system in San Francisco, guide investments in phone service, and
allocate housing for the elderly. More recently, McFadden has developed
new conceptual approaches and statistical methods for estimating the
value of natural resources, used in such applications as quantifying the
welfare losses due to the Exxon Valdez oil spill.

For a list of 2000 Nobel laureates, see:
http://www.nobel.se/announcement/2000/index.html. For historical context
on NSF\'s connection to the Nobel prizes, see:
http://www.nsf.gov/od/lpa/news/media/2000/nsfnobels.html.

# 

# III. Assessment and Evaluation Process

## A. Types of Goals

NSF employs a mix of both qualitative and quantitative goals, and makes
use of both qualitative information and quantitative data in determining
annual progress made toward goal achievement. NSF's Outcome Goals are
expressed in a qualitative form, and most Management Goals and
Investment Process Goals are quantitative.

## B. Types of Assessments

NSF has traditionally made use of various types of assessments and
evaluations to monitor quality and process. Programs and plans are
assessed and evaluated throughout the year on a continuing basis by
internal staff and senior management. Evaluations are carried out by
externally-contracted groups to review the progress of programs in
meeting specific program objectives. NSF engages committees of external
experts or evaluators, called Committees of Visitors (COVs), and
Advisory Committees (ACs), to review program practices, processes, and
results for the Outcome Goals. The Management and Investment Process
Goals are reviewed internally by NSF staff and audited by third parties.

To determine program performance toward meeting the Outcome Goals, NSF
depends on assessments provided by the COVs and ACs. Each NSF program
has traditionally been assessed by COVs on a three-year cycle for
quality of science and integrity of process. Each year, COVs assess a
one-third portion of NSF's portfolio, looking at the program performance
over the previous three years. COVs review practices and processes, and,
with the implementation of GPRA, include an assessment of results. The
schedule for COVs has been impacted by GPRA. Programs may be clustered
together and evaluated as a group by COVs, to facilitate the assessment
process. This has led to rescheduling COVs for some programs to
accommodate the GPRA schedule.

In FY 2000, about 37% of NSF's portfolio of 200 programs were evaluated
by COVs for quality of process and progress made in achieving NSF's
Outcome Goals. Last year, about 40% of NSF's portfolio that was
evaluated by COVs. The remaining portions of NSF's portfolio will be
evaluated by COVs in FY 2001 to complete the full three-year cycle of
assessment of NSF's programs under GPRA.

Approximately 250 COV members and 150 advisory committee members
participated in the performance assessment process in FY 2000. Together,
for the Outcome Goal assessment process alone, COVs and ACs generated a
total of 64 reports which covered 78 of NSF's approximately 200 programs
(see Section XV. for a schedule of program evaluations)[^2]. We
anticipate that electronic copies of the COV and advisory committee
reports will become available in December 2001.

NSF makes use of internal data systems to monitor and report progress in
achieving the quantitative Management Goals and Investment Process
Goals. With the exception of one Investment Process Goal (Investment
Goal 2), these goals and results are assessed and reviewed by internal
management and staff rather than by external committees.

Reviews by external groups provide useful information for identifying
issues, establishing new goals, and redirecting efforts. Changes to
programs and plans may be necessitated by difficulty in meeting a goal,
lack of appropriateness of a goal, or an inability to measure a goal.
Some FY 2001 and FY 2002 goal levels are being adjusted based on FY 1999
and FY 2000 results, and on realistic expectations for future progress.
Changes of this type are handled through performance plans and internal
management. Major changes, when necessary, are handled through revisions
to the Strategic Plan, and make use of input provided by external
Advisory Committees.

## C. Assessment processes

Assessments and evaluations for gauging progress in achieving NSF's
goals involve different steps for the different types of goals. In FY
1998, NSF developed and established new reporting systems and
procedures, reporting guidelines, and templates to enable the
collection, assessment, and analysis of the qualitative information and
quantitative data necessary for reporting performance across the agency.
The systems and templates are continually upgraded and revised for
reporting using feedback from previous experiences. The reporting
templates were developed to permit more uniform and consistent reporting
of the qualitative goals across the agency.

For the quantitative Management and Investment Process Goals, the
assessment process is straightforward. The agency collects relevant data
using internal corporate data systems and compares the result with the
performance level targeted for the fiscal year. Most quantitative goals
are evaluated on a quarterly basis, with the information undergoing
review by senior management. In FY 2000, an agency-wide GPRA module for
data relevant to the quantitative goals was developed to enable staff to
follow or track their progress throughout the year.

For the qualitative Outcome Goals, NSF programs are judged by groups of
external evaluators. The following discussion focuses primarily on
Outcome Goal assessment. NSF receives and maintains performance
information in the form of reports from external COVs and ACs, whose
meetings are subject to Federal Advisory Committee Act rules.

Assessment of goal achievement by external evaluators takes into account
such factors as:

-   identified performance indicators for each performance goal;

-   the success to which NSF strategies and plans are implemented;

-   the level of resources invested;

-   external events beyond control of the agency; and

-   the agency's capability to be flexible and respond rapidly to
    emerging opportunities.

The focus of this portfolio assessment is the quality of past
investments (the quality of outputs and outcomes) and the likelihood
that the package of awards will produce strong results in the future.

Much of this performance assessment is retrospective, addressing
investments made at some time in the past. NSF performance is
*successful* if the outcomes of NSF investments reported during a fiscal
year are judged to have achieved or to have made significant progress in
achieving the specific performance goals. COVs use their collective
experienced-based norms in determining the level of "significance"
necessary for a rating of successful. COVs also address the quality of
the sets of awards made and the integrity of the process for the period
under review.

External COVs review approximately one-third of NSF's programs each
year, so that all programs have been reviewed at the end of a three-year
period. The judgements contained in COV reports and AC reports are
combined with data from internal databases, and are integrated by NSF
management to form the basis for NSF's performance report.

The flow-chart below represents the overall assessment process for the
Outcome Goals and shows how data for the Management Goals and Investment
Process Goals are submitted for integration in NSF's Performance Report.
NSF staff prepare materials as input for the COVs (reports, evaluations,
studies, highlights), for use by COVs in developing their reports and
making their assessments. The COV reports are used by NSF staff in
preparing directorate/office annual reports, and are also reviewed and
approved by directorate/office ACs. Each directorate/office must prepare
a subsequent response to the recommendations of the COVs, which is
reviewed by the directorate/office AC. Directorates/offices also prepare
an annual report for AC review (in October and November) which
summarizes activities of the directorate/office for the fiscal year in
addition to the activities examined in the COV process. The process
culminates with the AC report, which caps the annual progress of the
directorate toward achieving NSF's Outcome Goals.

### **[Outcome Results Data]{.smallcaps}**

At the close of the fiscal year, each directorate submits all
GPRA-related supporting materials (COV and AC reports, directorate
annual reports, and responses to recommendations made by COVs) to the
Office of the Director. Simultaneously, data relevant to the Management
Goals and Investment Process Goals are finalized by NSF staff and
submitted to the Office of the Director.

NSF makes use of several stages in the proposal and award process to
assess performance. These include the following steps:

-   Applicant and Grantee Information/Merit Review

> All applicants and grantees provide results from previous NSF support,
> information about existing facilities and equipment available to
> conduct the proposed research and education, location of proposed
> activities, biographical information on the primary investigators,
> other sources of support, and certifications specific to NSF.
> Information is required at the time of application, at the time of an
> award, and in annual and final project reports. Awards are made based
> on merit review by peers who are experts in the field using NSF's
> merit review criteria, and the availability of resources. Award
> decisions also take into account the quality of prior results. This
> type of information is part of the package of information made
> available to COVs when assessing program performance toward meeting
> Outcome Goals.

-   Annual Progress Review by Program Officers

> Program officers review the annual progress of awards that have a
> duration longer than one year. This review typically takes place
> before the anniversary date of the award and prior to the release of
> any continuing funds. The Principal Investigator (PI) responsible for
> the award submits the annual progress report electronically via
> FastLane. The progress report includes information on significant
> accomplishments, progress achieved in the prior year, plans for the
> next year consistent with the proposed project, and points out issues
> that may impact progress or completion of the project on schedule and
> within budget. Once this report is approved, funds for the ensuing
> year are approved by the administering program officer and released.
> Annual progress reports are made available to COVs for during the
> assessment process.

-   Program Evaluation by Committees of Visitors (COVs)

> To ensure the highest quality in processing and recommending proposals
> for awards NSF convenes committees of qualified external evaluators
> (COVs) to review each program every three years. Strict guidelines are
> followed in selecting COV members to ensure independence, programmatic
> coverage, and balanced representation. COVs are committees composed of
> independent, external experts from academia, industry, government, and
> the public sector.
>
> COVs have traditionally assessed the integrity and efficiency of the
> processes for proposal review. With the implementation of GPRA in FY
> 1999, NSF added a retrospective assessment of the quality of results
> of NSF's programs in the form of outputs and outcomes. NSF asks COVs
> to report on the noteworthy achievements of each year, to identify
> ways in which projects have collectively affected progress, and
> expectations for future performance. The recommendations of COVs are
> reviewed by management and taken into consideration by NSF when
> evaluating existing programs and future directions for the Foundation.
>
> In FY 2000, COVs were asked to judge whether NSF programs were
> successful or not in achieving Outcome Goals 1-4.a, and in
> implementing the merit review criteria (Investment Goal 2). To conduct
> their assessments, COVs use a standardized reporting template with a
> set of core questions addressing process, program management, and
> quality of outcomes. COVs are asked to justify their judgements and
> provide examples illustrating success. The results of their judgements
> are aggregated and collectively weighed to determine NSF's overall
> progress in achieving the Outcome Goals.
>
> Each COV typically consists of between six and twelve members who
> review one or more programs over a two day period. In FY 2000
> approximately 250 COV members participated in the performance
> assessment process. The Outcome Goal assessment process generated 55
> COV reports covering 78 of NSF's approximately 200 programs (see
> Section XV for a schedule of program evaluations). Typically, there
> are fewer COV reports than programs as some reports evaluate clusters
> of programs. Electronic copies of COV reports will become available in
> December, 2001.

-   Directorate/Office Assessment by Advisory Committees (ACs)

> Eight Advisory Committees (ACs) advise the seven directorates and the
> Office of Polar Programs (OPP). Advisory Committees are composed of
> external experts who have broad experience in academia, industry, and
> government. Each AC typically has 18-25 members to work with the NSF
> in assessing annual progress.
>
> Advisory committees annually review COV reports, available internal
> and external assessments, and directorate/OPP annual reports to judge
> program effectiveness and to report on strengths and weaknesses. In FY
> 2000 approximately 150 advisory committee members participated in the
> performance assessment process. Each AC reviews the portfolio of its
> respective directorate/office for progress in achieving NSF's Outcome
> Goals.
>
> In their assessment capacity ACs respond to a set of standardized
> questions developed by NSF staff to indicate the success of the
> directorate/OPP in achieving each Outcome goal, and to provide a
> justification for their assessment. NSF management reviews the eight
> Advisory Committee reports and integrates the assessments into the NSF
> Annual Performance Report.

## 

## Aggregation of Committee reports

All COVs and Advisory Committees are asked to complete a report template
with questions addressing how well programs achieve NSF's goals.
Committees are asked to address (A) the integrity and efficiency of the
**processes** which involve proposal review; and (B) the quality of the
**results** of NSF's investments. However, each goal may not apply to
each program being evaluated, and therefore a goal may not be rated in
every report. For example, in FY 2000 only 58% of all reports gave a
rating for Goal 4.a. In many cases where the goals are not rated,
committee reports provided comments indicating either full success,
limited success, or indicated that the goal did not apply to the
programs under review. In some cases the committee may not have had
adequate information to provide a rating. Information may not have been
available where programs were too new to have produced results for this
report period.

Most committees provided a rating with sufficient information to justify
how they arrived at the rating. Some committees provided comments that
could be used to determine the success of a program. This year, to
arrive at an aggregated result for the agency, comments provided in
reports were used when ratings for a goal were not evident, provided
that the comments were clearly relevant and sufficiently well justified.
In cases where a high rating was assigned, but comments indicated
performance was not fully successful, the comments were used to
determine a rating. Committee reports were reviewed and results were
tabulated for each goal. A tabulated summary was produced which combined
results across the agency for each goal. In FY 2000, the tabulated
summary was audited by PricewaterhouseCoopers LLP for Outcome Goals 1,
2, 3, and 4.a, and Investment Process Goal 2.

## D. How Assessments and Evaluations are Utilized

NSF management reviews program annual reports, reports by COVs, reports
by ACs, and reports by other external groups or organizations. NSF
management writes a response to the performance issues raised in COV
reports, and submits this response to the relevant AC for review,
identifying steps that will be implemented to address specific issues
raised in the COV reports. The NSF management review the recommendations
and issues that are raised in the COV and AC reports in order to
identify management concerns as well as areas of scientific opportunity.
COV and AC reports address a broad range of issues ranging from staffing
and quality of merit review to specifics of a scientific issue. They
extend beyond the scope of the GPRA goals, and have traditionally been
used by NSF management to improve program performance and set funding
priorities.

##### 

##### IV. Summary Table of FY 2000 Performance Goals and Results

## A. Agency Results for FY 2000

Overall, NSF was successful in achieving 64% -- 18 of 28 -- of its
performance goals. Results in this second year are consistent with those
obtained in FY 1999. For example, the following areas were identified as
needing improvement: (1) use of both merit review criteria by reviewers
and applicants; (2) customer service goals such as decreasing time to
decision on proposals; and (3) increasing participation of
under-represented groups. NSF will continue to focus on achieving
improved performance in these areas in FY 2001 and beyond.

NSF was more rigorous this year in evaluating goal achievement than last
year. Options for grading the qualitative Outcome Goals were limited to
either *successful* or *not successful* and justification was required
for *successful* ratings.

Characterization of NSF's performance for goals stated in the
alternative form in FY 2000 benefited from changes made in presentation,
process, and use of information from external committees of experts.
Changes resulted, in part, from reviews of NSF's FY 1999 Performance
Report made by the Administration, Congress, and the private sector that
raised issues of quality of information, appropriate justification of
ratings, and that questioned NSF's use of a two-tiered rating schema. To
alleviate these concerns, NSF engaged an external firm in FY 2000,
PricewaterhouseCoopers LLP, to verify achievement data for most goals.

+----------------------------------+-----------------------------------+
| #                                |                                   |
| ## FY 2000 Aggregated Performanc |                                   |
| e Results {#fy-2000-aggregated-p |                                   |
| erformance-results .Box-Heading} |                                   |
+----------------------------------+-----------------------------------+
|                                  | > **Number of Goals Achieved**    |
+----------------------------------+-----------------------------------+
| **Outcome Goals**                | 6 out of 8 (75%)                  |
+----------------------------------+-----------------------------------+
| **Management Goals**             | 5 out of 6 (83%)                  |
+----------------------------------+-----------------------------------+
| > **Investment Process Goals**   | 7 out of 14 (50%);\               |
|                                  | one goal did not apply            |
+----------------------------------+-----------------------------------+
| > **TOTAL**                      | 18 out of 28 (64%)                |
+----------------------------------+-----------------------------------+

Aggregated performance results for the agency are presented in brief
summary form for each goal in Table 1. Each goal is defined in the
table, and results for the goal are briefly stated. A more complete
discussion of results for each goal is presented in Section V.

##### Results for Outcome Goals

Six of the eight Outcome Goals (75%) were achieved in FY 2000. External
evaluators indicated that NSF successfully achieved Outcome Goals 1 and
2. Information provided by external evaluators indicated that NSF did
not achieve Outcome Goals 3 and 4.a, although they noted that NSF
performance had improved. NSF also achieved the quantitative Outcome
Goals 4.b, 4.c, 5.a and 5.b.

External evaluators commented that programs are showing improvement over
FY 1999 performance in the area of increasing diversity through
increased participation of under-represented groups. However, they
indicated that participation remains lower than expected. Evaluators
commented that increasing participation of under-represented groups is
an area needing more attention by NSF. In addition, evaluators noted
that some NSF program portfolios should include more "high risk"
activities. Common issues emerged that could result in reduced program
performance; these issues include increasing workload issues and delays
in processing proposals (see Investment Process Goal 7). Further
discussion of Outcome Goal results and how NSF is addressing these
issues as well as others is provided in Section V.A.

##### Results for Management Goals

Five of NSF\'s six Management Goals (83%) were achieved in FY 2000.
Performance improvements were identified in the orientation and training
of NSF staff using FastLane, NSF\'s electronic system for proposal
submission, proposal review, project reporting, and increasing the use
of the electronic Project Reporting System for project reporting by
awardees.

One Management Goal was not achieved involving NSF's development of the
technological capability to submit proposals electronically. The
difficulty encountered in FY 2000 which prevented this goal from being
achieved, was the establishment of electronic signature protocols. In FY
2001 NSF will pilot ten full electronic review projects to assess the
effectiveness of its electronic signature protocols. Further discussion
of the Management Goals is provided in Section V.B.

##### Results for Investment Process Goals

Seven of NSF\'s fifteen Investment Process Goals were achieved in FY
2000; seven were not achieved and one of the facility goals did not
apply (because there were no construction projects completed during the
year). Areas identified as needing improvement include: use of the new
merit review criteria by reviewers and applicants; identifying best
practices and training for improving customer service; allowing three
months time to prepare proposals; decreasing the time to decision;
increasing the percentage of awards to new investigators; maintaining
facility upgrades and construction on schedule; and keeping operating
time lost due to unscheduled downtime to less than 10% of the total
scheduled operating time. Discussion of these goals and how NSF is
addressing issues is provided in Section V.C.

Table 1 provides a brief summary of the results for each FY 2000 goal
which appeared in the NSF FY 2000 Performance Plan. For more detail and
discussion of results and goals, see Section V of this report.

### Table 1. FY 2000 Performance Goals and Results

##### ANNUAL PERFORMANCE GOALS and RESULTS FOR OUTCOMES

+--------------+------------------------------+------------------------+
| ###          | ### Performance Goals for    | ### Results {          |
| Outcomes  {# |  Outcomes {#performance-goal | #results .Box-Heading} |
| outcomes-1 . | s-for-outcomes .Box-Heading} |                        |
| Box-Heading} |                              |                        |
+--------------+------------------------------+------------------------+
|              | *NSF is judged successful    |                        |
|              | when*                        |                        |
+--------------+------------------------------+------------------------+
| Outcome Goal | Performance Goal 1:          | **Baseline:**          |
| 1:           |                              | Experiments using FY   |
|              | NSF awards lead to important | 1997 and FY 1998       |
| Discoveries  | discoveries; new knowledge   | information indicated  |
| at and       | and techniques, both         | successful             |
| across the   | expected and unexpected,     | achievement.           |
| frontier of  | within and across            |                        |
| science and  | traditional disciplinary     | **FY 1999:** Goal      |
| engineering  | boundaries; and              | achieved. Judged       |
|              | high-potential links across  | successful by external |
|              | these boundaries, as judged  | experts in all         |
|              | by independent external      | reports.               |
|              | experts.                     |                        |
|              |                              | **FY 2000:** Goal      |
|              |                              | achieved. Reports by   |
|              |                              | external experts       |
|              |                              | indicate that in the   |
|              |                              | aggregate NSF is       |
|              |                              | successful in          |
|              |                              | achieving this goal.   |
+--------------+------------------------------+------------------------+
| Outcome Goal | Performance Goal 2:          | **Baseline:**          |
| 2:           |                              | Experiments using FY   |
|              | The results of NSF awards    | 1997 and FY 1998       |
| Connections  | are rapidly and readily      | information indicated  |
| between      | available and feed, as       | successful             |
| discoveries  | appropriate, into education, | achievement.           |
| and their    | policy development, or use   |                        |
| use in       | by other federal agencies or | **FY 1999:** Goal      |
| service to   | the private sector, as       | achieved. Judged       |
| society      | judged by independent        | successful in the      |
|              | external experts.            | aggregate by external  |
|              |                              | experts who noted      |
|              |                              | improvements can be    |
|              |                              | made in some programs. |
|              |                              |                        |
|              |                              | **FY 2000:** Goal      |
|              |                              | achieved. Judged       |
|              |                              | successful in the      |
|              |                              | aggregate by external  |
|              |                              | experts who noted      |
|              |                              | improvements can be    |
|              |                              | made in some programs, |
|              |                              | as in FY 1999.         |
+--------------+------------------------------+------------------------+
| Outcome Goal | Performance Goal 3:          | **Baseline**:          |
| 3:           |                              | Experiments using FY   |
|              | Participants in NSF          | 1997 and FY 1998       |
| A diverse,   | activities experience        | information indicated  |
| globa        | world-class professional     | successful             |
| lly-oriented | practices in research and    | achievement.           |
| workforce of | education, using modern      |                        |
| scientists   | technologies and             | **FY 1999**: Goal      |
| and          | incorporating international  | achieved**.** Judged   |
| engineers    | points of reference; when    | successful in most     |
|              | academia, government,        | areas by external      |
|              | business, and industry       | experts.               |
|              | recognize their quality; and |                        |
|              | when the science and         | **FY 2000**: Judged    |
|              | engineering workforce shows  | successful in a        |
|              | increased participation of   | limited context-- goal |
|              | under-represented groups, as | not fully achieved in  |
|              | judged by independent        | the aggregate. Most    |
|              | external experts.            | programs with specific |
|              |                              | responsibilities for   |
|              |                              | these areas were       |
|              |                              | judged successful.     |
|              |                              | Improvements still     |
|              |                              | needed in the same     |
|              |                              | areas that were        |
|              |                              | identified in FY 1999. |
+--------------+------------------------------+------------------------+

###  ANNUAL PERFORMANCE GOALS and RESULTS FOR OUTCOMES -- continued

+--------------+---+------------------------------+---+-----------------------+
| ###          | # |                              | # |                       |
| Outcomes  {# | # |                              | # |                       |
| outcomes-2 . | # |                              | # |                       |
| Box-Heading} |   |                              |   |                       |
|              | P |                              | R |                       |
|              | e |                              | e |                       |
|              | r |                              | s |                       |
|              | f |                              | u |                       |
|              | o |                              | l |                       |
|              | r |                              | t |                       |
|              | m |                              | s |                       |
|              | a |                              |   |                       |
|              | n |                              | { |                       |
|              | c |                              | # |                       |
|              | e |                              | r |                       |
|              |   |                              | e |                       |
|              | G |                              | s |                       |
|              | o |                              | u |                       |
|              | a |                              | l |                       |
|              | l |                              | t |                       |
|              | s |                              | s |                       |
|              |   |                              | - |                       |
|              | f |                              | 1 |                       |
|              | o |                              |   |                       |
|              | r |                              | . |                       |
|              |   |                              | B |                       |
|              | O |                              | o |                       |
|              | u |                              | x |                       |
|              | t |                              | - |                       |
|              | c |                              | H |                       |
|              | o |                              | e |                       |
|              | m |                              | a |                       |
|              | e |                              | d |                       |
|              | s |                              | i |                       |
|              |   |                              | n |                       |
|              | { |                              | g |                       |
|              | # |                              | } |                       |
|              | p |                              |   |                       |
|              | e |                              |   |                       |
|              | r |                              |   |                       |
|              | f |                              |   |                       |
|              | o |                              |   |                       |
|              | r |                              |   |                       |
|              | m |                              |   |                       |
|              | a |                              |   |                       |
|              | n |                              |   |                       |
|              | c |                              |   |                       |
|              | e |                              |   |                       |
|              | - |                              |   |                       |
|              | g |                              |   |                       |
|              | o |                              |   |                       |
|              | a |                              |   |                       |
|              | l |                              |   |                       |
|              | s |                              |   |                       |
|              | - |                              |   |                       |
|              | f |                              |   |                       |
|              | o |                              |   |                       |
|              | r |                              |   |                       |
|              | - |                              |   |                       |
|              | o |                              |   |                       |
|              | u |                              |   |                       |
|              | t |                              |   |                       |
|              | c |                              |   |                       |
|              | o |                              |   |                       |
|              | m |                              |   |                       |
|              | e |                              |   |                       |
|              | s |                              |   |                       |
|              | - |                              |   |                       |
|              | 1 |                              |   |                       |
|              |   |                              |   |                       |
|              | . |                              |   |                       |
|              | B |                              |   |                       |
|              | o |                              |   |                       |
|              | x |                              |   |                       |
|              | - |                              |   |                       |
|              | H |                              |   |                       |
|              | e |                              |   |                       |
|              | a |                              |   |                       |
|              | d |                              |   |                       |
|              | i |                              |   |                       |
|              | n |                              |   |                       |
|              | g |                              |   |                       |
|              | } |                              |   |                       |
+--------------+---+------------------------------+---+-----------------------+
| Outcome Goal |   | Performance Goal 4.a:        |   | **Baseline:**         |
| 4:           |   |                              |   | Preliminary pilot     |
|              |   | NSF awards lead to the       |   | efforts did not       |
| Improved     |   | development, adoption,       |   | provide sufficient    |
| achievement  |   | adaptation, and              |   | information to yield  |
| in           |   | implementation of effective  |   | a valid baseline.     |
| mathematics  |   | models, products, and        |   |                       |
| and science  |   | practices that address the   |   | **FY 1999:** Goal     |
| skills       |   | needs of all students;       |   | achieved. Judged      |
| needed by    |   | well-trained teachers who    |   | successful in the     |
| all          |   | implement standards-based    |   | aggregate by external |
| Americans    |   | approaches in their          |   | experts for programs  |
|              |   | classrooms; and improved     |   | to which goal         |
|              |   | student performance in       |   | applies.              |
|              |   | participating schools and    |   |                       |
|              |   | districts, as judged by      |   | **FY 2000:** NSF was  |
|              |   | independent external         |   | judged successful in  |
|              |   | experts.                     |   | a limited context in  |
|              |   |                              |   | the aggregate, and    |
|              |   |                              |   | reports this goal as  |
|              |   |                              |   | not fully achieved    |
|              |   |                              |   | overall. NSF was      |
|              |   |                              |   | successful where      |
|              |   |                              |   | programs had clear    |
|              |   |                              |   | objectives directed   |
|              |   |                              |   | toward this goal.     |
+--------------+---+------------------------------+---+-----------------------+
|              |   | Performance Goal 4.b:        |   | **FY 1999:** Goal     |
|              |   |                              |   | achieved.             |
|              |   | Over 80 percent of schools   |   |                       |
|              |   | participating in a systemic  |   | **FY 2000:** Goal     |
|              |   | initiative program will:     |   | achieved.             |
|              |   |                              |   |                       |
|              |   | (1) implement a              |   |                       |
|              |   |     standards-based          |   |                       |
|              |   |     curriculum in science    |   |                       |
|              |   |     and mathematics;         |   |                       |
|              |   |                              |   |                       |
|              |   | (2) further professional     |   |                       |
|              |   |     development of the       |   |                       |
|              |   |     instructional workforce; |   |                       |
|              |   |     and                      |   |                       |
|              |   |                              |   |                       |
|              |   | (3) improve student          |   |                       |
|              |   |     achievement on a         |   |                       |
|              |   |     selected battery of      |   |                       |
|              |   |     tests, after three years |   |                       |
|              |   |     of NSF support.          |   |                       |
+--------------+---+------------------------------+---+-----------------------+
|              |   | Performance Goal 4.c:        |   | **FY 1999:** Goal     |
|              |   |                              |   | achieved.             |
|              |   | Through systemic initiatives |   |                       |
|              |   | and related teacher          |   | **FY 2000:** Goal     |
|              |   | enhancement programs, NSF    |   | achieved.             |
|              |   | will provide intensive       |   |                       |
|              |   | professional development     |   |                       |
|              |   | experiences annually for at  |   |                       |
|              |   | least 65,000 precollege      |   |                       |
|              |   | teachers.                    |   |                       |
+--------------+---+------------------------------+---+-----------------------+

###  

### ANNUAL PERFORMANCE GOALS and RESULTS FOR OUTCOMES-- continued

+----------------+------------------------------------+----------------+
| ### Outc       | ### Performance                    | ### Resul      |
| ome  {#outcome |  Goals for Outcomes {#performance- | ts {#results-2 |
|  .Box-Heading} | goals-for-outcomes-2 .Box-Heading} |  .Box-Heading} |
+----------------+------------------------------------+----------------+
| Outcome Goal   | Performance Goal 5.a:              | **Baseline**:  |
| 5:             |                                    | 540 days in    |
|                | Maintain FY 1999 gains in          | 1995-1996.     |
| Timely and     | timeliness for an average of 486   |                |
| relevant       | days as the time interval between  | **FY 1999:**   |
| information on | reference period (the time to      | Goal achieved. |
| the national   | which the data refer) and          |                |
| and            | reporting of data.                 | **FY 2000:**   |
| international  |                                    | Goal achieved. |
| science and    | > [FY 1999-2000]{.underline}       |                |
| engineering    | >                                  |                |
| enterprise.    | > Goal 486 days                    |                |
|                | >                                  |                |
|                | > Actual 461 days                  |                |
+----------------+------------------------------------+----------------+
|                | Performance Goal 5.b:              | **Baseline:**  |
|                |                                    | New goal in FY |
|                | Establish a standard set of data   | 2000.          |
|                | quality measures for reporting of  |                |
|                | Science Resource Studies products. | **FY 2000:**   |
|                | Prepare reports on these measures  | Goal achieved. |
|                | for all SRS surveys and publish    |                |
|                | them in electronic formats to      |                |
|                | inform users of SRS data quality.  |                |
|                | New in FY 2000, replacing the FY   |                |
|                | 1999 goal on relevance.            |                |
+----------------+------------------------------------+----------------+

##### ANNUAL PERFORMANCE GOALS and RESULTS FOR MANAGEMENT

+------------+------------------------------------------+-------------+
| ###        | ### P                                    | ###         |
|  Performan | erformance Goals for Management {#perfor | Results {#r |
| ce Area {# | mance-goals-for-management .Box-Heading} | esults-3 .B |
| performanc |                                          | ox-Heading} |
| e-area .Bo |                                          |             |
| x-Heading} |                                          |             |
+------------+------------------------------------------+-------------+
| New and    |                                          |             |
| emerging   |                                          |             |
| te         |                                          |             |
| chnologies |                                          |             |
+------------+------------------------------------------+-------------+
| Electronic | Management Goal 1:                       | **FY        |
| proposal   |                                          | 1999:**     |
| submission | NSF will receive at least 60% of full    | Goal        |
|            | proposal submissions electronically      | achieved.   |
|            | through FastLane.                        |             |
|            |                                          | **FY        |
|            | > [FY 1997]{.underline} [FY              | 2000:**     |
|            | > 1998]{.underline} [FY                  | Goal        |
|            | > 1999]{.underline} [FY                  | achieved.   |
|            | > 2000]{.underline} [FY                  |             |
|            | > 2001]{.underline}                      |             |
|            |                                          |             |
|            | Baseline 4.4% 17%                        |             |
|            |                                          |             |
|            | Goal 25% 60% 95%                         |             |
|            |                                          |             |
|            | Result 44% 81%                           |             |
+------------+------------------------------------------+-------------+
| Electronic | Management Goal 2:                       | New goal in |
| proposal   |                                          | FY 2000.    |
| processing | By the end of FY 2000, NSF will have the |             |
|            | technological capability to take         | **FY        |
|            | competitive proposals submitted          | 2000:**     |
|            | electronically through the entire        | Goal not    |
|            | proposal and award/declination process   | achieved.   |
|            | without generating paper within NSF.     |             |
+------------+------------------------------------------+-------------+

### ANNUAL PERFORMANCE GOALS and RESULTS FOR MANAGEMENT -- continued

### 

+------------+---+------------------------------------------+---+-------------+
| ### P      |   | ### Per                                  |   | ###         |
| erformance |   | formance Goals for Management {#performa |   | Results {#r |
|  Area {#pe |   | nce-goals-for-management-1 .Box-Heading} |   | esults-4 .B |
| rformance- |   |                                          |   | ox-Heading} |
| area-1 .Bo |   |                                          |   |             |
| x-Heading} |   |                                          |   |             |
+------------+---+------------------------------------------+---+-------------+
| NSF Staff  |   |                                          |   |             |
+------------+---+------------------------------------------+---+-------------+
| Diversity  | M |                                          | N |             |
|            | a |                                          | e |             |
|            | n |                                          | w |             |
|            | a |                                          | g |             |
|            | g |                                          | o |             |
|            | e |                                          | a |             |
|            | m |                                          | l |             |
|            | e |                                          | i |             |
|            | n |                                          | n |             |
|            | t |                                          | F |             |
|            | G |                                          | Y |             |
|            | o |                                          | 2 |             |
|            | a |                                          | 0 |             |
|            | l |                                          | 0 |             |
|            | 3 |                                          | 0 |             |
|            | : |                                          | . |             |
|            |   |                                          |   |             |
|            | I |                                          | * |             |
|            | n |                                          | * |             |
|            | F |                                          | F |             |
|            | Y |                                          | Y |             |
|            | 2 |                                          | 2 |             |
|            | 0 |                                          | 0 |             |
|            | 0 |                                          | 0 |             |
|            | 0 |                                          | 0 |             |
|            | , |                                          | : |             |
|            | N |                                          | * |             |
|            | S |                                          | * |             |
|            | F |                                          | G |             |
|            | w |                                          | o |             |
|            | i |                                          | a |             |
|            | l |                                          | l |             |
|            | l |                                          | a |             |
|            | s |                                          | c |             |
|            | h |                                          | h |             |
|            | o |                                          | i |             |
|            | w |                                          | e |             |
|            | a |                                          | v |             |
|            | n |                                          | e |             |
|            | i |                                          | d |             |
|            | n |                                          | . |             |
|            | c |                                          |   |             |
|            | r |                                          |   |             |
|            | e |                                          |   |             |
|            | a |                                          |   |             |
|            | s |                                          |   |             |
|            | e |                                          |   |             |
|            | o |                                          |   |             |
|            | v |                                          |   |             |
|            | e |                                          |   |             |
|            | r |                                          |   |             |
|            | 1 |                                          |   |             |
|            | 9 |                                          |   |             |
|            | 9 |                                          |   |             |
|            | 7 |                                          |   |             |
|            | i |                                          |   |             |
|            | n |                                          |   |             |
|            | t |                                          |   |             |
|            | h |                                          |   |             |
|            | e |                                          |   |             |
|            | t |                                          |   |             |
|            | o |                                          |   |             |
|            | t |                                          |   |             |
|            | a |                                          |   |             |
|            | l |                                          |   |             |
|            | n |                                          |   |             |
|            | u |                                          |   |             |
|            | m |                                          |   |             |
|            | b |                                          |   |             |
|            | e |                                          |   |             |
|            | r |                                          |   |             |
|            | o |                                          |   |             |
|            | f |                                          |   |             |
|            | h |                                          |   |             |
|            | i |                                          |   |             |
|            | r |                                          |   |             |
|            | e |                                          |   |             |
|            | s |                                          |   |             |
|            | t |                                          |   |             |
|            | o |                                          |   |             |
|            | S |                                          |   |             |
|            | & |                                          |   |             |
|            | E |                                          |   |             |
|            | p |                                          |   |             |
|            | o |                                          |   |             |
|            | s |                                          |   |             |
|            | i |                                          |   |             |
|            | t |                                          |   |             |
|            | i |                                          |   |             |
|            | o |                                          |   |             |
|            | n |                                          |   |             |
|            | s |                                          |   |             |
|            | f |                                          |   |             |
|            | r |                                          |   |             |
|            | o |                                          |   |             |
|            | m |                                          |   |             |
|            | u |                                          |   |             |
|            | n |                                          |   |             |
|            | d |                                          |   |             |
|            | e |                                          |   |             |
|            | r |                                          |   |             |
|            | - |                                          |   |             |
|            | r |                                          |   |             |
|            | e |                                          |   |             |
|            | p |                                          |   |             |
|            | r |                                          |   |             |
|            | e |                                          |   |             |
|            | s |                                          |   |             |
|            | e |                                          |   |             |
|            | n |                                          |   |             |
|            | t |                                          |   |             |
|            | e |                                          |   |             |
|            | d |                                          |   |             |
|            | g |                                          |   |             |
|            | r |                                          |   |             |
|            | o |                                          |   |             |
|            | u |                                          |   |             |
|            | p |                                          |   |             |
|            | s |                                          |   |             |
|            | . |                                          |   |             |
|            |   |                                          |   |             |
|            | * |                                          |   |             |
|            | * |                                          |   |             |
|            | F |                                          |   |             |
|            | Y |                                          |   |             |
|            | 1 |                                          |   |             |
|            | 9 |                                          |   |             |
|            | 9 |                                          |   |             |
|            | 7 |                                          |   |             |
|            | B |                                          |   |             |
|            | a |                                          |   |             |
|            | s |                                          |   |             |
|            | e |                                          |   |             |
|            | l |                                          |   |             |
|            | i |                                          |   |             |
|            | n |                                          |   |             |
|            | e |                                          |   |             |
|            | * |                                          |   |             |
|            | * |                                          |   |             |
|            | : |                                          |   |             |
|            | O |                                          |   |             |
|            | f |                                          |   |             |
|            | S |                                          |   |             |
|            | & |                                          |   |             |
|            | E |                                          |   |             |
|            | h |                                          |   |             |
|            | i |                                          |   |             |
|            | r |                                          |   |             |
|            | e |                                          |   |             |
|            | s |                                          |   |             |
|            | i |                                          |   |             |
|            | n |                                          |   |             |
|            | 1 |                                          |   |             |
|            | 9 |                                          |   |             |
|            | 9 |                                          |   |             |
|            | 7 |                                          |   |             |
|            | , |                                          |   |             |
|            | 1 |                                          |   |             |
|            | 6 |                                          |   |             |
|            | w |                                          |   |             |
|            | e |                                          |   |             |
|            | r |                                          |   |             |
|            | e |                                          |   |             |
|            | f |                                          |   |             |
|            | e |                                          |   |             |
|            | m |                                          |   |             |
|            | a |                                          |   |             |
|            | l |                                          |   |             |
|            | e |                                          |   |             |
|            | a |                                          |   |             |
|            | n |                                          |   |             |
|            | d |                                          |   |             |
|            | 1 |                                          |   |             |
|            | 5 |                                          |   |             |
|            | w |                                          |   |             |
|            | e |                                          |   |             |
|            | r |                                          |   |             |
|            | e |                                          |   |             |
|            | f |                                          |   |             |
|            | r |                                          |   |             |
|            | o |                                          |   |             |
|            | m |                                          |   |             |
|            | u |                                          |   |             |
|            | n |                                          |   |             |
|            | d |                                          |   |             |
|            | e |                                          |   |             |
|            | r |                                          |   |             |
|            | - |                                          |   |             |
|            | r |                                          |   |             |
|            | e |                                          |   |             |
|            | p |                                          |   |             |
|            | r |                                          |   |             |
|            | e |                                          |   |             |
|            | s |                                          |   |             |
|            | e |                                          |   |             |
|            | n |                                          |   |             |
|            | t |                                          |   |             |
|            | e |                                          |   |             |
|            | d |                                          |   |             |
|            | m |                                          |   |             |
|            | i |                                          |   |             |
|            | n |                                          |   |             |
|            | o |                                          |   |             |
|            | r |                                          |   |             |
|            | i |                                          |   |             |
|            | t |                                          |   |             |
|            | y |                                          |   |             |
|            | g |                                          |   |             |
|            | r |                                          |   |             |
|            | o |                                          |   |             |
|            | u |                                          |   |             |
|            | p |                                          |   |             |
|            | s |                                          |   |             |
|            | . |                                          |   |             |
|            |   |                                          |   |             |
|            | * |                                          |   |             |
|            | * |                                          |   |             |
|            | F |                                          |   |             |
|            | Y |                                          |   |             |
|            | 2 |                                          |   |             |
|            | 0 |                                          |   |             |
|            | 0 |                                          |   |             |
|            | 0 |                                          |   |             |
|            | R |                                          |   |             |
|            | e |                                          |   |             |
|            | s |                                          |   |             |
|            | u |                                          |   |             |
|            | l |                                          |   |             |
|            | t |                                          |   |             |
|            | * |                                          |   |             |
|            | * |                                          |   |             |
|            | : |                                          |   |             |
|            | O |                                          |   |             |
|            | f |                                          |   |             |
|            | 1 |                                          |   |             |
|            | 1 |                                          |   |             |
|            | 3 |                                          |   |             |
|            | S |                                          |   |             |
|            | & |                                          |   |             |
|            | E |                                          |   |             |
|            | h |                                          |   |             |
|            | i |                                          |   |             |
|            | r |                                          |   |             |
|            | e |                                          |   |             |
|            | s |                                          |   |             |
|            | , |                                          |   |             |
|            | 3 |                                          |   |             |
|            | 5 |                                          |   |             |
|            | w |                                          |   |             |
|            | e |                                          |   |             |
|            | r |                                          |   |             |
|            | e |                                          |   |             |
|            | f |                                          |   |             |
|            | e |                                          |   |             |
|            | m |                                          |   |             |
|            | a |                                          |   |             |
|            | l |                                          |   |             |
|            | e |                                          |   |             |
|            | a |                                          |   |             |
|            | n |                                          |   |             |
|            | d |                                          |   |             |
|            | 1 |                                          |   |             |
|            | 9 |                                          |   |             |
|            | w |                                          |   |             |
|            | e |                                          |   |             |
|            | r |                                          |   |             |
|            | e |                                          |   |             |
|            | f |                                          |   |             |
|            | r |                                          |   |             |
|            | o |                                          |   |             |
|            | m |                                          |   |             |
|            | m |                                          |   |             |
|            | i |                                          |   |             |
|            | n |                                          |   |             |
|            | o |                                          |   |             |
|            | r |                                          |   |             |
|            | i |                                          |   |             |
|            | t |                                          |   |             |
|            | y |                                          |   |             |
|            | g |                                          |   |             |
|            | r |                                          |   |             |
|            | o |                                          |   |             |
|            | u |                                          |   |             |
|            | p |                                          |   |             |
|            | s |                                          |   |             |
|            | . |                                          |   |             |
|            | C |                                          |   |             |
|            | o |                                          |   |             |
|            | m |                                          |   |             |
|            | p |                                          |   |             |
|            | a |                                          |   |             |
|            | r |                                          |   |             |
|            | e |                                          |   |             |
|            | d |                                          |   |             |
|            | w |                                          |   |             |
|            | i |                                          |   |             |
|            | t |                                          |   |             |
|            | h |                                          |   |             |
|            | F |                                          |   |             |
|            | Y |                                          |   |             |
|            | 1 |                                          |   |             |
|            | 9 |                                          |   |             |
|            | 9 |                                          |   |             |
|            | 7 |                                          |   |             |
|            | b |                                          |   |             |
|            | a |                                          |   |             |
|            | s |                                          |   |             |
|            | e |                                          |   |             |
|            | l |                                          |   |             |
|            | i |                                          |   |             |
|            | n |                                          |   |             |
|            | e |                                          |   |             |
|            | , |                                          |   |             |
|            | t |                                          |   |             |
|            | h |                                          |   |             |
|            | i |                                          |   |             |
|            | s |                                          |   |             |
|            | r |                                          |   |             |
|            | e |                                          |   |             |
|            | p |                                          |   |             |
|            | r |                                          |   |             |
|            | e |                                          |   |             |
|            | s |                                          |   |             |
|            | e |                                          |   |             |
|            | n |                                          |   |             |
|            | t |                                          |   |             |
|            | s |                                          |   |             |
|            | a |                                          |   |             |
|            | 1 |                                          |   |             |
|            | 2 |                                          |   |             |
|            | 0 |                                          |   |             |
|            | % |                                          |   |             |
|            | i |                                          |   |             |
|            | n |                                          |   |             |
|            | c |                                          |   |             |
|            | r |                                          |   |             |
|            | e |                                          |   |             |
|            | a |                                          |   |             |
|            | s |                                          |   |             |
|            | e |                                          |   |             |
|            | i |                                          |   |             |
|            | n |                                          |   |             |
|            | f |                                          |   |             |
|            | e |                                          |   |             |
|            | m |                                          |   |             |
|            | a |                                          |   |             |
|            | l |                                          |   |             |
|            | e |                                          |   |             |
|            | h |                                          |   |             |
|            | i |                                          |   |             |
|            | r |                                          |   |             |
|            | e |                                          |   |             |
|            | s |                                          |   |             |
|            | a |                                          |   |             |
|            | n |                                          |   |             |
|            | d |                                          |   |             |
|            | a |                                          |   |             |
|            | 2 |                                          |   |             |
|            | 7 |                                          |   |             |
|            | % |                                          |   |             |
|            | i |                                          |   |             |
|            | n |                                          |   |             |
|            | c |                                          |   |             |
|            | r |                                          |   |             |
|            | e |                                          |   |             |
|            | a |                                          |   |             |
|            | s |                                          |   |             |
|            | e |                                          |   |             |
|            | i |                                          |   |             |
|            | n |                                          |   |             |
|            | m |                                          |   |             |
|            | i |                                          |   |             |
|            | n |                                          |   |             |
|            | o |                                          |   |             |
|            | r |                                          |   |             |
|            | i |                                          |   |             |
|            | t |                                          |   |             |
|            | y |                                          |   |             |
|            | h |                                          |   |             |
|            | i |                                          |   |             |
|            | r |                                          |   |             |
|            | e |                                          |   |             |
|            | s |                                          |   |             |
|            | . |                                          |   |             |
+------------+---+------------------------------------------+---+-------------+
| Capability | M |                                          | * |             |
| in use of  | a |                                          | * |             |
| i          | n |                                          | F |             |
| nformation | a |                                          | Y |             |
| technology | g |                                          | 1 |             |
|            | e |                                          | 9 |             |
|            | m |                                          | 9 |             |
|            | e |                                          | 9 |             |
|            | n |                                          | : |             |
|            | t |                                          | * |             |
|            | G |                                          | * |             |
|            | o |                                          | G |             |
|            | a |                                          | o |             |
|            | l |                                          | a |             |
|            | 4 |                                          | l |             |
|            | : |                                          | n |             |
|            |   |                                          | o |             |
|            | B |                                          | t |             |
|            | y |                                          | a |             |
|            | t |                                          | c |             |
|            | h |                                          | h |             |
|            | e |                                          | i |             |
|            | e |                                          | e |             |
|            | n |                                          | v |             |
|            | d |                                          | e |             |
|            | o |                                          | d |             |
|            | f |                                          | . |             |
|            | F |                                          |   |             |
|            | Y |                                          | * |             |
|            | 2 |                                          | * |             |
|            | 0 |                                          | F |             |
|            | 0 |                                          | Y |             |
|            | 0 |                                          | 2 |             |
|            | , |                                          | 0 |             |
|            | a |                                          | 0 |             |
|            | l |                                          | 0 |             |
|            | l |                                          | : |             |
|            | s |                                          | * |             |
|            | t |                                          | * |             |
|            | a |                                          | G |             |
|            | f |                                          | o |             |
|            | f |                                          | a |             |
|            | w |                                          | l |             |
|            | i |                                          | a |             |
|            | l |                                          | c |             |
|            | l |                                          | h |             |
|            | r |                                          | i |             |
|            | e |                                          | e |             |
|            | c |                                          | v |             |
|            | e |                                          | e |             |
|            | i |                                          | d |             |
|            | v |                                          | . |             |
|            | e |                                          |   |             |
|            | a |                                          |   |             |
|            | n |                                          |   |             |
|            | o |                                          |   |             |
|            | r |                                          |   |             |
|            | i |                                          |   |             |
|            | e |                                          |   |             |
|            | n |                                          |   |             |
|            | t |                                          |   |             |
|            | a |                                          |   |             |
|            | t |                                          |   |             |
|            | i |                                          |   |             |
|            | o |                                          |   |             |
|            | n |                                          |   |             |
|            | t |                                          |   |             |
|            | o |                                          |   |             |
|            | F |                                          |   |             |
|            | a |                                          |   |             |
|            | s |                                          |   |             |
|            | t |                                          |   |             |
|            | L |                                          |   |             |
|            | a |                                          |   |             |
|            | n |                                          |   |             |
|            | e |                                          |   |             |
|            | , |                                          |   |             |
|            | a |                                          |   |             |
|            | n |                                          |   |             |
|            | d |                                          |   |             |
|            | a |                                          |   |             |
|            | t |                                          |   |             |
|            | l |                                          |   |             |
|            | e |                                          |   |             |
|            | a |                                          |   |             |
|            | s |                                          |   |             |
|            | t |                                          |   |             |
|            | 8 |                                          |   |             |
|            | 0 |                                          |   |             |
|            | % |                                          |   |             |
|            | o |                                          |   |             |
|            | f |                                          |   |             |
|            | p |                                          |   |             |
|            | r |                                          |   |             |
|            | o |                                          |   |             |
|            | g |                                          |   |             |
|            | r |                                          |   |             |
|            | a |                                          |   |             |
|            | m |                                          |   |             |
|            | a |                                          |   |             |
|            | n |                                          |   |             |
|            | d |                                          |   |             |
|            | p |                                          |   |             |
|            | r |                                          |   |             |
|            | o |                                          |   |             |
|            | g |                                          |   |             |
|            | r |                                          |   |             |
|            | a |                                          |   |             |
|            | m |                                          |   |             |
|            | s |                                          |   |             |
|            | u |                                          |   |             |
|            | p |                                          |   |             |
|            | p |                                          |   |             |
|            | o |                                          |   |             |
|            | r |                                          |   |             |
|            | t |                                          |   |             |
|            | s |                                          |   |             |
|            | t |                                          |   |             |
|            | a |                                          |   |             |
|            | f |                                          |   |             |
|            | f |                                          |   |             |
|            | w |                                          |   |             |
|            | i |                                          |   |             |
|            | l |                                          |   |             |
|            | l |                                          |   |             |
|            | r |                                          |   |             |
|            | e |                                          |   |             |
|            | c |                                          |   |             |
|            | e |                                          |   |             |
|            | i |                                          |   |             |
|            | v |                                          |   |             |
|            | e |                                          |   |             |
|            | p |                                          |   |             |
|            | r |                                          |   |             |
|            | a |                                          |   |             |
|            | c |                                          |   |             |
|            | t |                                          |   |             |
|            | i |                                          |   |             |
|            | c |                                          |   |             |
|            | e |                                          |   |             |
|            | i |                                          |   |             |
|            | n |                                          |   |             |
|            | u |                                          |   |             |
|            | s |                                          |   |             |
|            | i |                                          |   |             |
|            | n |                                          |   |             |
|            | g |                                          |   |             |
|            | i |                                          |   |             |
|            | t |                                          |   |             |
|            | s |                                          |   |             |
|            | k |                                          |   |             |
|            | e |                                          |   |             |
|            | y |                                          |   |             |
|            | m |                                          |   |             |
|            | o |                                          |   |             |
|            | d |                                          |   |             |
|            | u |                                          |   |             |
|            | l |                                          |   |             |
|            | e |                                          |   |             |
|            | s |                                          |   |             |
|            | . |                                          |   |             |
|            |   |                                          |   |             |
|            | > |                                          |   |             |
|            |   |                                          |   |             |
|            | [ |                                          |   |             |
|            | O |                                          |   |             |
|            | r |                                          |   |             |
|            | i |                                          |   |             |
|            | e |                                          |   |             |
|            | n |                                          |   |             |
|            | t |                                          |   |             |
|            | a |                                          |   |             |
|            | t |                                          |   |             |
|            | i |                                          |   |             |
|            | o |                                          |   |             |
|            | n |                                          |   |             |
|            | ] |                                          |   |             |
|            | { |                                          |   |             |
|            | . |                                          |   |             |
|            | u |                                          |   |             |
|            | n |                                          |   |             |
|            | d |                                          |   |             |
|            | e |                                          |   |             |
|            | r |                                          |   |             |
|            | l |                                          |   |             |
|            | i |                                          |   |             |
|            | n |                                          |   |             |
|            | e |                                          |   |             |
|            | } |                                          |   |             |
|            | > |                                          |   |             |
|            |   |                                          |   |             |
|            | [ |                                          |   |             |
|            | F |                                          |   |             |
|            | Y |                                          |   |             |
|            | > |                                          |   |             |
|            |   |                                          |   |             |
|            | 1 |                                          |   |             |
|            | 9 |                                          |   |             |
|            | 9 |                                          |   |             |
|            | 9 |                                          |   |             |
|            | ] |                                          |   |             |
|            | { |                                          |   |             |
|            | . |                                          |   |             |
|            | u |                                          |   |             |
|            | n |                                          |   |             |
|            | d |                                          |   |             |
|            | e |                                          |   |             |
|            | r |                                          |   |             |
|            | l |                                          |   |             |
|            | i |                                          |   |             |
|            | n |                                          |   |             |
|            | e |                                          |   |             |
|            | } |                                          |   |             |
|            | > |                                          |   |             |
|            |   |                                          |   |             |
|            | [ |                                          |   |             |
|            | F |                                          |   |             |
|            | Y |                                          |   |             |
|            | > |                                          |   |             |
|            |   |                                          |   |             |
|            | 2 |                                          |   |             |
|            | 0 |                                          |   |             |
|            | 0 |                                          |   |             |
|            | 0 |                                          |   |             |
|            | ] |                                          |   |             |
|            | { |                                          |   |             |
|            | . |                                          |   |             |
|            | u |                                          |   |             |
|            | n |                                          |   |             |
|            | d |                                          |   |             |
|            | e |                                          |   |             |
|            | r |                                          |   |             |
|            | l |                                          |   |             |
|            | i |                                          |   |             |
|            | n |                                          |   |             |
|            | e |                                          |   |             |
|            | } |                                          |   |             |
|            | > |                                          |   |             |
|            |   |                                          |   |             |
|            | [ |                                          |   |             |
|            | F |                                          |   |             |
|            | Y |                                          |   |             |
|            | > |                                          |   |             |
|            |   |                                          |   |             |
|            | 2 |                                          |   |             |
|            | 0 |                                          |   |             |
|            | 0 |                                          |   |             |
|            | 1 |                                          |   |             |
|            | ] |                                          |   |             |
|            | { |                                          |   |             |
|            | . |                                          |   |             |
|            | u |                                          |   |             |
|            | n |                                          |   |             |
|            | d |                                          |   |             |
|            | e |                                          |   |             |
|            | r |                                          |   |             |
|            | l |                                          |   |             |
|            | i |                                          |   |             |
|            | n |                                          |   |             |
|            | e |                                          |   |             |
|            | } |                                          |   |             |
|            | > |                                          |   |             |
|            | > |                                          |   |             |
|            |   |                                          |   |             |
|            | G |                                          |   |             |
|            | o |                                          |   |             |
|            | a |                                          |   |             |
|            | l |                                          |   |             |
|            | > |                                          |   |             |
|            |   |                                          |   |             |
|            | 1 |                                          |   |             |
|            | 0 |                                          |   |             |
|            | 0 |                                          |   |             |
|            | % |                                          |   |             |
|            | > |                                          |   |             |
|            |   |                                          |   |             |
|            | 1 |                                          |   |             |
|            | 0 |                                          |   |             |
|            | 0 |                                          |   |             |
|            | % |                                          |   |             |
|            | > |                                          |   |             |
|            |   |                                          |   |             |
|            | \ |                                          |   |             |
|            | * |                                          |   |             |
|            | > |                                          |   |             |
|            | > |                                          |   |             |
|            |   |                                          |   |             |
|            | R |                                          |   |             |
|            | e |                                          |   |             |
|            | s |                                          |   |             |
|            | u |                                          |   |             |
|            | l |                                          |   |             |
|            | t |                                          |   |             |
|            | > |                                          |   |             |
|            |   |                                          |   |             |
|            | 8 |                                          |   |             |
|            | 0 |                                          |   |             |
|            | % |                                          |   |             |
|            | > |                                          |   |             |
|            |   |                                          |   |             |
|            | 1 |                                          |   |             |
|            | 0 |                                          |   |             |
|            | 0 |                                          |   |             |
|            | % |                                          |   |             |
|            | > |                                          |   |             |
|            | > |                                          |   |             |
|            |   |                                          |   |             |
|            | [ |                                          |   |             |
|            | T |                                          |   |             |
|            | r |                                          |   |             |
|            | a |                                          |   |             |
|            | i |                                          |   |             |
|            | n |                                          |   |             |
|            | i |                                          |   |             |
|            | n |                                          |   |             |
|            | g |                                          |   |             |
|            | ] |                                          |   |             |
|            | { |                                          |   |             |
|            | . |                                          |   |             |
|            | u |                                          |   |             |
|            | n |                                          |   |             |
|            | d |                                          |   |             |
|            | e |                                          |   |             |
|            | r |                                          |   |             |
|            | l |                                          |   |             |
|            | i |                                          |   |             |
|            | n |                                          |   |             |
|            | e |                                          |   |             |
|            | } |                                          |   |             |
|            | > |                                          |   |             |
|            | > |                                          |   |             |
|            |   |                                          |   |             |
|            | G |                                          |   |             |
|            | o |                                          |   |             |
|            | a |                                          |   |             |
|            | l |                                          |   |             |
|            | > |                                          |   |             |
|            |   |                                          |   |             |
|            | 9 |                                          |   |             |
|            | 5 |                                          |   |             |
|            | % |                                          |   |             |
|            | > |                                          |   |             |
|            |   |                                          |   |             |
|            | 8 |                                          |   |             |
|            | 0 |                                          |   |             |
|            | % |                                          |   |             |
|            | > |                                          |   |             |
|            |   |                                          |   |             |
|            | \ |                                          |   |             |
|            | * |                                          |   |             |
|            | > |                                          |   |             |
|            | > |                                          |   |             |
|            |   |                                          |   |             |
|            | R |                                          |   |             |
|            | e |                                          |   |             |
|            | s |                                          |   |             |
|            | u |                                          |   |             |
|            | l |                                          |   |             |
|            | t |                                          |   |             |
|            | > |                                          |   |             |
|            |   |                                          |   |             |
|            | 4 |                                          |   |             |
|            | 3 |                                          |   |             |
|            | % |                                          |   |             |
|            | > |                                          |   |             |
|            |   |                                          |   |             |
|            | 9 |                                          |   |             |
|            | 0 |                                          |   |             |
|            | % |                                          |   |             |
|            |   |                                          |   |             |
|            | \ |                                          |   |             |
|            | * |                                          |   |             |
|            | w |                                          |   |             |
|            | i |                                          |   |             |
|            | l |                                          |   |             |
|            | l |                                          |   |             |
|            | n |                                          |   |             |
|            | o |                                          |   |             |
|            | t |                                          |   |             |
|            | b |                                          |   |             |
|            | e |                                          |   |             |
|            | s |                                          |   |             |
|            | h |                                          |   |             |
|            | o |                                          |   |             |
|            | w |                                          |   |             |
|            | n |                                          |   |             |
|            | a |                                          |   |             |
|            | s |                                          |   |             |
|            | a |                                          |   |             |
|            | g |                                          |   |             |
|            | o |                                          |   |             |
|            | a |                                          |   |             |
|            | l |                                          |   |             |
|            | i |                                          |   |             |
|            | n |                                          |   |             |
|            | F |                                          |   |             |
|            | Y |                                          |   |             |
|            | 2 |                                          |   |             |
|            | 0 |                                          |   |             |
|            | 0 |                                          |   |             |
|            | 1 |                                          |   |             |
+------------+---+------------------------------------------+---+-------------+
| Impl       |   |                                          |   |             |
| ementation |   |                                          |   |             |
| of         |   |                                          |   |             |
| management |   |                                          |   |             |
| reforms    |   |                                          |   |             |
+------------+---+------------------------------------------+---+-------------+
| Year 2000  | M |                                          | * |             |
|            | a |                                          | * |             |
|            | n |                                          | F |             |
|            | a |                                          | Y |             |
|            | g |                                          | 1 |             |
|            | e |                                          | 9 |             |
|            | m |                                          | 9 |             |
|            | e |                                          | 9 |             |
|            | n |                                          | : |             |
|            | t |                                          | * |             |
|            | G |                                          | * |             |
|            | o |                                          | G |             |
|            | a |                                          | o |             |
|            | l |                                          | a |             |
|            | 5 |                                          | l |             |
|            | : |                                          | a |             |
|            |   |                                          | c |             |
|            | N |                                          | h |             |
|            | S |                                          | i |             |
|            | F |                                          | e |             |
|            | w |                                          | v |             |
|            | i |                                          | e |             |
|            | l |                                          | d |             |
|            | l |                                          | ; |             |
|            | c |                                          | r |             |
|            | o |                                          | e |             |
|            | m |                                          | v |             |
|            | p |                                          | i |             |
|            | l |                                          | s |             |
|            | e |                                          | e |             |
|            | t |                                          | d |             |
|            | e |                                          | f |             |
|            | a |                                          | o |             |
|            | l |                                          | r |             |
|            | l |                                          | F |             |
|            | a |                                          | Y |             |
|            | c |                                          | 2 |             |
|            | t |                                          | 0 |             |
|            | i |                                          | 0 |             |
|            | v |                                          | 0 |             |
|            | i |                                          | . |             |
|            | t |                                          |   |             |
|            | i |                                          | * |             |
|            | e |                                          | * |             |
|            | s |                                          | F |             |
|            | n |                                          | Y |             |
|            | e |                                          | 2 |             |
|            | e |                                          | 0 |             |
|            | d |                                          | 0 |             |
|            | e |                                          | 0 |             |
|            | d |                                          | : |             |
|            | t |                                          | * |             |
|            | o |                                          | * |             |
|            | a |                                          | G |             |
|            | d |                                          | o |             |
|            | d |                                          | a |             |
|            | r |                                          | l |             |
|            | e |                                          | a |             |
|            | s |                                          | c |             |
|            | s |                                          | h |             |
|            | t |                                          | i |             |
|            | h |                                          | e |             |
|            | e |                                          | v |             |
|            | Y |                                          | e |             |
|            | e |                                          | d |             |
|            | a |                                          | . |             |
|            | r |                                          |   |             |
|            | 2 |                                          |   |             |
|            | 0 |                                          |   |             |
|            | 0 |                                          |   |             |
|            | 0 |                                          |   |             |
|            | p |                                          |   |             |
|            | r |                                          |   |             |
|            | o |                                          |   |             |
|            | b |                                          |   |             |
|            | l |                                          |   |             |
|            | e |                                          |   |             |
|            | m |                                          |   |             |
|            | f |                                          |   |             |
|            | o |                                          |   |             |
|            | r |                                          |   |             |
|            | i |                                          |   |             |
|            | t |                                          |   |             |
|            | s |                                          |   |             |
|            | i |                                          |   |             |
|            | n |                                          |   |             |
|            | f |                                          |   |             |
|            | o |                                          |   |             |
|            | r |                                          |   |             |
|            | m |                                          |   |             |
|            | a |                                          |   |             |
|            | t |                                          |   |             |
|            | i |                                          |   |             |
|            | o |                                          |   |             |
|            | n |                                          |   |             |
|            | s |                                          |   |             |
|            | y |                                          |   |             |
|            | s |                                          |   |             |
|            | t |                                          |   |             |
|            | e |                                          |   |             |
|            | m |                                          |   |             |
|            | s |                                          |   |             |
|            | a |                                          |   |             |
|            | c |                                          |   |             |
|            | c |                                          |   |             |
|            | o |                                          |   |             |
|            | r |                                          |   |             |
|            | d |                                          |   |             |
|            | i |                                          |   |             |
|            | n |                                          |   |             |
|            | g |                                          |   |             |
|            | t |                                          |   |             |
|            | o |                                          |   |             |
|            | p |                                          |   |             |
|            | l |                                          |   |             |
|            | a |                                          |   |             |
|            | n |                                          |   |             |
|            | , |                                          |   |             |
|            | o |                                          |   |             |
|            | n |                                          |   |             |
|            | s |                                          |   |             |
|            | c |                                          |   |             |
|            | h |                                          |   |             |
|            | e |                                          |   |             |
|            | d |                                          |   |             |
|            | u |                                          |   |             |
|            | l |                                          |   |             |
|            | e |                                          |   |             |
|            | a |                                          |   |             |
|            | n |                                          |   |             |
|            | d |                                          |   |             |
|            | w |                                          |   |             |
|            | i |                                          |   |             |
|            | t |                                          |   |             |
|            | h |                                          |   |             |
|            | i |                                          |   |             |
|            | n |                                          |   |             |
|            | b |                                          |   |             |
|            | u |                                          |   |             |
|            | d |                                          |   |             |
|            | g |                                          |   |             |
|            | e |                                          |   |             |
|            | t |                                          |   |             |
|            | . |                                          |   |             |
|            |   |                                          |   |             |
|            | * |                                          |   |             |
|            | * |                                          |   |             |
|            | R |                                          |   |             |
|            | e |                                          |   |             |
|            | s |                                          |   |             |
|            | u |                                          |   |             |
|            | l |                                          |   |             |
|            | t |                                          |   |             |
|            | : |                                          |   |             |
|            | * |                                          |   |             |
|            | * |                                          |   |             |
|            | A |                                          |   |             |
|            | l |                                          |   |             |
|            | l |                                          |   |             |
|            | a |                                          |   |             |
|            | c |                                          |   |             |
|            | t |                                          |   |             |
|            | i |                                          |   |             |
|            | v |                                          |   |             |
|            | i |                                          |   |             |
|            | t |                                          |   |             |
|            | i |                                          |   |             |
|            | e |                                          |   |             |
|            | s |                                          |   |             |
|            | n |                                          |   |             |
|            | e |                                          |   |             |
|            | e |                                          |   |             |
|            | d |                                          |   |             |
|            | e |                                          |   |             |
|            | d |                                          |   |             |
|            | t |                                          |   |             |
|            | o |                                          |   |             |
|            | a |                                          |   |             |
|            | d |                                          |   |             |
|            | d |                                          |   |             |
|            | r |                                          |   |             |
|            | e |                                          |   |             |
|            | s |                                          |   |             |
|            | s |                                          |   |             |
|            | t |                                          |   |             |
|            | h |                                          |   |             |
|            | e |                                          |   |             |
|            | Y |                                          |   |             |
|            | e |                                          |   |             |
|            | a |                                          |   |             |
|            | r |                                          |   |             |
|            | 2 |                                          |   |             |
|            | 0 |                                          |   |             |
|            | 0 |                                          |   |             |
|            | 0 |                                          |   |             |
|            | p |                                          |   |             |
|            | r |                                          |   |             |
|            | o |                                          |   |             |
|            | b |                                          |   |             |
|            | l |                                          |   |             |
|            | e |                                          |   |             |
|            | m |                                          |   |             |
|            | w |                                          |   |             |
|            | e |                                          |   |             |
|            | r |                                          |   |             |
|            | e |                                          |   |             |
|            | c |                                          |   |             |
|            | o |                                          |   |             |
|            | m |                                          |   |             |
|            | p |                                          |   |             |
|            | l |                                          |   |             |
|            | e |                                          |   |             |
|            | t |                                          |   |             |
|            | e |                                          |   |             |
|            | d |                                          |   |             |
|            | a |                                          |   |             |
|            | c |                                          |   |             |
|            | c |                                          |   |             |
|            | o |                                          |   |             |
|            | r |                                          |   |             |
|            | d |                                          |   |             |
|            | i |                                          |   |             |
|            | n |                                          |   |             |
|            | g |                                          |   |             |
|            | t |                                          |   |             |
|            | o |                                          |   |             |
|            | p |                                          |   |             |
|            | l |                                          |   |             |
|            | a |                                          |   |             |
|            | n |                                          |   |             |
|            | , |                                          |   |             |
|            | o |                                          |   |             |
|            | n |                                          |   |             |
|            | s |                                          |   |             |
|            | c |                                          |   |             |
|            | h |                                          |   |             |
|            | e |                                          |   |             |
|            | d |                                          |   |             |
|            | u |                                          |   |             |
|            | l |                                          |   |             |
|            | e |                                          |   |             |
|            | , |                                          |   |             |
|            | a |                                          |   |             |
|            | n |                                          |   |             |
|            | d |                                          |   |             |
|            | w |                                          |   |             |
|            | i |                                          |   |             |
|            | t |                                          |   |             |
|            | h |                                          |   |             |
|            | i |                                          |   |             |
|            | n |                                          |   |             |
|            | b |                                          |   |             |
|            | u |                                          |   |             |
|            | d |                                          |   |             |
|            | g |                                          |   |             |
|            | e |                                          |   |             |
|            | t |                                          |   |             |
|            | . |                                          |   |             |
+------------+---+------------------------------------------+---+-------------+
| Project    | M |                                          | * |             |
| Reporting  | a |                                          | * |             |
| System     | n |                                          | F |             |
|            | a |                                          | Y |             |
|            | g |                                          | 1 |             |
|            | e |                                          | 9 |             |
|            | m |                                          | 9 |             |
|            | e |                                          | 9 |             |
|            | n |                                          | : |             |
|            | t |                                          | * |             |
|            | G |                                          | * |             |
|            | o |                                          | G |             |
|            | a |                                          | o |             |
|            | l |                                          | a |             |
|            | 6 |                                          | l |             |
|            | : |                                          | a |             |
|            |   |                                          | c |             |
|            | D |                                          | h |             |
|            | u |                                          | i |             |
|            | r |                                          | e |             |
|            | i |                                          | v |             |
|            | n |                                          | e |             |
|            | g |                                          | d |             |
|            | F |                                          | ; |             |
|            | Y |                                          | t |             |
|            | 2 |                                          | a |             |
|            | 0 |                                          | r |             |
|            | 0 |                                          | g |             |
|            | 0 |                                          | e |             |
|            | , |                                          | t |             |
|            | a |                                          | r |             |
|            | t |                                          | e |             |
|            | l |                                          | v |             |
|            | e |                                          | i |             |
|            | a |                                          | s |             |
|            | s |                                          | e |             |
|            | t |                                          | d |             |
|            | 8 |                                          | f |             |
|            | 5 |                                          | o |             |
|            | % |                                          | r |             |
|            | o |                                          | F |             |
|            | f |                                          | Y |             |
|            | a |                                          | 2 |             |
|            | l |                                          | 0 |             |
|            | l |                                          | 0 |             |
|            | p |                                          | 0 |             |
|            | r |                                          | . |             |
|            | o |                                          |   |             |
|            | j |                                          | * |             |
|            | e |                                          | * |             |
|            | c |                                          | F |             |
|            | t |                                          | Y |             |
|            | r |                                          | 2 |             |
|            | e |                                          | 0 |             |
|            | p |                                          | 0 |             |
|            | o |                                          | 0 |             |
|            | r |                                          | : |             |
|            | t |                                          | * |             |
|            | s |                                          | * |             |
|            | w |                                          | G |             |
|            | i |                                          | o |             |
|            | l |                                          | a |             |
|            | l |                                          | l |             |
|            | b |                                          | a |             |
|            | e |                                          | c |             |
|            | s |                                          | h |             |
|            | u |                                          | i |             |
|            | b |                                          | e |             |
|            | m |                                          | v |             |
|            | i |                                          | e |             |
|            | t |                                          | d |             |
|            | t |                                          | . |             |
|            | e |                                          |   |             |
|            | d |                                          |   |             |
|            | t |                                          |   |             |
|            | h |                                          |   |             |
|            | r |                                          |   |             |
|            | o |                                          |   |             |
|            | u |                                          |   |             |
|            | g |                                          |   |             |
|            | h |                                          |   |             |
|            | t |                                          |   |             |
|            | h |                                          |   |             |
|            | e |                                          |   |             |
|            | n |                                          |   |             |
|            | e |                                          |   |             |
|            | w |                                          |   |             |
|            | e |                                          |   |             |
|            | l |                                          |   |             |
|            | e |                                          |   |             |
|            | c |                                          |   |             |
|            | t |                                          |   |             |
|            | r |                                          |   |             |
|            | o |                                          |   |             |
|            | n |                                          |   |             |
|            | i |                                          |   |             |
|            | c |                                          |   |             |
|            | P |                                          |   |             |
|            | r |                                          |   |             |
|            | o |                                          |   |             |
|            | j |                                          |   |             |
|            | e |                                          |   |             |
|            | c |                                          |   |             |
|            | t |                                          |   |             |
|            | R |                                          |   |             |
|            | e |                                          |   |             |
|            | p |                                          |   |             |
|            | o |                                          |   |             |
|            | r |                                          |   |             |
|            | t |                                          |   |             |
|            | i |                                          |   |             |
|            | n |                                          |   |             |
|            | g |                                          |   |             |
|            | S |                                          |   |             |
|            | y |                                          |   |             |
|            | s |                                          |   |             |
|            | t |                                          |   |             |
|            | e |                                          |   |             |
|            | m |                                          |   |             |
|            | . |                                          |   |             |
|            |   |                                          |   |             |
|            | > |                                          |   |             |
|            |   |                                          |   |             |
|            | [ |                                          |   |             |
|            | F |                                          |   |             |
|            | Y |                                          |   |             |
|            | > |                                          |   |             |
|            |   |                                          |   |             |
|            | 1 |                                          |   |             |
|            | 9 |                                          |   |             |
|            | 9 |                                          |   |             |
|            | 9 |                                          |   |             |
|            | ] |                                          |   |             |
|            | { |                                          |   |             |
|            | . |                                          |   |             |
|            | u |                                          |   |             |
|            | n |                                          |   |             |
|            | d |                                          |   |             |
|            | e |                                          |   |             |
|            | r |                                          |   |             |
|            | l |                                          |   |             |
|            | i |                                          |   |             |
|            | n |                                          |   |             |
|            | e |                                          |   |             |
|            | } |                                          |   |             |
|            | > |                                          |   |             |
|            |   |                                          |   |             |
|            | [ |                                          |   |             |
|            | F |                                          |   |             |
|            | Y |                                          |   |             |
|            | > |                                          |   |             |
|            |   |                                          |   |             |
|            | 2 |                                          |   |             |
|            | 0 |                                          |   |             |
|            | 0 |                                          |   |             |
|            | 0 |                                          |   |             |
|            | ] |                                          |   |             |
|            | { |                                          |   |             |
|            | . |                                          |   |             |
|            | u |                                          |   |             |
|            | n |                                          |   |             |
|            | d |                                          |   |             |
|            | e |                                          |   |             |
|            | r |                                          |   |             |
|            | l |                                          |   |             |
|            | i |                                          |   |             |
|            | n |                                          |   |             |
|            | e |                                          |   |             |
|            | } |                                          |   |             |
|            | > |                                          |   |             |
|            | > |                                          |   |             |
|            |   |                                          |   |             |
|            | G |                                          |   |             |
|            | o |                                          |   |             |
|            | a |                                          |   |             |
|            | l |                                          |   |             |
|            | > |                                          |   |             |
|            |   |                                          |   |             |
|            | 7 |                                          |   |             |
|            | 0 |                                          |   |             |
|            | % |                                          |   |             |
|            | > |                                          |   |             |
|            |   |                                          |   |             |
|            | 8 |                                          |   |             |
|            | 5 |                                          |   |             |
|            | % |                                          |   |             |
|            | > |                                          |   |             |
|            | > |                                          |   |             |
|            |   |                                          |   |             |
|            | R |                                          |   |             |
|            | e |                                          |   |             |
|            | s |                                          |   |             |
|            | u |                                          |   |             |
|            | l |                                          |   |             |
|            | t |                                          |   |             |
|            | > |                                          |   |             |
|            |   |                                          |   |             |
|            | 5 |                                          |   |             |
|            | 9 |                                          |   |             |
|            | % |                                          |   |             |
|            | > |                                          |   |             |
|            |   |                                          |   |             |
|            | 9 |                                          |   |             |
|            | 2 |                                          |   |             |
|            | % |                                          |   |             |
+------------+---+------------------------------------------+---+-------------+

##### ANNUAL PERFORMANCE GOALS and RESULTS FOR INVESTMENT PROCESS

+------------+-----------------------------------------+--------------+
| ### P      | ### Performance Goals f                 | #            |
| erformance | or Investment Process {#performance-goa | ## Results { |
|  Area {#pe | ls-for-investment-process .Box-Heading} | #results-5 . |
| rformance- |                                         | Box-Heading} |
| area-2 .Bo |                                         |              |
| x-Heading} |                                         |              |
+------------+-----------------------------------------+--------------+
| Proposal   |                                         |              |
| and Award  |                                         |              |
| Processes  |                                         |              |
+------------+-----------------------------------------+--------------+
| Use of     | Investment Goal 1:                      | **FY 1999:** |
| Merit      |                                         | Goal         |
| Review     | At least 90 percent of NSF funds will   | achieved.    |
|            | be allocated to projects reviewed by    |              |
|            | appropriate peers external to NSF and   | **FY 2000:** |
|            | selected through a merit-based          | Goal         |
|            | competitive process.                    | achieved.    |
|            |                                         |              |
|            | > [Baseline]{.underline}                |              |
|            | > [Goal]{.underline}                    |              |
|            | > [Result]{.underline}                  |              |
|            | >                                       |              |
|            | > FY 1998 90%\*                         |              |
|            | >                                       |              |
|            | > FY 1999 90%\* 95%\*                   |              |
|            | >                                       |              |
|            | > *FY 2000 80%\*\* (was 90%\*) 87%\*\*  |              |
|            | > (95%)\**                              |              |
|            | >                                       |              |
|            | > FY 2001 85%\*\*                       |              |
|            |                                         |              |
|            | *\*Based on old definition.*            |              |
|            |                                         |              |
|            | *\*\*Based on most recent revisions of  |              |
|            | definitions by OMB.*                    |              |
+------------+-----------------------------------------+--------------+
| Impl       | Investment Goal 2:                      | **FY 1999:** |
| ementation |                                         | Goal         |
| of Merit   | NSF performance in implementation of    | achieved.    |
| Review     | the new merit review criteria is        |              |
| Criteria   | *successful* when reviewers address the | **FY 2000:** |
|            | elements of both generic review         | Goal not     |
|            | criteria appropriate to the proposal at | achieved.    |
|            | hand and when program officers take the |              |
|            | information provided into account in    |              |
|            | their decisions on awards, as judged by |              |
|            | external independent experts.           |              |
|            |                                         |              |
|            | > **FY 1999 Result:** Largely           |              |
|            | > successful as judged by experts.      |              |
|            | >                                       |              |
|            | > Needs improvement.                    |              |
|            | >                                       |              |
|            | > **FY 2000 Result:** Not fully         |              |
|            | > successful as judged by experts.      |              |
|            | >                                       |              |
|            | > Needs improvement.                    |              |
+------------+-----------------------------------------+--------------+
| Customer   | Investment Goal 3:                      | New goal in  |
| service    |                                         | FY 2000.     |
|            | Identify possible reasons for customer  |              |
| -- General | dissatisfaction with NSF\'s merit       | **FY 2000**: |
|            | review system and with NSF\'s complaint | Goal         |
|            | system.                                 | achieved.    |
|            |                                         |              |
|            | New goal in FY 2000; not continued in   |              |
|            | FY 2001.                                |              |
|            |                                         |              |
|            | **FY 2000 Result:** An external         |              |
|            | customer service survey of NSF          |              |
|            | applicants was conducted in FY 2000.    |              |
+------------+-----------------------------------------+--------------+
| Customer   | Investment Goal 4:                      | New goal in  |
| service    |                                         | FY 2000.     |
|            | Identify best practices and training    |              |
| -- General | necessary for NSF staff to conduct      | **FY 2000:** |
|            | merit review and answer questions about | Goal not     |
|            | the review criteria and process;        | achieved.    |
|            | identify best practices and training    |              |
|            | necessary for NSF staff to answer       |              |
|            | questions from the community and to     |              |
|            | deal with complaints in a forthright    |              |
|            | manner.                                 |              |
|            |                                         |              |
|            | **FY 2000 Results:** Goal not completed |              |
|            | in FY 2000. Plans to finalize           |              |
|            | implementation in FY 2001.              |              |
+------------+-----------------------------------------+--------------+

### ANNUAL PERFORMANCE GOALS FOR INVESTMENT PROCESS --continued

+------------+---+---+-----------------------------------------+---+---+---------------+
| ### P      |   |   | ### Performance Goals for               |   |   | ### Results   |
| erformance |   |   |  Investment Process {#performance-goals |   |   |  {#results-6  |
|  Area {#pe |   |   | -for-investment-process-1 .Box-Heading} |   |   | .Box-Heading} |
| rformance- |   |   |                                         |   |   |               |
| area-3 .Bo |   |   |                                         |   |   |               |
| x-Heading} |   |   |                                         |   |   |               |
+------------+---+---+-----------------------------------------+---+---+---------------+
| Customer   | I |   |                                         | N |   |               |
| service -- | n |   |                                         | e |   |               |
| General    | v |   |                                         | w |   |               |
|            | e |   |                                         | g |   |               |
|            | s |   |                                         | o |   |               |
|            | t |   |                                         | a |   |               |
|            | m |   |                                         | l |   |               |
|            | e |   |                                         | i |   |               |
|            | n |   |                                         | n |   |               |
|            | t |   |                                         | F |   |               |
|            | G |   |                                         | Y |   |               |
|            | o |   |                                         | 2 |   |               |
|            | a |   |                                         | 0 |   |               |
|            | l |   |                                         | 0 |   |               |
|            | 5 |   |                                         | 0 |   |               |
|            | : |   |                                         | . |   |               |
|            |   |   |                                         |   |   |               |
|            | I |   |                                         | * |   |               |
|            | m |   |                                         | * |   |               |
|            | p |   |                                         | F |   |               |
|            | r |   |                                         | Y |   |               |
|            | o |   |                                         | 2 |   |               |
|            | v |   |                                         | 0 |   |               |
|            | e |   |                                         | 0 |   |               |
|            | N |   |                                         | 0 |   |               |
|            | S |   |                                         | : |   |               |
|            | F |   |                                         | * |   |               |
|            | \ |   |                                         | * |   |               |
|            | ' |   |                                         | G |   |               |
|            | s |   |                                         | o |   |               |
|            | o |   |                                         | a |   |               |
|            | v |   |                                         | l |   |               |
|            | e |   |                                         | a |   |               |
|            | r |   |                                         | c |   |               |
|            | a |   |                                         | h |   |               |
|            | l |   |                                         | i |   |               |
|            | l |   |                                         | e |   |               |
|            | A |   |                                         | v |   |               |
|            | m |   |                                         | e |   |               |
|            | e |   |                                         | d |   |               |
|            | r |   |                                         | . |   |               |
|            | i |   |                                         |   |   |               |
|            | c |   |                                         |   |   |               |
|            | a |   |                                         |   |   |               |
|            | n |   |                                         |   |   |               |
|            | C |   |                                         |   |   |               |
|            | u |   |                                         |   |   |               |
|            | s |   |                                         |   |   |               |
|            | t |   |                                         |   |   |               |
|            | o |   |                                         |   |   |               |
|            | m |   |                                         |   |   |               |
|            | e |   |                                         |   |   |               |
|            | r |   |                                         |   |   |               |
|            | S |   |                                         |   |   |               |
|            | a |   |                                         |   |   |               |
|            | t |   |                                         |   |   |               |
|            | i |   |                                         |   |   |               |
|            | s |   |                                         |   |   |               |
|            | f |   |                                         |   |   |               |
|            | a |   |                                         |   |   |               |
|            | c |   |                                         |   |   |               |
|            | t |   |                                         |   |   |               |
|            | i |   |                                         |   |   |               |
|            | o |   |                                         |   |   |               |
|            | n |   |                                         |   |   |               |
|            | I |   |                                         |   |   |               |
|            | n |   |                                         |   |   |               |
|            | d |   |                                         |   |   |               |
|            | e |   |                                         |   |   |               |
|            | x |   |                                         |   |   |               |
|            | ( |   |                                         |   |   |               |
|            | A |   |                                         |   |   |               |
|            | C |   |                                         |   |   |               |
|            | S |   |                                         |   |   |               |
|            | I |   |                                         |   |   |               |
|            | ) |   |                                         |   |   |               |
|            | c |   |                                         |   |   |               |
|            | o |   |                                         |   |   |               |
|            | m |   |                                         |   |   |               |
|            | p |   |                                         |   |   |               |
|            | a |   |                                         |   |   |               |
|            | r |   |                                         |   |   |               |
|            | e |   |                                         |   |   |               |
|            | d |   |                                         |   |   |               |
|            | t |   |                                         |   |   |               |
|            | o |   |                                         |   |   |               |
|            | t |   |                                         |   |   |               |
|            | h |   |                                         |   |   |               |
|            | e |   |                                         |   |   |               |
|            | F |   |                                         |   |   |               |
|            | Y |   |                                         |   |   |               |
|            | 1 |   |                                         |   |   |               |
|            | 9 |   |                                         |   |   |               |
|            | 9 |   |                                         |   |   |               |
|            | 9 |   |                                         |   |   |               |
|            | I |   |                                         |   |   |               |
|            | n |   |                                         |   |   |               |
|            | d |   |                                         |   |   |               |
|            | e |   |                                         |   |   |               |
|            | x |   |                                         |   |   |               |
|            | o |   |                                         |   |   |               |
|            | f |   |                                         |   |   |               |
|            | 5 |   |                                         |   |   |               |
|            | 7 |   |                                         |   |   |               |
|            | ( |   |                                         |   |   |               |
|            | o |   |                                         |   |   |               |
|            | n |   |                                         |   |   |               |
|            | a |   |                                         |   |   |               |
|            | s |   |                                         |   |   |               |
|            | c |   |                                         |   |   |               |
|            | a |   |                                         |   |   |               |
|            | l |   |                                         |   |   |               |
|            | e |   |                                         |   |   |               |
|            | o |   |                                         |   |   |               |
|            | f |   |                                         |   |   |               |
|            | 0 |   |                                         |   |   |               |
|            | - |   |                                         |   |   |               |
|            | 1 |   |                                         |   |   |               |
|            | 0 |   |                                         |   |   |               |
|            | 0 |   |                                         |   |   |               |
|            | ) |   |                                         |   |   |               |
|            | . |   |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            | * |   |                                         |   |   |               |
|            | * |   |                                         |   |   |               |
|            | F |   |                                         |   |   |               |
|            | Y |   |                                         |   |   |               |
|            | 2 |   |                                         |   |   |               |
|            | 0 |   |                                         |   |   |               |
|            | 0 |   |                                         |   |   |               |
|            | 0 |   |                                         |   |   |               |
|            | R |   |                                         |   |   |               |
|            | e |   |                                         |   |   |               |
|            | s |   |                                         |   |   |               |
|            | u |   |                                         |   |   |               |
|            | l |   |                                         |   |   |               |
|            | t |   |                                         |   |   |               |
|            | s |   |                                         |   |   |               |
|            | : |   |                                         |   |   |               |
|            | * |   |                                         |   |   |               |
|            | * |   |                                         |   |   |               |
|            | N |   |                                         |   |   |               |
|            | S |   |                                         |   |   |               |
|            | F |   |                                         |   |   |               |
|            | a |   |                                         |   |   |               |
|            | c |   |                                         |   |   |               |
|            | h |   |                                         |   |   |               |
|            | i |   |                                         |   |   |               |
|            | e |   |                                         |   |   |               |
|            | v |   |                                         |   |   |               |
|            | e |   |                                         |   |   |               |
|            | d |   |                                         |   |   |               |
|            | a |   |                                         |   |   |               |
|            | n |   |                                         |   |   |               |
|            | A |   |                                         |   |   |               |
|            | C |   |                                         |   |   |               |
|            | S |   |                                         |   |   |               |
|            | I |   |                                         |   |   |               |
|            | i |   |                                         |   |   |               |
|            | n |   |                                         |   |   |               |
|            | d |   |                                         |   |   |               |
|            | e |   |                                         |   |   |               |
|            | x |   |                                         |   |   |               |
|            | o |   |                                         |   |   |               |
|            | f |   |                                         |   |   |               |
|            | 5 |   |                                         |   |   |               |
|            | 8 |   |                                         |   |   |               |
|            | i |   |                                         |   |   |               |
|            | n |   |                                         |   |   |               |
|            | F |   |                                         |   |   |               |
|            | Y |   |                                         |   |   |               |
|            | 2 |   |                                         |   |   |               |
|            | 0 |   |                                         |   |   |               |
|            | 0 |   |                                         |   |   |               |
|            | 0 |   |                                         |   |   |               |
|            | . |   |                                         |   |   |               |
+------------+---+---+-----------------------------------------+---+---+---------------+
| Customer   |   | I |                                         |   | * |               |
| service -- |   | n |                                         |   | * |               |
| Time to    |   | v |                                         |   | F |               |
| prepare    |   | e |                                         |   | Y |               |
| proposals  |   | s |                                         |   | 1 |               |
|            |   | t |                                         |   | 9 |               |
|            |   | m |                                         |   | 9 |               |
|            |   | e |                                         |   | 9 |               |
|            |   | n |                                         |   | : |               |
|            |   | t |                                         |   | * |               |
|            |   | G |                                         |   | * |               |
|            |   | o |                                         |   | G |               |
|            |   | a |                                         |   | o |               |
|            |   | l |                                         |   | a |               |
|            |   | 6 |                                         |   | l |               |
|            |   | : |                                         |   | n |               |
|            |   |   |                                         |   | o |               |
|            |   | 9 |                                         |   | t |               |
|            |   | 5 |                                         |   | a |               |
|            |   | % |                                         |   | c |               |
|            |   | o |                                         |   | h |               |
|            |   | f |                                         |   | i |               |
|            |   | p |                                         |   | e |               |
|            |   | r |                                         |   | v |               |
|            |   | o |                                         |   | e |               |
|            |   | g |                                         |   | d |               |
|            |   | r |                                         |   | . |               |
|            |   | a |                                         |   |   |               |
|            |   | m |                                         |   | * |               |
|            |   | a |                                         |   | * |               |
|            |   | n |                                         |   | F |               |
|            |   | n |                                         |   | Y |               |
|            |   | o |                                         |   | 2 |               |
|            |   | u |                                         |   | 0 |               |
|            |   | n |                                         |   | 0 |               |
|            |   | c |                                         |   | 0 |               |
|            |   | e |                                         |   | : |               |
|            |   | m |                                         |   | * |               |
|            |   | e |                                         |   | * |               |
|            |   | n |                                         |   | G |               |
|            |   | t |                                         |   | o |               |
|            |   | s |                                         |   | a |               |
|            |   | a |                                         |   | l |               |
|            |   | n |                                         |   | n |               |
|            |   | d |                                         |   | o |               |
|            |   | s |                                         |   | t |               |
|            |   | o |                                         |   | a |               |
|            |   | l |                                         |   | c |               |
|            |   | i |                                         |   | h |               |
|            |   | c |                                         |   | i |               |
|            |   | i |                                         |   | e |               |
|            |   | t |                                         |   | v |               |
|            |   | a |                                         |   | e |               |
|            |   | t |                                         |   | d |               |
|            |   | i |                                         |   | . |               |
|            |   | o |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | s |                                         |   |   |               |
|            |   | w |                                         |   |   |               |
|            |   | i |                                         |   |   |               |
|            |   | l |                                         |   |   |               |
|            |   | l |                                         |   |   |               |
|            |   | b |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | a |                                         |   |   |               |
|            |   | v |                                         |   |   |               |
|            |   | a |                                         |   |   |               |
|            |   | i |                                         |   |   |               |
|            |   | l |                                         |   |   |               |
|            |   | a |                                         |   |   |               |
|            |   | b |                                         |   |   |               |
|            |   | l |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | a |                                         |   |   |               |
|            |   | t |                                         |   |   |               |
|            |   | l |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | a |                                         |   |   |               |
|            |   | s |                                         |   |   |               |
|            |   | t |                                         |   |   |               |
|            |   | t |                                         |   |   |               |
|            |   | h |                                         |   |   |               |
|            |   | r |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | m |                                         |   |   |               |
|            |   | o |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | t |                                         |   |   |               |
|            |   | h |                                         |   |   |               |
|            |   | s |                                         |   |   |               |
|            |   | p |                                         |   |   |               |
|            |   | r |                                         |   |   |               |
|            |   | i |                                         |   |   |               |
|            |   | o |                                         |   |   |               |
|            |   | r |                                         |   |   |               |
|            |   | t |                                         |   |   |               |
|            |   | o |                                         |   |   |               |
|            |   | p |                                         |   |   |               |
|            |   | r |                                         |   |   |               |
|            |   | o |                                         |   |   |               |
|            |   | p |                                         |   |   |               |
|            |   | o |                                         |   |   |               |
|            |   | s |                                         |   |   |               |
|            |   | a |                                         |   |   |               |
|            |   | l |                                         |   |   |               |
|            |   | d |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | a |                                         |   |   |               |
|            |   | d |                                         |   |   |               |
|            |   | l |                                         |   |   |               |
|            |   | i |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | s |                                         |   |   |               |
|            |   | o |                                         |   |   |               |
|            |   | r |                                         |   |   |               |
|            |   | t |                                         |   |   |               |
|            |   | a |                                         |   |   |               |
|            |   | r |                                         |   |   |               |
|            |   | g |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | t |                                         |   |   |               |
|            |   | d |                                         |   |   |               |
|            |   | a |                                         |   |   |               |
|            |   | t |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | s |                                         |   |   |               |
|            |   | . |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | [ |                                         |   |   |               |
|            |   | F |                                         |   |   |               |
|            |   | Y |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | 1 |                                         |   |   |               |
|            |   | 9 |                                         |   |   |               |
|            |   | 9 |                                         |   |   |               |
|            |   | 8 |                                         |   |   |               |
|            |   | ] |                                         |   |   |               |
|            |   | { |                                         |   |   |               |
|            |   | . |                                         |   |   |               |
|            |   | u |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | d |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | r |                                         |   |   |               |
|            |   | l |                                         |   |   |               |
|            |   | i |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | } |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | [ |                                         |   |   |               |
|            |   | F |                                         |   |   |               |
|            |   | Y |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | 1 |                                         |   |   |               |
|            |   | 9 |                                         |   |   |               |
|            |   | 9 |                                         |   |   |               |
|            |   | 9 |                                         |   |   |               |
|            |   | ] |                                         |   |   |               |
|            |   | { |                                         |   |   |               |
|            |   | . |                                         |   |   |               |
|            |   | u |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | d |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | r |                                         |   |   |               |
|            |   | l |                                         |   |   |               |
|            |   | i |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | } |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | [ |                                         |   |   |               |
|            |   | F |                                         |   |   |               |
|            |   | Y |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | 2 |                                         |   |   |               |
|            |   | 0 |                                         |   |   |               |
|            |   | 0 |                                         |   |   |               |
|            |   | 0 |                                         |   |   |               |
|            |   | ] |                                         |   |   |               |
|            |   | { |                                         |   |   |               |
|            |   | . |                                         |   |   |               |
|            |   | u |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | d |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | r |                                         |   |   |               |
|            |   | l |                                         |   |   |               |
|            |   | i |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | } |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | [ |                                         |   |   |               |
|            |   | F |                                         |   |   |               |
|            |   | Y |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | 2 |                                         |   |   |               |
|            |   | 0 |                                         |   |   |               |
|            |   | 0 |                                         |   |   |               |
|            |   | 1 |                                         |   |   |               |
|            |   | ] |                                         |   |   |               |
|            |   | { |                                         |   |   |               |
|            |   | . |                                         |   |   |               |
|            |   | u |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | d |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | r |                                         |   |   |               |
|            |   | l |                                         |   |   |               |
|            |   | i |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | } |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | B |                                         |   |   |               |
|            |   | a |                                         |   |   |               |
|            |   | s |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | l |                                         |   |   |               |
|            |   | i |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | 6 |                                         |   |   |               |
|            |   | 6 |                                         |   |   |               |
|            |   | % |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | G |                                         |   |   |               |
|            |   | o |                                         |   |   |               |
|            |   | a |                                         |   |   |               |
|            |   | l |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | 9 |                                         |   |   |               |
|            |   | 5 |                                         |   |   |               |
|            |   | % |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | 9 |                                         |   |   |               |
|            |   | 5 |                                         |   |   |               |
|            |   | % |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | 9 |                                         |   |   |               |
|            |   | 5 |                                         |   |   |               |
|            |   | % |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | R |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | s |                                         |   |   |               |
|            |   | u |                                         |   |   |               |
|            |   | l |                                         |   |   |               |
|            |   | t |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | 7 |                                         |   |   |               |
|            |   | 5 |                                         |   |   |               |
|            |   | % |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | 8 |                                         |   |   |               |
|            |   | 9 |                                         |   |   |               |
|            |   | % |                                         |   |   |               |
+------------+---+---+-----------------------------------------+---+---+---------------+
| Customer   |   | I |                                         |   | * |               |
| service -- |   | n |                                         |   | * |               |
| Time to    |   | v |                                         |   | F |               |
| decision   |   | e |                                         |   | Y |               |
|            |   | s |                                         |   | 1 |               |
|            |   | t |                                         |   | 9 |               |
|            |   | m |                                         |   | 9 |               |
|            |   | e |                                         |   | 9 |               |
|            |   | n |                                         |   | : |               |
|            |   | t |                                         |   | * |               |
|            |   | G |                                         |   | * |               |
|            |   | o |                                         |   | G |               |
|            |   | a |                                         |   | o |               |
|            |   | l |                                         |   | a |               |
|            |   | 7 |                                         |   | l |               |
|            |   | : |                                         |   | n |               |
|            |   |   |                                         |   | o |               |
|            |   | M |                                         |   | t |               |
|            |   | a |                                         |   | a |               |
|            |   | i |                                         |   | c |               |
|            |   | n |                                         |   | h |               |
|            |   | t |                                         |   | i |               |
|            |   | a |                                         |   | e |               |
|            |   | i |                                         |   | v |               |
|            |   | n |                                         |   | e |               |
|            |   | t |                                         |   | d |               |
|            |   | h |                                         |   | . |               |
|            |   | e |                                         |   |   |               |
|            |   | F |                                         |   | * |               |
|            |   | Y |                                         |   | * |               |
|            |   | 1 |                                         |   | F |               |
|            |   | 9 |                                         |   | Y |               |
|            |   | 9 |                                         |   | 2 |               |
|            |   | 9 |                                         |   | 0 |               |
|            |   | g |                                         |   | 0 |               |
|            |   | o |                                         |   | 0 |               |
|            |   | a |                                         |   | : |               |
|            |   | l |                                         |   | * |               |
|            |   | t |                                         |   | * |               |
|            |   | o |                                         |   | G |               |
|            |   | p |                                         |   | o |               |
|            |   | r |                                         |   | a |               |
|            |   | o |                                         |   | l |               |
|            |   | c |                                         |   | n |               |
|            |   | e |                                         |   | o |               |
|            |   | s |                                         |   | t |               |
|            |   | s |                                         |   | a |               |
|            |   | 7 |                                         |   | c |               |
|            |   | 0 |                                         |   | h |               |
|            |   | % |                                         |   | i |               |
|            |   | o |                                         |   | e |               |
|            |   | f |                                         |   | v |               |
|            |   | p |                                         |   | e |               |
|            |   | r |                                         |   | d |               |
|            |   | o |                                         |   | . |               |
|            |   | p |                                         |   |   |               |
|            |   | o |                                         |   |   |               |
|            |   | s |                                         |   |   |               |
|            |   | a |                                         |   |   |               |
|            |   | l |                                         |   |   |               |
|            |   | s |                                         |   |   |               |
|            |   | w |                                         |   |   |               |
|            |   | i |                                         |   |   |               |
|            |   | t |                                         |   |   |               |
|            |   | h |                                         |   |   |               |
|            |   | i |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | s |                                         |   |   |               |
|            |   | i |                                         |   |   |               |
|            |   | x |                                         |   |   |               |
|            |   | m |                                         |   |   |               |
|            |   | o |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | t |                                         |   |   |               |
|            |   | h |                                         |   |   |               |
|            |   | s |                                         |   |   |               |
|            |   | o |                                         |   |   |               |
|            |   | f |                                         |   |   |               |
|            |   | r |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | c |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | i |                                         |   |   |               |
|            |   | p |                                         |   |   |               |
|            |   | t |                                         |   |   |               |
|            |   | , |                                         |   |   |               |
|            |   | i |                                         |   |   |               |
|            |   | m |                                         |   |   |               |
|            |   | p |                                         |   |   |               |
|            |   | r |                                         |   |   |               |
|            |   | o |                                         |   |   |               |
|            |   | v |                                         |   |   |               |
|            |   | i |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | g |                                         |   |   |               |
|            |   | u |                                         |   |   |               |
|            |   | p |                                         |   |   |               |
|            |   | o |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | t |                                         |   |   |               |
|            |   | h |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | F |                                         |   |   |               |
|            |   | Y |                                         |   |   |               |
|            |   | 1 |                                         |   |   |               |
|            |   | 9 |                                         |   |   |               |
|            |   | 9 |                                         |   |   |               |
|            |   | 8 |                                         |   |   |               |
|            |   | b |                                         |   |   |               |
|            |   | a |                                         |   |   |               |
|            |   | s |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | l |                                         |   |   |               |
|            |   | i |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | . |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | [ |                                         |   |   |               |
|            |   | F |                                         |   |   |               |
|            |   | Y |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | 1 |                                         |   |   |               |
|            |   | 9 |                                         |   |   |               |
|            |   | 9 |                                         |   |   |               |
|            |   | 8 |                                         |   |   |               |
|            |   | ] |                                         |   |   |               |
|            |   | { |                                         |   |   |               |
|            |   | . |                                         |   |   |               |
|            |   | u |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | d |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | r |                                         |   |   |               |
|            |   | l |                                         |   |   |               |
|            |   | i |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | } |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | [ |                                         |   |   |               |
|            |   | F |                                         |   |   |               |
|            |   | Y |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | 1 |                                         |   |   |               |
|            |   | 9 |                                         |   |   |               |
|            |   | 9 |                                         |   |   |               |
|            |   | 9 |                                         |   |   |               |
|            |   | ] |                                         |   |   |               |
|            |   | { |                                         |   |   |               |
|            |   | . |                                         |   |   |               |
|            |   | u |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | d |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | r |                                         |   |   |               |
|            |   | l |                                         |   |   |               |
|            |   | i |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | } |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | [ |                                         |   |   |               |
|            |   | F |                                         |   |   |               |
|            |   | Y |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | 2 |                                         |   |   |               |
|            |   | 0 |                                         |   |   |               |
|            |   | 0 |                                         |   |   |               |
|            |   | 0 |                                         |   |   |               |
|            |   | ] |                                         |   |   |               |
|            |   | { |                                         |   |   |               |
|            |   | . |                                         |   |   |               |
|            |   | u |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | d |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | r |                                         |   |   |               |
|            |   | l |                                         |   |   |               |
|            |   | i |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | } |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | [ |                                         |   |   |               |
|            |   | F |                                         |   |   |               |
|            |   | Y |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | 2 |                                         |   |   |               |
|            |   | 0 |                                         |   |   |               |
|            |   | 0 |                                         |   |   |               |
|            |   | 1 |                                         |   |   |               |
|            |   | ] |                                         |   |   |               |
|            |   | { |                                         |   |   |               |
|            |   | . |                                         |   |   |               |
|            |   | u |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | d |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | r |                                         |   |   |               |
|            |   | l |                                         |   |   |               |
|            |   | i |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | } |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | B |                                         |   |   |               |
|            |   | a |                                         |   |   |               |
|            |   | s |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | l |                                         |   |   |               |
|            |   | i |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | 5 |                                         |   |   |               |
|            |   | 9 |                                         |   |   |               |
|            |   | % |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | G |                                         |   |   |               |
|            |   | o |                                         |   |   |               |
|            |   | a |                                         |   |   |               |
|            |   | l |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | 7 |                                         |   |   |               |
|            |   | 0 |                                         |   |   |               |
|            |   | % |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | 7 |                                         |   |   |               |
|            |   | 0 |                                         |   |   |               |
|            |   | % |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | 7 |                                         |   |   |               |
|            |   | 0 |                                         |   |   |               |
|            |   | % |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | R |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | s |                                         |   |   |               |
|            |   | u |                                         |   |   |               |
|            |   | l |                                         |   |   |               |
|            |   | t |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | 5 |                                         |   |   |               |
|            |   | 8 |                                         |   |   |               |
|            |   | % |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | 5 |                                         |   |   |               |
|            |   | 4 |                                         |   |   |               |
|            |   | % |                                         |   |   |               |
+------------+---+---+-----------------------------------------+---+---+---------------+
| M          |   | I |                                         |   | * |               |
| aintaining |   | n |                                         |   | * |               |
| openness   |   | v |                                         |   | F |               |
| in the     |   | e |                                         |   | Y |               |
| system     |   | s |                                         |   | 1 |               |
|            |   | t |                                         |   | 9 |               |
|            |   | m |                                         |   | 9 |               |
|            |   | e |                                         |   | 9 |               |
|            |   | n |                                         |   | : |               |
|            |   | t |                                         |   | * |               |
|            |   | G |                                         |   | * |               |
|            |   | o |                                         |   | G |               |
|            |   | a |                                         |   | o |               |
|            |   | l |                                         |   | a |               |
|            |   | 8 |                                         |   | l |               |
|            |   | : |                                         |   | n |               |
|            |   |   |                                         |   | o |               |
|            |   | T |                                         |   | t |               |
|            |   | h |                                         |   | a |               |
|            |   | e |                                         |   | c |               |
|            |   | p |                                         |   | h |               |
|            |   | e |                                         |   | i |               |
|            |   | r |                                         |   | e |               |
|            |   | c |                                         |   | v |               |
|            |   | e |                                         |   | e |               |
|            |   | n |                                         |   | d |               |
|            |   | t |                                         |   | . |               |
|            |   | a |                                         |   |   |               |
|            |   | g |                                         |   | * |               |
|            |   | e |                                         |   | * |               |
|            |   | o |                                         |   | F |               |
|            |   | f |                                         |   | Y |               |
|            |   | c |                                         |   | 2 |               |
|            |   | o |                                         |   | 0 |               |
|            |   | m |                                         |   | 0 |               |
|            |   | p |                                         |   | 0 |               |
|            |   | e |                                         |   | : |               |
|            |   | t |                                         |   | * |               |
|            |   | i |                                         |   | * |               |
|            |   | t |                                         |   | G |               |
|            |   | i |                                         |   | o |               |
|            |   | v |                                         |   | a |               |
|            |   | e |                                         |   | l |               |
|            |   | r |                                         |   | n |               |
|            |   | e |                                         |   | o |               |
|            |   | s |                                         |   | t |               |
|            |   | e |                                         |   | a |               |
|            |   | a |                                         |   | c |               |
|            |   | r |                                         |   | h |               |
|            |   | c |                                         |   | i |               |
|            |   | h |                                         |   | e |               |
|            |   | g |                                         |   | v |               |
|            |   | r |                                         |   | e |               |
|            |   | a |                                         |   | d |               |
|            |   | n |                                         |   | . |               |
|            |   | t |                                         |   |   |               |
|            |   | s |                                         |   |   |               |
|            |   | g |                                         |   |   |               |
|            |   | o |                                         |   |   |               |
|            |   | i |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | g |                                         |   |   |               |
|            |   | t |                                         |   |   |               |
|            |   | o |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | w |                                         |   |   |               |
|            |   | i |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | v |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | s |                                         |   |   |               |
|            |   | t |                                         |   |   |               |
|            |   | i |                                         |   |   |               |
|            |   | g |                                         |   |   |               |
|            |   | a |                                         |   |   |               |
|            |   | t |                                         |   |   |               |
|            |   | o |                                         |   |   |               |
|            |   | r |                                         |   |   |               |
|            |   | s |                                         |   |   |               |
|            |   | w |                                         |   |   |               |
|            |   | i |                                         |   |   |               |
|            |   | l |                                         |   |   |               |
|            |   | l |                                         |   |   |               |
|            |   | b |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | a |                                         |   |   |               |
|            |   | t |                                         |   |   |               |
|            |   | l |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | a |                                         |   |   |               |
|            |   | s |                                         |   |   |               |
|            |   | t |                                         |   |   |               |
|            |   | 3 |                                         |   |   |               |
|            |   | 0 |                                         |   |   |               |
|            |   | % |                                         |   |   |               |
|            |   | . |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | [ |                                         |   |   |               |
|            |   | F |                                         |   |   |               |
|            |   | Y |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | 1 |                                         |   |   |               |
|            |   | 9 |                                         |   |   |               |
|            |   | 9 |                                         |   |   |               |
|            |   | 8 |                                         |   |   |               |
|            |   | ] |                                         |   |   |               |
|            |   | { |                                         |   |   |               |
|            |   | . |                                         |   |   |               |
|            |   | u |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | d |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | r |                                         |   |   |               |
|            |   | l |                                         |   |   |               |
|            |   | i |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | } |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | [ |                                         |   |   |               |
|            |   | F |                                         |   |   |               |
|            |   | Y |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | 1 |                                         |   |   |               |
|            |   | 9 |                                         |   |   |               |
|            |   | 9 |                                         |   |   |               |
|            |   | 9 |                                         |   |   |               |
|            |   | ] |                                         |   |   |               |
|            |   | { |                                         |   |   |               |
|            |   | . |                                         |   |   |               |
|            |   | u |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | d |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | r |                                         |   |   |               |
|            |   | l |                                         |   |   |               |
|            |   | i |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | } |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | [ |                                         |   |   |               |
|            |   | F |                                         |   |   |               |
|            |   | Y |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | 2 |                                         |   |   |               |
|            |   | 0 |                                         |   |   |               |
|            |   | 0 |                                         |   |   |               |
|            |   | 0 |                                         |   |   |               |
|            |   | ] |                                         |   |   |               |
|            |   | { |                                         |   |   |               |
|            |   | . |                                         |   |   |               |
|            |   | u |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | d |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | r |                                         |   |   |               |
|            |   | l |                                         |   |   |               |
|            |   | i |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | } |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | [ |                                         |   |   |               |
|            |   | F |                                         |   |   |               |
|            |   | Y |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | 2 |                                         |   |   |               |
|            |   | 0 |                                         |   |   |               |
|            |   | 0 |                                         |   |   |               |
|            |   | 1 |                                         |   |   |               |
|            |   | ] |                                         |   |   |               |
|            |   | { |                                         |   |   |               |
|            |   | . |                                         |   |   |               |
|            |   | u |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | d |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | r |                                         |   |   |               |
|            |   | l |                                         |   |   |               |
|            |   | i |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | } |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | B |                                         |   |   |               |
|            |   | a |                                         |   |   |               |
|            |   | s |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | l |                                         |   |   |               |
|            |   | i |                                         |   |   |               |
|            |   | n |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | 2 |                                         |   |   |               |
|            |   | 7 |                                         |   |   |               |
|            |   | % |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | G |                                         |   |   |               |
|            |   | o |                                         |   |   |               |
|            |   | a |                                         |   |   |               |
|            |   | l |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | 3 |                                         |   |   |               |
|            |   | 0 |                                         |   |   |               |
|            |   | % |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | 3 |                                         |   |   |               |
|            |   | 0 |                                         |   |   |               |
|            |   | % |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | 3 |                                         |   |   |               |
|            |   | 0 |                                         |   |   |               |
|            |   | % |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | R |                                         |   |   |               |
|            |   | e |                                         |   |   |               |
|            |   | s |                                         |   |   |               |
|            |   | u |                                         |   |   |               |
|            |   | l |                                         |   |   |               |
|            |   | t |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | 2 |                                         |   |   |               |
|            |   | 7 |                                         |   |   |               |
|            |   | % |                                         |   |   |               |
|            |   | > |                                         |   |   |               |
|            |   |   |                                         |   |   |               |
|            |   | 2 |                                         |   |   |               |
|            |   | 8 |                                         |   |   |               |
|            |   | % |                                         |   |   |               |
+------------+---+---+-----------------------------------------+---+---+---------------+

### ANNUAL PERFORMANCE GOALS FOR INVESTMENT PROCESS---continued

+------------+------------------------------------------+--------------+
| ### P      | ### Performance Goals f                  | #            |
| erformance | or Investment Process {#performance-goal | ## Results { |
|  Area {#pe | s-for-investment-process-2 .Box-Heading} | #results-7 . |
| rformance- |                                          | Box-Heading} |
| area-4 .Bo |                                          |              |
| x-Heading} |                                          |              |
+------------+------------------------------------------+--------------+
| I          |                                          |              |
| ntegration |                                          |              |
| of         |                                          |              |
| Research   |                                          |              |
| and        |                                          |              |
| Education  |                                          |              |
+------------+------------------------------------------+--------------+
| In         | Investment Goal 9:                       | New goal in  |
| Proposals  |                                          | FY 2000.     |
|            | NSF will develop a plan and system to    |              |
|            | request that Principal Investigators     | **FY 2000:** |
|            | (PIs) address the integration of         | Goal         |
|            | research and education in their          | achieved.    |
|            | proposals, and develop and implement a   |              |
|            | system to verify that PIs have done so.  |              |
|            |                                          |              |
|            | **Result:** In FY 2000, NSF implemented  |              |
|            | an electronic program announcement       |              |
|            | template clearance process (PAT) that is |              |
|            | used by NSF staff to generate            |              |
|            | announcements and solicitations. Use of  |              |
|            | the PAT ensures that the integration of  |              |
|            | research and education is emphasized in  |              |
|            | all announcements and solicitations for  |              |
|            | PIs to address in their submissions.     |              |
+------------+------------------------------------------+--------------+
| In Reviews | Investment Goal 10:                      | New goal in  |
|            |                                          | FY 2000.     |
|            | NSF will develop and implement a         |              |
|            | system/mechanism to request and track    | **FY 2000:** |
|            | reviewer comments tied to merit review   | Goal         |
|            | criterion #2, \"what are the broader     | achieved.    |
|            | impacts of the proposed activity?"       |              |
|            |                                          |              |
|            | (Revised goal.) No baseline.             |              |
|            |                                          |              |
|            | **Result:** In FY 2000, screens in       |              |
|            | FastLane were redesigned so that         |              |
|            | reviewers can address each merit-review  |              |
|            | criterion separately. The performance    |              |
|            | data will be collected from the FastLane |              |
|            | database. This will be fully implemented |              |
|            | in FY 2001.                              |              |
+------------+------------------------------------------+--------------+
| Diversity  |                                          |              |
+------------+------------------------------------------+--------------+
| NSF        | Investment Goal 11:                      | New goal in  |
| Applicants |                                          | FY 2000.     |
|            | NSF will identify mechanisms to increase |              |
|            | the number of women and                  | **FY 2000:** |
|            | under-represented minorities in the      | Goal         |
|            | proposal applicant pool, and will        | achieved.    |
|            | identify mechanisms to retain that pool. |              |
|            |                                          |              |
|            | **Result:** NSF identified and put into  |              |
|            | place mechanisms to increase the         |              |
|            | diversity of NSF applicants.             |              |
+------------+------------------------------------------+--------------+
|            |                                          |              |
+------------+------------------------------------------+--------------+
| ### P      | ### Performance Goals f                  | #            |
| erformance | or Investment Process {#performance-goal | ## Results { |
|  Area {#pe | s-for-investment-process-3 .Box-Heading} | #results-8 . |
| rformance- |                                          | Box-Heading} |
| area-5 .Bo |                                          |              |
| x-Heading} |                                          |              |
+------------+------------------------------------------+--------------+
| Facilities |                                          |              |
| Oversight  |                                          |              |
+------------+------------------------------------------+--------------+
| Co         | Investment Goal 12:                      | **FY 1999:** |
| nstruction |                                          | Goal         |
| and        | Maintain FY 1999 goal to keep            | achieved.    |
| upgrade    | construction and upgrades within annual  |              |
|            | expenditure plan, not to exceed 110      | **FY 2000:** |
|            | percent of estimates.                    | Goal         |
|            |                                          | achieved.    |
|            | **FY 1999 Result:** Majority of          |              |
|            | facilities were within 110 % of annual   |              |
|            | spending estimates.                      |              |
|            |                                          |              |
|            | **FY 2000 Result:** Of the eleven        |              |
|            | construction and upgrade projects        |              |
|            | supported by NSF, all were within annual |              |
|            | expenditure plans; most were under       |              |
|            | budget.                                  |              |
+------------+------------------------------------------+--------------+
|            | Investment Goal 13:                      | **FY 1999:** |
|            |                                          | Goal         |
|            | Maintain FY 1999 goal to keep            | achieved.    |
|            | construction and upgrades within annual  |              |
|            | schedule, total time required for major  | **FY 2000:** |
|            | components of the project not to exceed  | Goal not     |
|            | 110 percent of estimates.                | achieved.    |
|            |                                          |              |
|            | **FY 1999 Result:** Majority of          |              |
|            | facilities on schedule.                  |              |
|            |                                          |              |
|            | **FY 2000 Result:** Of the eleven        |              |
|            | construction and upgrade projects        |              |
|            | supported by NSF, seven (64%) were       |              |
|            | within the annual schedule goal and four |              |
|            | were not.                                |              |
+------------+------------------------------------------+--------------+
|            | Investment Goal 14:                      | **FY 1999:** |
|            |                                          | Goal did not |
|            | For all construction and upgrade         | apply in FY  |
|            | projects initiated after FY 1996, keep   | 1999.        |
|            | total cost within 110 percent of         |              |
|            | estimates made at the initiation of      | **FY 2000:** |
|            | construction.                            | Goal did not |
|            |                                          | apply in FY  |
|            | **FY 1999 Result:** Did not apply in FY  | 2000.        |
|            | 1999.                                    |              |
|            |                                          |              |
|            | **FY 2000 Result:** This goal did not    |              |
|            | apply in FY 2000.                        |              |
+------------+------------------------------------------+--------------+
| Operations | Investment Goal 15:                      | **FY 1999:** |
|            |                                          | I            |
|            | Maintain FY 1999 goal to keep operating  | nconclusive. |
|            | time lost due to unscheduled downtime to |              |
|            | less than 10 percent of the total        | **FY 2000:** |
|            | scheduled operating time.                | Goal not     |
|            |                                          | achieved.    |
|            | **FY 1999 Result:** Reporting data base  |              |
|            | under development.                       |              |
|            |                                          |              |
|            | **FY 2000 Result:** Of the 26 reporting  |              |
|            | facilities, 22 (85%) met the goal of     |              |
|            | keeping unscheduled downtime to below    |              |
|            | 10% of the total scheduled operating     |              |
|            | time.                                    |              |
+------------+------------------------------------------+--------------+

# 

# 

# V. Findings from Program Assessments and Evaluations

Findings for program assessments and evaluations completed in FY 2000
for the Outcome Goals are discussed in Section V.A., followed by agency
results for the Management Goals (Section V.B.), and Investment Process
Goals (Section V.C.). It is important to note that, with the exception
of Investment Process Goal 2, the findings for the Management Goals and
Investment Process Goals are prepared by NSF staff with the use of
central data systems, and are not judged in the COV assessment process.

The findings for Outcome Goals include summarized judgments reported in
the Committee of Visitor (COV) and Advisory Committee (AC) assessment
reports. General findings and goal ratings in COV and AC reports are
aggregated across NSF, and summarized in a qualitative format for
Outcome Goals 1, 2, 3, and 4a, in the following Section V.A.. The COV
and AC reports are also the sources of results for Investment Goal 2,
presented in Section V.C.

For each Outcome Goal, a brief introduction to the Goal is provided,
followed by the annual performance goal and indicators for this fiscal
year. Aggregated results of the assessment process follow the
Performance Indicators, and a discussion of performance and plans for
the next year is presented. Examples of results demonstrating successful
performance as identified by COVs and ACs in the assessment process are
presented to illustrate the impact of NSF support. Examples of results
are organized by goal and area of emphasis, as described in the FY 2000
Performance Plan. Examples of Outcome results may be relevant to more
than one goal or more than one area of emphasis.

In addition to the assessments provided by COVs and ACs, studies and
evaluations are carried out by independent contractors to address
specific issues not specifically linked to the GPRA performance goals.
Evaluations completed in FY 2000 are presented in Table 2 in Section
V.D., following the Investment Process goal discussion, and for the most
part, are not used in the performance assessment process (with the
exception of one COV report as noted in Table 2). Information from
evaluations is useful to programs to identify issues and opportunities
for future investments.

## 

## 

## A. Outcome Goals and Results 

The true value of NSF investments can only be measured by the outcomes
identified over time. Outcomes might be the results of research or
training sponsored by NSF, as long as 10 or 20 years ago. Therefore,
NSF's Outcome Goals are very long-term goals, designed to ensure the
progress of science and engineering, and to improve the future health,
security, and quality of life for U.S. citizens. They focus on the
results of NSF awards for research and education in science,
mathematics, and engineering and are designed to promote the mission of
NSF. The key strategy for success in achieving these goals is the use of
rigorous merit review to make awards for activities that will influence
research and education, both directly and indirectly, over the
long-term.

Outcome Goals are expressed in a qualitative format. To determine the
progress NSF makes in achieving these goals, the outputs and outcomes of
NSF programs are judged qualitatively against the stated goals by groups
of external evaluators known as Committees of Visitors (COVs) and
advisory committees (ACs). More information about COVs and ACs is
provided in Section III, "*Assessment and Evaluation Process."*

Following the discussion of each Outcome Goal, performance results
reported in FY 2000 from awards made in earlier years are presented.
These examples include only a few of the many noteworthy achievements
reported by programs, Committees of Visitors and Advisory Committees in
FY 2000. The examples are selected to cover the full range of activities
supported by NSF and illustrate the impact and success of NSF programs
and offer only a glimpse of NSF's broad range of supported activities.
In each case a grant number issued by NSF can be used to identify the
example for purposes of verification.

+-----------------------------------------------------------------------+
| ## Performance Indicators                                             |
+-----------------------------------------------------------------------+
| -   importance and quality of discoveries, new ideas, new tools, and  |
|     new technologies;                                                 |
|                                                                       |
| -   interplay of disciplinary and interdisciplinary research; and     |
|                                                                       |
| -   balance of the portfolio.                                         |
+-----------------------------------------------------------------------+

### Baseline:

Pilot projects used FY 1997 and FY 1998 information and expert judgment
in performance assessments that indicated NSF was successful in meeting
this goal.

## Comparison of actual performance with projected performance 

Approximately one-third of NSF's portfolio of programs was assessed by
Committees of Visitors (COVs) for progress in achieving this Outcome
Goal. (Section III, "*Assessment and Evaluation Process*," contains
information on the process of evaluating NSF programs). For FY 2000,
evaluators were asked to judge whether programs being evaluated were
successful or not in meeting the goal.

In aggregating results for the agency, the reports of COVs and Advisory
Committees were used, taking into account only those reports with
substantive comments and ratings which were clearly justified. We find
that all reports that provided a rating for this goal judged NSF
successful in meeting this goal in FY 2000. Therefore, we report this
goal as achieved.

Each year, NSF asks COVs to examine the portfolio of project support to
identify activities they would characterize as high risk,
multidisciplinary, or innovative, and to make an assessment of the
overall scientific quality and balance with respect to these specific
characteristics.

NSF identified "Balance of innovative, risky, interdisciplinary
research" as an area of emphasis in FY 2000, and stated it as a goal in
FY 1999. In FY 1999, of the COV reports that gave an opinion on balance
of projects in the programs under review, most indicated that the
balance was appropriate. For FY 2000, of the COV reports that gave an
opinion on the balance, more than half indicated good balance, less than
half indicated programs could fund more high-risk projects, and a few
indicated they would like to see more innovative proposals.

## Comparison: FY 1999 - FY 2000 

This Outcome Goal was continued from FY 1999 with one modification. In
FY 1999, the goal was stated using two levels of achievement:
*successful* and *minimally effective*, with indicators for each level.
Based on comments from COVs and ACs in FY 1999, NSF determined that the
definitions for the *minimally effective* level of performance did not
provide additional information in evaluating the programs.

In FY 2000, the indicators were refined to improve correspondence
between information sought and information that can actually be
collected, and the *minimally effective* standard was removed. A single
definition for the *successful* standard is stated as the target level
of performance for each Outcome Goal. In FY 2000, a stricter definition
of allowed success was applied when reviewing reports of external
evaluators, which required clear justification of ratings in reports.

The successful result in FY 2000 was also the finding by COVs and ACs in
FY 1999. It is important to recognize that the evaluation was carried
out on a different subset of NSF's portfolio and by a different group of
external evaluators.

## FY 2001 and beyond

This goal will be incorporated under a new Outcome Goal heading for FY
2001, which rearranges NSF's five outcome goals into three broad
Strategic Outcome areas: People, Ideas, and Tools. A table depicting the
structural rearrangement is shown in Section VIII, *"Transition to FY
2001 and Beyond*." This improves the alignment of NSF's Outcome Goals
with its mission and allows closer correlation between budget categories
and NSF's Strategic Plan. This Outcome Goal will be combined with FY
2000 Outcome Goal 2 to become part of the Ideas Strategic Outcome area
as described in the NSF FY 2001 Performance Plan.

# Examples[^3] of FY 2000 Achievements Cited by External Evaluators

## Outcome Goal 1

### Discoveries At and Across

### the Frontier of Science and Engineering

External evaluators cited the following examples of results from NSF
awards as demonstrating success in support of Outcome Goal 1. These
examples illustrate important discoveries, new knowledge and techniques,
both expected and unexpected, within and across traditional boundaries,
and high-potential links across these boundaries.

The examples also illustrate NSF-supported results reported in the FY
2000 areas of emphasis for this Outcome Goal. These areas include
balance of innovative, risky, interdisciplinary research; new types of
scientific databases and tools to use them; life in extreme
environments; biocomplexity; and nanoscience and engineering. It is
interesting to note that many results cross the boundaries between
discoveries, new knowledge, interdisciplinary research, biocomplexity,
and nanoscience. Where results are forthcoming, the diverse portfolios
of awards show potential for significant impact in many of these areas.

-   **[Mapping the Arctic Ocean Floor]{.smallcaps}** A most impressive
    example of using innovative tools and, as a result, developing new
    databases, is the mapping of the Arctic Ocean floor using the
    nuclear submarine USS Hawkbill, and the Seafloor Characterization
    and Mapping Pods. The resulting data sets of high-resolution and
    narrow-beam bathymetry as well as chirp sub-bottom profiles will
    revolutionize Arctic Ocean modeling and have driven the development
    of advanced visualization techniques and multi-dimensional
    Geographic Information Systems. Sidescan images from the Lomonosov
    Ridge crest, collected during the Hawkbill mapping, show an ice
    scoured appearance marked by ploughmarks several kilometers long and
    several hundred meters wide. The ploughmarks are generally parallel,
    pointing to either the Barents Sea, or the Arlis Plateau area, as
    source regions of the ice. The parallel nature and size of the
    ploughmarks suggests grounding of a floating ice shelf rather than
    scouring of individual iceberg keels.

-   **[Interdisciplinary Research at Home]{.smallcaps}** Most U.S.
    archaeologists study the Native American past, yet very few are
    Native Americans themselves. This has often created sharp
    disagreements between these two groups. To help bridge this gap the
    Society for American Archaeology has established a fellowship
    program that allows Native Americans to participate in both field
    and traditional academic settings. Although most will not become
    professionals in archaeology, the goal is to develop a cadre of
    individuals who can act as translators and mediators between two
    often divergent cultures. NSF funding has helped to increase the
    size and number of the fellowship awards.

-   **[Bending Light Around Corners -- Improving
    Telecommunications?]{.smallcaps}** Telecommunication wavelengths are
    normally considered to be in the wavelength range where optical
    fiber has the lowest loss. Conventional single mode fibers have two
    low attenuation ranges, one about 1.3 micrometers, and another about
    1.55 micrometers. Between these two ranges there is a high
    attenuation range, 1.35-1.45 micrometers, due to the presence of the
    OH radical. What\'s used in telecommunications also depends on the
    light sources and amplifiers available. NSF-supported researchers
    have created "omniguides" -- or phototonic bandgaps using
    alternating concentric layers of polystyrene (plastic) and tellurium
    (a metal) having specified thickness. These "omniguides" cause
    complete internal reflection of photons, regardless of the direction
    of polarization of the light, and allows the guiding of light around
    sharp corners. Depending upon the tube diameter, the guides can be
    tuned over a wide range of wavelengths, for use anywhere from
    CO~2~-laser (about 10 micrometers -- one inch is 25,400 micrometers)
    to telecommunications wavelengths (between 1.3 and 1.55
    micrometers). *Science* cited this discovery as one of its Top 10
    "Breakthroughs of the Year."

-   **[Quantum control - quantum optics]{.smallcaps}** Precise control
    and measurement of a variety of quantum systems were demonstrated
    that could have profound implications for nanoscale technology,
    chemical physics, and information science. The first completely
    quantum feedback scheme was developed, which followed the
    development of a scheme for the complete characterization of the
    quantum state of the internal degrees of freedom of atoms and
    molecules. Techniques developed for laser cooling of atoms led to
    the improvement of optical tweezers that are now capable of holding
    and moving individual molecules. An important example is the
    combination of techniques from biology, chemistry, and physics to
    manipulate single DNA molecules.

-   **[Tools to Benefit Medical applications]{.smallcaps}**
    NSF-supported researchers have developed a needle-shaped accelerator
    tube that, when installed on a particle accelerator, can be used to
    deliver tumor-destroying neutron radiation directly to a tumor with
    minimum damage to healthy tissue. The prototype is undergoing
    engineering studies in preparation for studies on prostate tumor
    irradiation.

-   **[advancing knowledge - Shared Protein Structure
    Database]{.smallcaps}** With the tremendous increase in the amount
    of DNA sequence information now available, the opportunity exists to
    characterize the structure and function of all proteins. The support
    of a world-wide protein database was facilitated by NSF\'s long-term
    (\~25 years) commitment to support a world-wide protein database
    developed by universities in cooperation with a national laboratory.
    The database is serving an international community of researchers
    (60% US, 30% European, 10% Japanese) interested in protein
    structure. X-ray coordinates are deposited into this database is
    then available to the scientific community world-wide. This
    NSF-sponsored protein database is the only one in the world and
    includes many features that will serve the advancing genome
    initiatives at NSF and other agencies in this country and throughout
    the world.

-   **[Computational Biology]{.smallcaps}** Research in molecular
    biology confronts many problems of high computational complexity.
    Large amounts of genomic data have been collected that require
    high-speed algorithms for searching, analysis, and prediction of
    function. Pattern-matching methods developed by the theoretical
    computer science community were instrumental in expediting the
    sequencing of the human genome. New algorithms for generating
    phylogenetic trees are used in inferring evolutionary development of
    species. NSF-supported research in computational biology has
    contributed extensively to phylogenetic tree algorithms as well as
    biological sequence pattern-matching and the specific problem of
    finding repetitions in genomic data. Using NSF support, researchers
    developed a much more efficient algorithm for correlating diseases
    with genetic defects.

> *Life in Extreme Environments*, begun as a focused investment theme in
> FY 1997, reflects an aspect of Biocomplexity in the Environment. NSF
> awards produced a wide variety of important discoveries in both the
> Arctic and Antarctic. Many discoveries concern regional environmental
> changes that have implications for global climate change.

-   NSF interacts with several other federal agencies (Coast Guard,
    NASA, Army, Air Force, NOAA, USGS, and CIA) and is involved in
    interagency funding of many projects. Recent conclusions of a
    jointly-supported NSF and NASA research project have yielded new
    insights on a controversial subject, the evidence of possible life
    on early Mars. It was found that the carbonate minerals, one of the
    key components at the center of the controversy, originated through
    multiple inorganic processes rather than through biological
    processes, and that isotopes of iron record evidence of biological
    fractionation. As a result, iron isotopes can now be used as a new
    tool for recognizing potential evidence of life.

-   In the Arctic, the international Surface Heat Budget of the Arctic
    (SHEBA) project demonstrated the increased importance of low clouds
    in warming the lower atmosphere and melting sea-ice The SHEBA Ocean
    project involved placement of the first-ever, year-long science
    program in the drifting Arctic ice pack. SHEBA was conducted from an
    icebreaker frozen in place 300 miles north of Prudhoe Bay, AK, but
    which drifted over 400 miles to a position 400 miles north of
    Barrow, AK. Upon arrival, scientists immediately confirmed that a
    major ice melting event in Summer, 1997, had thinned the ice pack
    and left thin ice conditions well into 1998. The cross-directorate,
    interagency (ONR, DOE, NASA, NOAA), and international (Japan,
    Canada) science project has collected a suite of ice, atmosphere,
    and ocean measurements to determine the environmental variables
    responsible for maintenance of the climatically important Arctic ice
    pack. The measurements address some of the most important unknowns
    required for improving computer simulations of climate change,
    weather predictions, and satellite retrievals.

-   **[Extrasolar Planetary Discovery]{.smallcaps}** The first detection
    of a multiple-planet solar system outside our own has been widely
    interpreted as evidence that solar systems like ours may be fairly
    common companions to sun-like stars. A long-standing aim of many
    astrophysicists has been to detect and characterize sun-like
    pulsations in distant stars. The technology required to make such
    studies involves extremely precise measurements of the line-of-sight
    velocity or brightness of the target stars. It turns out that these
    measurements are precisely those needed to detect planets circling
    other stars. Seeking to identify new extra-solar planets provided an
    exciting result in April: the discovery that three planets orbit the
    star Upsilon Andromedae, each with a mass comparable to the mass of
    Jupiter. The three planets are located at distances from their star
    that range from .05 to 2.5 astronomical units -- one astronomical
    unit is the distance between the earth and the sun. This discovery
    was the result of a collaboration involving NSF-supported scientists
    at the Harvard-Smithsonian Center for Astrophysics and San Francisco
    State University, using the Anglo-Australian Telescope.

> *Biocomplexity* represented a focused emphasis opportunity for NSF in
> FY 2000. However, NSF had made related investments in previous years
> and related investment outcomes underpin this FY 2000 emphasis area.

-   **[Unexpected Discovery]{.smallcaps}** The structures of proteins
    that catalyze steps in metabolism and that orchestrate growth and
    development are specified by the genetic code in DNA. Quality
    control mechanisms exist at several levels to ensure that all
    proteins are produced exactly according to genetic instructions. The
    genetic code is translated into protein structural information
    through an intermediary called messenger RNA (mRNA), which is a
    transcript of the information in the gene. A quality control
    mechanism called RNA surveillance has recently been discovered that
    ensures that all mRNAs produce full-length functional proteins. RNA
    surveillance is accomplished by a mechanism that causes the rapid
    destruction of mRNAs that have mistakes in them that prevent their
    coding of full-length proteins. NSF-supported research at the
    University of Wisconsin led to the discovery of a novel and
    unanticipated pathway for surveillance of aberrant mRNA molecules.
    Components of this pathway were identified in yeast using a clever
    genetic selection initially designed to identify factors that affect
    ribosomal frame shifting. Instead, a novel set of genes was
    identified that encodes components of a pathway that mediates
    turnover of mRNAs containing nonsense mutations. This discovery
    offers an explanation for the long-standing problem of how cells
    contend with toxic proteins resulting from translation mRNAs
    containing nonsense or frame shift mutations.

-   **[Evolutionary Relationships]{.smallcaps}** A recent series of
    discoveries grew out of the field recovery and analysis of fossil
    dinosaurs, birds, and mammals from the Gobi Desert. The expeditions
    recovered a wealth of fossil material. Analysis of this matrix
    showed, among other things, that birds had a complex origin from
    therapod dinosaurs. The large data base gathered, in part, with
    support from NSF is important not only to the understanding of
    animal life in the Gobi Desert, but to the understanding of the
    evolution of vertebrates worldwide.

-   **[Plant Growth and Development]{.smallcaps}** Studies of basic
    plant developmental mechanisms include studies of the molecular
    genetics of plant cells and tissues that lead to root and root hair
    development. The plant root and root hair allows the plant to absorb
    or restrict nutrients that are present in the soil environment. The
    success of studies of this sort sheds additional information on root
    uptake mechanisms to allow for future work on varying nutrient
    uptake and sequestration by the plant. The development of the shoot
    and root apical cells and tissues is considered the "holy grail" of
    plant developmental biology because these two structures give rise
    to all above and below ground parts of all plants. Several
    NSF-supported research groups have lead the field in identifying
    genes that are necessary to initiate these cells and tissues during
    embryo development as well as maintain their organization throughout
    the growth of the plants.

> Nanoscience and engineering represented a focused investment emphasis
> for NSF in FY 2000, an emphasis that builds upon the following
> discoveries and others like them.

-   The development of sophisticated nanoscale optical measurement
    techniques that are broadly useful for the study of very fast
    dynamics in excited atoms, chemical reactions, carrier motions in
    semiconductors, and nanoelectronic devices is resulting from a
    breakthrough in stable short-wavelength, short time- duration
    lasers. The innovative work of NSF supported researchers has
    appeared in *Science* 280, 1412 (1998) and *Nature* 406, 164 (2000).
    One NSF-supported researcher was recently recognized by the John D.
    & Catherine T. MacArthur Foundation Award for 2000.

-   NSF support has led to new understanding of manufacturing processes
    and equipment that hold great promise for the future. As the size of
    all kinds of electromechanical devices becomes smaller and smaller,
    accurate measuring devices are needed to enable manufacturing and
    ensure product quality. NSF-supported researchers have:

```{=html}
<!-- -->
```
-   Collaborated to develop the world's highest-resolution and
    highest-accuracy magnetic suspension positioners. These positioners
    have been used to demonstrate the principles of ultra-precision
    positioners for semiconductor processing and advanced imaging
    systems.

-   Made discoveries leading to two key rapid prototyping technologies -
    selective laser sintering and 3D printing, respectively. These
    projects addressed fundamental interdisciplinary research issues in
    materials science and manufacturing processes. NSF support provided
    since the late 1980's has played a huge role in the evolution of
    rapid prototyping from an emerging technology to the mature field
    with commercial applications that it is today.

-   Studied precision engineering for high-quality products has resulted
    in major findings in grinding and metrology, both important for
    traditional manufacturing processes.

```{=html}
<!-- -->
```
-   Nanoscale molecular engineering of surfaces has been achieved by
    NSF-funded investigators in their creation of molecular corrals a
    few hundred angstroms in diameter and only one molecular layer deep.
    These molecular corrals have potential to serve as containers
    wherein a variety of biologically active chemical receptors could be
    anchored, providing a new basis for future sensor design and
    application. Other advances in nanoscale design and supramolecular
    self-assembly are bringing the diverse fields of synthetic and
    analytical chemistry, physics, materials science, mathematics, and
    information technology together. For example, families of
    mechanically interlocked molecules called rotaxanes and catenanes
    form the architectural foundation of a nanoscale machine that can be
    switched from one state to another - representing a molecular logic
    gate. These molecular logic gates are being used in ongoing efforts
    to design prototype molecular computers.

+-----------------------------------------------------------------------+
| ## Performance Indicators                                             |
+-----------------------------------------------------------------------+
| -   outputs and outcomes of NSF awards are made available to and put  |
|     to use by others; and                                             |
|                                                                       |
| -   NSF-sponsored activities demonstrate a role in stimulating        |
|     innovation and policy development.                                |
+-----------------------------------------------------------------------+

### Baseline:

Pilot experiments using FY 1997 and FY 1998 information and expert
judgment in performance assessment by external experts indicate NSF was
successful in meeting this goal.

## Comparison of actual performance with projected performance

COVs were asked to judge whether the programs being evaluated were
successful or not in meeting the FY 2000 performance goal for this
Outcome Goal. Programs evaluated in FY 2000 were judged successful by
experts in a significant majority of the reports. Several reports
indicated that programs were successful in a limited context, a few
reports indicated that programs were not fully successful, and a few
reports did not provide judgements. Issues identified in FY 2000 are
similar to those reported in FY 1999.

For those programs rated not fully successful, one was found to have
awards that limited the scope and duration of the activity. Hence,
connections between discoveries and service to society were not
described in reports although some proposals had promised such
connections. One report found the programs under review to be generally
successful, but noted that room for improving the delivery of scientific
research results to society, or more specifically to end-user
communities, could be made. Another program not fully successful is
described as funding primarily "basic" science, and hence it was left to
others to make applications to society. Finally, one program was found
to be producing results that have benefited a small community of users,
including students and educators, and is showing promise for a much
wider applicability, but insufficient time has elapsed for the products
of this program to have penetrated into the potential broad user
community.

We find, from aggregating the results of all reports which rated this
goal, and using only reports with substantive comments and ratings which
were clearly justified, that the majority of reports from external
evaluators indicate that most NSF programs evaluated were successful in
meeting this goal in FY 2000. Therefore, this goal was determined to
have been achieved in the aggregate. However, as was noted in FY 1999,
there is room for improvement in some programs. For those activities
that were not judged fully successful, increased award size and duration
are recommended by evaluators. NSF is emphasizing award size and
duration as explicit management goals in FY 2001.

## Comparison: FY 1999 - FY 2000

This goal was continued from FY 1999, with some modification of
indicators to improve the correlation between information available and
the intent of the goal. In FY 1999, the goal was stated using two levels
of achievement: *successful* and *minimally effective*, with indicators
for each level. Based on comments from COVs and ACs, it was determined
that the definitions for the *minimally effective* level of performance
did not provide additional information in evaluating the programs.

In FY 2000, a single definition for the *successful* standard was used
as the target level of performance. A stricter definition of allowed
success was applied that required clear justification of ratings in
reports. The overall result in FY 2000 identified issues similar to
those identified in FY 1999, even though the evaluation was carried out
on a different subset of NSF's portfolio, and by a different group of
external experts.

## FY 2001 and beyond

NSF can conduct outreach and awareness efforts, thus encouraging efforts
toward connections but, generally, cannot mandate connections for all
awards. NSF communicates the importance of its Outcome Goals, investment
strategies, and expectations for the set of awards to the science and
engineering community. Staff outreach efforts are emphasized for
activities with strong potential to serve society. Regular reporting
requirements for all awards help program staff understand the outputs
and outcomes of their award portfolio and provide the context for
decisions on new awards. Many investigators do not think about the
possible connections their work might have in serving society. Many
potential users are not aware of results from NSF awards that could be
useful to them.

This Outcome Goal will be incorporated under a new Strategic Outcome
Goal heading for FY 2001 which rearranges NSF's five Outcome Goals into
three broad strategic Outcome areas: People, Ideas, and Tools. A table
depicting the new organization is shown in Section VIII of this report,
"*Transition to FY 2001 and Beyond*." The change to People, Ideas and
Tools improves the alignment of NSF's goals with its mission and allows
closer correlation between budget categories and NSF's Strategic Plan.
This Outcome Goal will be combined with FY 2000 Outcome Goal 1 to become
part of the Ideas Strategic Outcome in FY 2001. Results obtained in FY
1999 and FY 2000 have led NSF to refine this goal and to identify ways
to improve data/information collection to assess this goal.

# FY 2000 Examples[^4] of Achievements Cited by External Evaluators

## Outcome Goal 2

### Connections between Discoveries and their use in Service to Society

External evaluators cited the following examples of results from NSF
awards as demonstrating the criteria for success in support of Outcome
Goal 2. These examples made the connections between discoveries and
their use in society, were rapidly and readily available, and were used
as appropriate in education, policy development, or by other federal
agencies or the private sector.

The examples below are shown to illustrate the variety of results of NSF
awards reported in FY 2000. A few examples also demonstrate results in
areas of emphasis, which include elements of Information Technology
Research (ITR), Global Change, Research on Learning and Education, Plant
Genome Research, Urban Communities, and Science and Technology Centers -
Integrative Partnerships. The diverse portfolio of FY 2000 awards
promise significant impact in one or more of these areas.

-   **[Up-to-Date local weather information]{.smallcaps}** The
    Auto-Nowcaster system, jointly sponsored by the Federal Aviation
    Administration, the Department of the Army, the National Weather
    Service, and NSF under the U.S. Weather Research Program, provides
    one-hour Nowcasts of thunderstorms and strong winds. Demonstrations
    of the Auto-Nowcaster system were held at weather forecast offices
    of the National Weather Service, the Army Forecast Office, and the
    Aviation Weather Center. The demonstrations were highly successful
    the products are extensively used by operational personnel. The
    Sterling Virginia National Weather Forecast Office's severe storm
    warnings for 1998 were far more accurate than any previous year, and
    they give partial credit to the Auto-Nowcaster system for the
    improvement.

-   **[Anticipating Power Shortages]{.smallcaps}** Research sponsored by
    NSF has catalyzed interaction between government, academe and
    industry to achieve breakthroughs with immediate and lasting impact
    on society. In a multi-university center effort, researchers have
    discovered new methods to anticipate \"brownouts\" in electric power
    systems. Software has been developed to quickly assess the transfer
    capability and operational margins of electric power systems, and
    software is currently being implemented in electric utilities. Seven
    of the participating researchers were appointed by the Secretary of
    Energy to study last summer\'s blackouts and they were asked to make
    recommendations about the federal role in reducing future failures.

-   **[Impacting Telecommunications]{.smallcaps}** A microphone-array
    technology developed with ten years of NSF support has demonstrated
    both high-quality sound pick-up and the ability to identify and
    direct a camera to the speaker in a group of up to five people in a
    room. The sound quality achievable is comparable with face-to-face
    sound quality. Scientific advances underlying these capabilities
    include the development of new beam-forming algorithms, advanced
    hardware for real-time processing of multiple microphone inputs, and
    fast location algorithms. As a result of these breakthroughs, a
    major teleconferencing company has licensed the technology, and will
    develop a commercial product based on their prototype.

-   **[A Better Look on Life]{.smallcaps}** Biological instrumentation
    and instrument development as well as training programs at the
    undergraduate, graduate and postdoctoral levels have led to
    invention of the confocal microscope and, more recently, the
    development of both \"two photon\" and \"near-field scanning
    optical\" microscopes. Because of these developments, confocal
    microscopy has become a standard component of laboratory
    instrumentation important to the area of cell biology. Advances in
    cell biology have, in turn, resulted in a better understanding of
    the basic biological processes in plants and animals. Using the
    microscopy now available through NSF funding, the private sector has
    commercialized high-technology products that have been marketed both
    in the U.S. and abroad. The development of the two-photon microscope
    allows one to optically section cells, to follow the dynamics of
    intracellular movements in living cells, and to reconstruct the
    three dimensional structure of cells at different stages of
    development or in response to environmental signals. This instrument
    has revolutionized how scientists in all areas of cell biology view
    and study cell function. This microscope was commercialized, and is
    in great demand by the scientific community.

-   [**Impact on Industrial/Agricultural Productivity**]{.smallcaps}
    Methodologies that facilitate higher yields and better selectivities
    for chemical processes, and that systematically optimize the
    performance and integration of chemical processes, are important for
    maintaining and enhancing global competitiveness and lead to a large
    positive balance of payments in the chemical industry. NSF research
    projects aim at improvement of processes with potentially large
    economic gains.

```{=html}
<!-- -->
```
-   NSF-supported studies of the fundamentals of "thermal switch
    membranes" have had important results. The membranes are made from
    polymers with long side chains that crystallize. Switching membranes
    have been designed that open or close to particular molecules
    depending on temperature. This characteristic has been exploited to
    form coatings on seeds to control germination by blocking moisture
    permeation at low temperatures. This leads to a savings in seed
    costs and improvements in crop yields.

-   The reuse of materials in the semiconductor industry is critically
    important in controlling both cost and environmental impact. NSF
    supported engineering researchers in collaboration with the
    Semiconductor Research Corporation (SRC) have developed reactive
    membrane technology for removing trace impurities from gases and
    treatment systems for the production and recycle of ultra-pure water
    using photoactive catalysts. Four patents have resulted from the
    work, and members were recently recognized for their leadership by
    the Landmark Innovation Award.

> This research area has vast potential implications for smart networks,
> wireless networking and telecommunications, speech and image
> processing, access and retrieval of data, and processing of sensor
> data.

-   **[Checking Systems specifications]{.smallcaps}** Nearly twenty
    years of NSF support has resulted in major contributions in the
    mathematical foundations for verifying the correctness of hardware
    and software. An NSF-supported researcher was a co-recipient of the
    Association for Computing Machinery Kanellakis Award in 1999 for the
    development of Symbolic Model Checking. Symbolic Model Checking is
    the most successful method yet devised for formally verifying that
    hardware and software systems meet their specifications. It has
    successfully uncovered subtle errors in hardware systems (such as
    dividers) and software systems (such as networking protocols) that
    extensive simulation failed to identify, and has been adapted by
    such companies as Intel, Motorola, IBM, and Siemens.

-   **[Correcting for Data Lost in Transmission]{.smallcaps}** One of
    the basic building blocks of most communications over the Internet
    is known as transmission control protocol, or TCP. In spite of its
    ubiquity, TCP has been poorly understood. NSF-supported researchers
    have developed a simple mathematical model for predicting TCP
    performance This model shows that transmission behavior is not what
    was expected. This work is having a significant impact on the
    continuing evolution of TCP and the design of new transport
    mechanisms. It also shows substantial benefit in the use of forward
    error correction (FEC) in the delivery of large data files between a
    single sender and many receivers. FEC provides a way to correct for
    data that is lost in transmission. One consequence of this work is
    that most multicast transport mechanisms now rely on the use of FEC.
    The researchers have been recognized with a prestigious award from
    the Institute of Electrical and Electronics Engineers (*IEEE*) for
    this work.

> By any criteria, NSF's support of the sequencing of the first plant
> genome is an impressive example of how a high-quality research
> resource can be generated, maintained, and made available worldwide.
> This sequencing effort, started in 1996, was coordinated through the
> *Arabidopsis* Genome Initiative (AGI) and an international consortium
> with two European and one Japanese laboratory.

-   ***Arabidopsis* Genome** Completion of the *Arabidopsis* genome
    sequence at the end of 2000 was a truly remarkable achievement. Work
    with this model plant, *Arabidopsis*, has led to a detailed
    understanding of the molecular and genetic control of flower
    development. Initial conclusions have generated great excitement in
    the science community since it appears there is significant
    evolutionary variation in the mechanism of flower patterning, and
    some of these variations may explain the variation seen in flower
    morphology in nature. Not only will this information be useful to
    researchers in public institutions and universities, it will be
    useful to the private sector as well. The sequence data will be used
    by biologists to compare and contrast the structure and function of
    similar protein domains across different kingdoms. To complement
    this research resource, a separately-funded project maintains an
    *Arabidopsis* Stock Center at Ohio State University. From here seed
    stocks are made available to the research community world-wide.

+-----------------------------------------------------------------------+
| ## Performance Indicators                                             |
+-----------------------------------------------------------------------+
| -   demographic data on participants in NSF-funded activities and in  |
|     the workforce;                                                    |
|                                                                       |
| -   character of experiences in NSF-funded activities aimed at        |
|     educating the next generation of the workforce; and               |
|                                                                       |
| -   outcome data from longitudinal studies as available.              |
+-----------------------------------------------------------------------+

### Baseline 

Preliminary efforts in FY 1997 and FY 1998 to pilot the use of expert
judgment in performance assessment indicated NSF was successful in
meeting this goal.

## Comparison of actual performance with projected performance 

As indicated by NSF's FY 2000 Performance Plan, exceptionally strong
performance in this goal is characterized by external recognition of
scientists or engineers who received NSF support during their training;
and when the production of degree recipients in science, mathematics,
and engineering increases markedly for under-represented groups.

NSF's performance toward this goal was judged successful in the
aggregate by external experts in committee reports with respect to
achieving a globally oriented workforce**,** but not fully successful
with respect to achieving diversity or increasing participation of
under-represented groups**.** Using only reports with substantive
comments and ratings that were clearly justified for *both* areas, we
find that overall, the majority of reports from external experts
indicate that NSF was not successful in meeting *both* areas of this
goal in FY 2000. However, programs specifically designed to increase
diversity and those designed to achieve a globally-oriented workforce
were judged to be successful.

Some COV reports noted that improvements have been made in the past
year. However, numbers of under-represented groups are still low and
should be increased. One report notes that the programs reviewed in FY
1999 did not achieve this goal, but that programs assessed this year did
achieve it.

From a sampling of reports which rated programs successful in a limited
context, one report notes that despite excellent efforts to fund
activities that increase diversity, there does not yet appear to have
been an increased participation of under-represented groups in the
scientific workforce. Reasons for this remain elusive and may include
time lags between intervention and effect.

One report notes that while the funding rate for women is not
statistically different from that for males, the number of female
proposers is significantly less than the number of male proposers.
Reports note that factors affecting improved performance in achieving
this goal are not always evident.

One report states that although NSF program officers work to involve
under-represented individuals in the range of NSF activities, increased
effort is needed, possibly through involvement of individuals from
undergraduate institutions. One report notes that recruitment efforts
for minority students have had only limited success. Other report
recommendations include recruiting young scientists into the field, and
in order to reduce attrition rates, nurturing them once they have
started. Another report indicates the participation of under-represented
groups in the workforce is low and slowly increasing, but that it is not
possible to make an unequivocal assessment of the impact of NSF
programs. One report states that a full evaluation of progress toward
this goal cannot be determined in a three year period, although it rates
the programs being evaluated as successful.

Experts agree that the current workforce does not meet national needs.
They also agree that NSF programs on the whole are successful, but may
not be sufficient to meet the national challenge. Changes in American
society may be necessary to bring about the desired change.

## Comparison FY 1999 -- FY 2000 

This goal was continued from FY 1999, with some modification of
indicators made in FY 2000 to improve the correlation between
information available and the intent of the goal.

In FY 1999, the goal was stated using two levels of achievement:
*successful* and *minimally effective*, with definitions for each level
of performance. In FY 1999, programs judged by external evaluators were
rated successful in achieving all or most aspects of this goal in most
reports. Several reports qualified their ratings by indicating that NSF
should do more in the area of showing increased participation of
under-represented groups. Based on comments from COVs and ACs in FY
1999, it was determined that the definitions for the *minimally
effective* level of performance did not provide additional information
in evaluating the programs.

For FY 2000, the indicators were refined to improve correspondence
between information sought and information that can actually be
collected. A single definition for the *successful* standard was used as
the target level of performance. A stricter definition of success was
applied when aggregating results, which required clear justification of
ratings in reports. As a result of using stricter definitions of
success, we have reported this goal as "not achieved" in the aggregate
for FY 2000 as opposed to "achieved" in FY 1999. However, we note that
many reviewers comment that NSF is making serious efforts to increase
participation of individuals from under-represented groups, even though
the numbers remain small.

## Steps to meet this goal in the future 

Evaluating the impact of NSF support in achieving diversity or
increasing the participation of under-represented groups is a long-term
ongoing challenge for NSF. Part of the challenge lies in a fundamental
inability to collect adequate quantitative information that describes
the diversity of NSF stakeholders, in order to enable tracking of
results. NSF cannot mandate full reporting from participants in order to
evaluate this goal, and must rely on voluntary reporting. Such reporting
is often incomplete and inaccurate. NSF also relies upon the involvement
of the institutions it supports to create opportunities for
under-represented groups.

In spite of these challenges, NSF remains fully committed to increasing
diversity through the increased participation of under-represented
groups in science and engineering. Thus this goal remains a primary
long-term objective of the agency. Significant progress toward meeting
this goal is not expected in the short term, and will only be realized
with continued efforts and investments over many years.

## FY 2001 and beyond

This Outcome Goal will be incorporated under a new Strategic Outcome
Goal heading for FY 2001 that rearranges NSF's five Outcome Goals into
three broad Strategic Outcome areas: People, Ideas, and Tools. A table
depicting the structural rearrangement is shown in Section VIII of this
report, "*Transition to FY 2001 and Beyond*". This change improves the
alignment of NSF's goal with it's mission and allows closer agreement
between budget categories and NSF's Strategic Plan. This Outcome Goal
will be restated to avoid mixing goal objectives and indicators, and is
more fully developed under the People Strategic Outcome area in FY 2001.
In addition, it will also be placed under a new category in FY 2001,
described as "Broadening Participation". This is included under NSF's
Investment Process Goals in the FY 2001 Performance Plan.

In FY 2001, NSF will focus on increasing the participation of
individuals from under-represented groups in the merit review process
and on increasing the diversity of the NSF staff. Some NSF
organizational units have taken steps to develop a broader effort to
increase diversity within their programs by developing new programs to
increase diversity.

In FY 2000, NSF organized a working group to review its increased
diversity goals. This working group produced a plan to improve diversity
within the agency and in the reviewer pool. NSF will maintain this goal,
and will focus on achieving a diverse science and engineering workforce
within its own ranks in order to establish a more diverse leadership.
NSF will continue to review approaches for improved evaluation of the
impact programs have in achieving increased participation of
under-represented groups outside the agency. Current program
announcements ask proposers to address how the activity they propose
will impact diversity in the science and engineering workforce.

NSF provides a relatively small investment in the overall federal
investment to develop the national science and engineering workforce.
Achieving this Outcome Goal in the long-term implies a gradual change in
process and philosophy of educating the scientific, engineering, and
technological community. A commitment on the part of institutions and
their faculties to enhance the diversity of the science and engineering
workforce and to provide a broader range of educational opportunities is
needed to meet this goal.

# FY 2000 Examples[^5] of Achievements Cited by External Evaluators

## Outcome Goal 3

### A Diverse, Globally-oriented Workforce of Scientists and Engineers

External evaluators cited the following examples of results from NSF as
demonstrating the criteria for success in support of Outcome Goal 3.
Noteworthy examples taken from committee reports have also been selected
to demonstrate results in FY 2000 areas of emphasis that include
integrative research and education opportunities, and participation of
under-represented groups in integrative research and education.

These examples have also been selected to show that participants in NSF
activities experience world-class professional practices in research and
education, using modern technologies and incorporating international
points of reference; that academia, government, business, and industry
recognize their quality; and that the science and engineering workforce
has shown increased participation of under-represented groups. In some
examples, the diverse portfolios of awards show potential for
significant impact in many of these areas.

-   **[Project LEARN]{.smallcaps}** The Laboratory Experience in
    Atmospheric Research (LEARN) is a four-year teacher enhancement
    project targeted at 5th through 8th grade science teachers from
    rural schools in Colorado. LEARN is comprised of two major
    components: a summer workshop and 3 days of in-district training.
    Between October and April, LEARN staff, NSF-supported scientists and
    science educators from the Science Discovery Program at the
    University of Colorado traveled to rural regions and conducted
    three, full-day, hands-on training programs for up to 21 teachers in
    each region. The training days drew 299 teachers from eight rural
    regions. For the first day, Science Explorers, 142 teachers
    participated as a team with five of their students in a full day of
    hands-on activities. Additionally, 41 teachers from an urban
    district also participated in Science Explorers in conjunction with
    LEARN. This brought the total number of students in attendance
    to 915. The teachers returned to their classrooms with written
    curriculum, material kits, and very excited students to help them
    teach the content and activities to the rest of the class.

-   **[Computational Geometry]{.smallcaps}** Two NSF-supported research
    groups at Smith College, an undergraduate women's college, conducted
    research with undergraduates in computational geometry. One group
    discovered a combinatorial structure that underlies all planar
    linkages (bar-and-joint frameworks), a wide class of mechanisms that
    play an important role in robotics. The other group released the
    first public program for finding the shortest paths on a polyhedral
    surface from one source point to all vertices that may be useful in
    medical applications (to flatten brain maps), robotics (for
    navigation over rough terrain), and manufacturing (to unfold 3D
    shapes for planar cutouts).

-   [**Mentoring for Success**]{.smallcaps} NSF supports activities
    designed to expand opportunities for women, minorities, and persons
    with disabilities in all areas including computer and information
    science and engineering. Among its most successful projects is the
    Distributed Mentor Project. A longitudinal evaluation by the Center
    of the University of Wisconsin shows the Distributed Mentor Project
    (DMP) to be successful at meeting its primary goal of increasing the
    number of women entering graduate school in computer science and
    engineering (CS&E). Using a Baccalaureate & Beyond study conducted
    in 1994 as a comparison, the best male CS&E graduates were 10 times
    more likely to enter graduate or professional school within one year
    of graduation than the best female CS&E graduates. The figure for
    men being 29.19% of graduates, for women being 2.53% of graduates.
    Of the DMP participants, over 50% were enrolled in graduate or
    professional school the year following their graduation. In both
    cases the surveys considered only graduates with GPA's greater than
    or equal to 3.5. In each of these past years, approximately
    twenty-five undergraduate women have participated in the research
    and mentoring activities of the DMP with resounding success.

-   **[Tools that Enable]{.smallcaps}** A variety of new tools have been
    developed which enable the learning of science and mathematics by
    persons with disabilities. Included are:

```{=html}
<!-- -->
```
-   A three-dimensional, tactile model of the periodic table with
    Braille labels;

-   Documented instructions for accessible chemistry laboratory
    assignments placed on the World-Wide-Web;

-   CD-ROM-based accessible interactive math instructional games; and

-   A prototype graphical calculator for blind students using a
    force-feedback mouse.

```{=html}
<!-- -->
```
-   **[Engaging Diversity]{.smallcaps}** An example of an approach to
    engage diverse students with differing scientific and technological
    ideas and techniques is used by a center in microelectronics that
    with collaborators from industry and in cooperation with the
    Semiconductor Industry Association. The center prepared 100 teaching
    models in 19 clusters using virtual reality and CD-ROM interactive
    teaching. It has seen a 50% increase in Hispanic population
    participation.

-   **[Enabling the Disadvantaged]{.smallcaps}** An international
    project has enabled U.S. students from economically disadvantaged
    backgrounds and from under-represented groups to participate in an
    Organization of Tropical Studies (OTS) ecology course in Costa Rica.
    The students were exposed to hands-on, field-oriented research, and
    the international experience was a first for many of them. While
    some of the students had no prior familiarity with scientific
    research outside of a laboratory, and some were initially tentative
    about exploring the tropical forest and engaging in hands-on
    research, by the end of the course they had not only learned from
    their experience but also felt that the course was academically
    enriching and had provided an opportunity for personal growth.

+-----------------------------------------------------------------------+
| ## Performance Indicators                                             |
+-----------------------------------------------------------------------+
| Models and practices to improve achievement, teacher training,        |
| teacher classroom work, and student achievement.                      |
+-----------------------------------------------------------------------+

### Baseline 

Preliminary efforts in FY 1997 and FY 1998 to pilot the use of expert
judgement in performance assessment either did not address this
performance goal or did so in the context of a small base of program
activity.

In 1999, 40 NSF-sponsored projects implemented mathematics and science
standards-based curricula in over 81 percent of participating schools,
and provided professional development for more than 156,000 teachers.
All participating educational systems demonstrated some level of
improvement in student achievement in mathematics and science on a
battery of system-selected assessment instruments.

In FY 2000:

-   Three major systemic initiatives implemented mathematics and science
    standards-based curricula in over 80% of the 7,630 participating
    schools.

-   The systemic initiatives furthered professional mathematics and
    science development in over 90% of 7,630 participating schools.

-   The systemic initiatives reported improved student achievement in
    mathematics in 81% of the 4,187 schools and improved student
    performance in science in 86% of the 2,474 schools using the same
    assessments for the last three years**.**

In FY 1999, systemic initiatives and related teacher enhancement
programs provided intensive professional development to a total of
82,400 teachers, exceeding the goal of 65,000.

In FY 2000, NSF awards provided intensive professional development (60
hours or more) to a total of 89,723 teachers, exceeding the goal of
65,000 for the second year.

## Comparison of actual performance with projected performance

Activities important to achieving success toward this goal include
systemic approaches, attention to teacher preparation and development,
partnership with other agencies, digital libraries, graduate teaching
fellows as content resources in K-12 schools, and developing a strong
research base for use by practitioners.

In the aggregate, when this goal was a clear objective of the programs
being evaluated and when there was sufficient information available to
carry out the evaluation, most reports indicated NSF programs were
successful in achieving this goal. However, external evaluators were
uncertain how to assess performance where programs did not have funds
directed to these objectives, resulting in an assessment of less than
successful or no assessment in many reports.

In aggregating results and using reports with substantive comments and
ratings which were clearly justified for each area, we find NSF's
performance toward this goal was judged as successful or successful in a
limited context by a majority of external evaluators, and therefore, we
describe this result as *successful in a limited context*, and report it
to be not fully achieved in the aggregate in FY 2000.

For FY 2000, evaluators were asked to judge whether programs being
evaluated were successful or not in meeting the FY 2000 performance goal
and indicators. In arriving at an aggregated assessment, it is very
likely that programs with objectives focused primarily in the areas of
this goal are more successful in meeting this goal for these indicators
than is indicated by the FY 2000 result statement of "not achieved".
However, there remains disagreement among external evaluators as to the
overall success of meeting the broad Outcome Goal as it pertains to
"***all*** Americans".

Many external evaluators view this goal as primarily relevant to NSF\'s
educational activities, and therefore tended to rate it only when
evaluating educational programs. A significant fraction of COV reports
indicate that the goal was not met because this goal was not a priority
objective for many programs. Many reports do not rate this goal because
the experts stated that the goal did not apply to the programs and there
was no information provided on which to evaluate performance. Several
reports give no comment at all.

For those reports which gave ratings of "not successful" the comments of
experts indicate the reasons for lack of success are lack of relevance
of this goal and that few of the awards are intended to focus on
educational development; hence they do not contribute to the achievement
of this goal. Those ratings are not automatically used in tabulating
results overall. In one report covering several programs, the experts
indicate that the programs were *minimally effective* in achieving the
goal, and yet are able to cite examples of success relevant to achieving
the goal. One report indicates that although they were aware of
activities aimed to address this goal which could be evaluated, they
could not locate data, and recommended that the staff summarize such
efforts in the future.

In effect, many of the programs evaluated did not provide clear evidence
of support for the objectives of this goal, external evaluators had
difficulty in providing a qualitative assessment, and success across the
agency is not apparent based on COV and AC reports. This goal is
difficult to evaluate as it is written, in part because the specific
activities referenced by the indicators are not widespread across all
programs. NSF is reviewing the components of this goal for FY 2001 and
FY 2002, to develop appropriate indicators more directly within the
agency's control.

## Comparison FY 1999 -- FY 2000 

This goal was continued from FY 1999, and includes two quantitative
subgoals achieved this year and also in FY 1999. In FY 1999, this
Outcome Goal was stated using two levels of achievement: *successful*
and *minimally effective*, with indicators for each level. Based on
comments from COVs and ACs in FY 1999, it was determined that the
definitions for the *minimally effective* level of performance did not
provide additional information in evaluating the programs.

In FY 2000, a single definition for the *successful* standard was used
as the target level of performance, and a stricter definition of allowed
success was applied, which required clear justification of ratings in
reports. The overall result in FY 2000 is similar to that obtained in FY
1999, even though the evaluation was carried out on a different subset
of NSF's portfolio by a different group of external experts. In
following stricter guidelines for definitions of success in FY 2000, we
are reporting this goal as "not achieved", as opposed to "achieved" as
we did in FY 1999.

## Steps to meet this goal in the future

Although NSF has a significant focused effort in mathematics and science
education, NSF provides very little of the overall investment in K-12
education. Meeting the performance goal implies a commitment on the part
of school districts, schools, and their faculty to modifying their
approaches to education in order to enhance achievement; it is also very
dependent upon the availability of resources to do so.

Results obtained in FY 1999 and FY 2000 have led NSF to refine this goal
and to identify ways to improve data/information collection to assess
progress by tracking contributions in achieving this goal more
effectively.

The goal and indicators will be modified to clarify applicability of
this goal to programs being evaluated in FY 2001. The reporting template
used by external evaluators to assess programs will be improved to
gather better information on achievement of programs for which this goal
is relevant in order to gain a better understanding of performance. COVs
have recommended that some NSF programs develop plans to address this
goal more fully in future years and some action has been taken.

## FY 2001 and beyond

This goal will be incorporated under a new Strategic Outcome Goal
heading for FY 2001 which rearranges NSF's five Outcome Goals into three
broad Strategic Outcome areas: People, Ideas, and Tools. A table
depicting the structural rearrangement is shown in Section VIII of this
report, "*Transition to FY 2001 and Beyond.*" The quantitative subgoals
will be maintained in FY 2001 as subgoals of the People Strategic
Outcome Goal.

The change to People, Ideas and Tools improves alignment of NSF's goals
with its mission and allows closer agreement between budget categories
and NSF's Strategic Plan. This Outcome Goal will be restated to avoid
mixing goal objectives, under the People Strategic Outcome area in FY
2001. It will also be contained under a new category in FY 2001,
described as "Broadening Participation". This is included in the
description of NSF's Investment Process Goals contained in the FY 2001
Performance Plan.

# 

# FY 2000 Examples[^6] of Achievements Cited by External Evaluators

## Outcome Goal 4

### Improved Achievement in Mathematics and Science Skills

### Needed by All Americans

External evaluators cited the following examples of results from NSF
awards as demonstrating the criteria for success for Outcome Goal 4.
Noteworthy examples taken from committee reports have also been selected
to demonstrate results in FY 2000 areas of emphasis, which include K-12
systemic activities; research on learning and education; graduate
teaching fellows in K-12 education; and K-16 digital libraries.

These examples are also shown to illustrate how NSF awards have led to
the development, adoption, adaptation, and implementation of effective
models, products, and practices that address the needs of all students;
well-trained teachers who implement standards-based approaches in their
classrooms; and improved student performance in participating schools
and districts. The diverse portfolios of awards show potential for
significant impact in many of these areas. NSF considers many of the
K-12/16 activities listed to be of interest to students to engage them
at an early state in their education in science, mathematics and
computer science. Early involvement is extremely important for retaining
students in science and engineering.

-   High Quality Instructional Materials for both teachers and students
    are benefiting from discoveries related to teacher and student
    learning. Professional development for teachers is now viewed as a
    continuing process that is tailored to the needs of the adult
    learner. The work of NSF-supported projects have shown that site
    administrators and parents must also be part of the professional
    development process.

```{=html}
<!-- -->
```
-   The *Hands-on Universe* project empowers teachers to use
    research-quality astronomical tools (remote telescopes, and
    software) in their classrooms with students. Last year, students in
    Massachusetts discovered a new asteroid in the Kuiper Belt. The
    announcement of their discovery and its confirmation made news
    worldwide. Two years ago, a different group of students using these
    tools discovered a supernova.

-   *Pattern Exploration* seeks to integrate mathematics and science
    using the new ideas of fractal geometry. Materials used in this
    teacher enhancement project were derived from two previous
    NSF-funded projects and help teachers deepen their understanding as
    well as their ability to use hands-on materials and software with
    their students to make patterns in nature visible.

```{=html}
<!-- -->
```
-   Results may come from large-scale national centers or close-to-home,
    small-scale experiments that enable teaching and learning of
    scientific and technological ideas. A few examples of results
    derived from advanced technological education projects include:

```{=html}
<!-- -->
```
-   The use of computer animation to visualize magnetic and other fields
    of force, to assist student understanding of complex physical
    concepts;

-   Centers that create and serve as depositories and disseminating
    agents for best techniques in technician education and industry
    practice, and have engaged in ground-breaking biological,
    telecommunications, semiconductor and marine discoveries; and

-   The adaptation, by an undergraduate program, of a sophisticated
    university field-based course on watershed management for teacher
    certification in environmental studies.

> Systemic reform projects have leveraged the products and expertise
> developed by NSF awardees.

-   Over the first six years of the Chicago Urban Systemic Initiative
    (USI) the percentage of fourth grade students meeting Illinois State
    Standards in science increased from 46 to 66.

-   For the San Antonio USI, the average scores of African-Americans in
    grade 4 on the Texas Assessment of Academic Skills increased by 32
    percentage points over four years, and those of Hispanic students by
    39 percentage points, compared to a 16 percentage point increase for
    Texas fourth-graders overall.

-   In the New York City USI, students in grades 3-8 scoring at or above
    grade level in mathematics on the California Achievement Test
    improved from 49% to 63% over a five-year period.

-   Noticeable gains on the Texas Assessment of Academic Skills (TAAS)
    were evidenced for students in classrooms of K-8 teachers who
    received one or more years of professional development through the
    Austin Collaborative for Mathematics Education. The most dramatic
    gains were made by African American, Hispanic and economically
    disadvantaged students, reducing the performance gap with majority
    students.

-   Recent findings from research studies indicate that NSF-supported
    efforts are decreasing disparities in student achievement across
    socioeconomic levels and identifiable populations. An evaluation
    conducted by the Wisconsin Center for Educational Research, showed
    evidence in a preliminary analysis of National Assessment of
    Educational Progress (NAEP) data that grade 8 mathematics
    achievement by African American students in Statewide Systemic
    Initiatives (SSI) states improved and exceeded the achievement in
    non-SSI states from 1990 to 1996.

> **Research on Learning and Education** was given high priority in the
> report of the President\'s Committee of Advisors in Science and
> Technology on the Use of Technology to Strengthen K- 12 Education in
> the United States (March 1997). NSF, in partnership with the
> Department of Education, has built on past investments in this area in
> FY 1999 and continued joint activities in FY 2000. The NSF portfolio
> of awards has led to the development of an extensive array of tools,
> models, products and practices that address the needs of all students.

-   NSF awards have created tools and resources to increase the
    assessment of science and mathematics learning, provide evidence on
    the quality of professional development, and enhance the capacity of
    professional developers. For example, *TECH-STAT*: Teaching
    Statistics Grades 1-6, a statewide implementation project in North
    Carolina, has developed both professional development manuals for
    teachers and statistics modules for students. Professional
    development materials are designed around the use of performance
    assessments to inform and strengthen classroom instruction.

-   Informal science education programs--through variety of media--reach
    over 150 million viewers yearly. For example*, The World We Create*,
    an exhibit at the Louisville Science Center, features 40 hands-on
    science activities and over 400 graphic panels highlighting science
    careers, inventors, and problem solving strategies. From 1997-2000,
    the exhibit and associated programs reached almost 1.5 million
    visitors, nearly one-third the population of the rural state of
    Kentucky.

-   Projects for developing professional materials produce printed
    materials as a major item but now include materials that require use
    of video as well as regular and on-line computer technologies (e.g.,
    CD-ROMs, listserve, other software). Some examples are:

```{=html}
<!-- -->
```
-   *Telemonitoring*--An Online Model to Sustain Professional
    Development in Science, Math, and Technology for Grades K-12.

-   *Developing Mathematical Ideas, and Problem Solving in the
    Sciences*--An Innovative Software Approach (IMMEX), is introducing
    secondary teachers to techniques and analyses using software
    developed for medical schools to teach problem-solving and monitor
    student and class mastery of concepts.

-   *Science K-6--Investigating Classrooms* has developed a library of
    videotapes and supporting print materials to illustrate the
    effective application of the National Science Education Standards in
    K-6 classrooms.

-   Teaching modules distributed by the American Chemical Society to
    secondary schools. They range from teaching the chemistry used in
    the carbonated beverage industry to treating waste-water. The
    modules have been field tested in 21 states by 58 teachers with 2200
    students.

## Timeliness

In a recent survey, a sample of the science and engineering policy
community indicated that improving timeliness of data was a high
priority for them. Data collected either refer to a specific date, such
as salary as of April 15 or fall enrollment as of October 15, or to a
period of time, such as a calendar or fiscal year. The reference date in
the latter case is calculated as the last day in the period. The time
between the reference date and the first public release of data from
each of eleven major surveys is calculated, and then an average is taken
across all surveys over a two-year period. Data are maintained by the
Science Resource Studies (SRS) Division.

**Means for achieving success**: Taking advantage of advances in
information and communications technologies; and regular reporting of
status to give ample time to take action to improve performance.

+-----------------------------------------------------------------------+
| ## Performance Indicators                                             |
+-----------------------------------------------------------------------+
| Average time interval between the reference period and reporting data |
| from SRS surveys.                                                     |
+-----------------------------------------------------------------------+

  -------------- ------------------------- -------------------------------
                 FY 1995-96                FY 1999-2000

  **Baseline**   540 days                  

  **Goal**                                 486 days

  **Actual**                               461 days
  -------------- ------------------------- -------------------------------

## Data Quality

The value of information on the science and engineering enterprise is
highly dependent on its ability to address issues of importance to those
who seek to use it in making policy decisions. Measures of data quality
help users determine the reliability of the information and the extent
of likely variance introduced by sampling processes. This goal replaced
a related FY 1999 performance goal which dealt with customer measures of
relevance. Data quality is one factor in addressing relevance.

**Means for achieving success:** NSF staff developed a standard set of
data quality measures that are now in place. Procedures were established
to ensure that appropriate information is provided electronically for
all surveys.

+-----------------------------------------------------------------------+
| ## Performance Indicators                                             |
+-----------------------------------------------------------------------+
| Data quality measures and their use in SRS products.                  |
+-----------------------------------------------------------------------+

### Baseline 

This is a new effort to provide standard measures. Their absence has
placed limits on the usefulness of surveys.

Data quality measures were developed by SRS after conducting a thorough
review of the written data quality standards for surveys conducted by
other statistical agencies such as the National Center for Education
Statistics, the Energy Information Administration, and the National
Center for Health Statistics. A general literature review was also
conducted, especially of material developed by the Office of Management
and Budget's Federal Committee on Statistical Methodology (OMB/FCSM).
Based on this research and analysis, a relevant set of measures was
chosen as the standard set of quality measures for SRS surveys.

A standard format for reporting the data quality measures was developed.
For each on-going SRS survey, the information on data quality measures,
critical for the user to know for proper use of the survey data, was
organized into the standard reporting format. These data quality reports
were placed on the SRS web site and linked to the other information
available for each SRS survey
(http://www.nsf.gov/sbe/srs/ssdr/start.htm).

## FY 2001 and beyond

This goal will not be continued in this form in FY 2000. The goal has
been redefined for FY 2001 to reflect the requirements established under
the NSF Act of 1950. For FY 2001, NSF's five Outcome Goals are
rearranged into three broad Strategic Outcome areas: People, Ideas, and
Tools. A table indicating the change is shown in Section VIII,
"*Transition to FY 2001 and Beyond."* The rearrangement into the three
areas improves alignment of NSF's Outcome Goals with its mission and
allows closer agreement between budget categories and NSF's Strategic
Plan. This topic will be addressed by a new area described as the Tools
Strategic Outcome area in FY 2001.

## Management Goals and Results 

### 

## Focus on Management

NSF's Management Goals address the Foundation's administrative,
operational and policy objectives. Excellence in managing the agency's
activities is the key to achieving successful performance for all of
NSF's goals. NSF's six Management Goals for FY 2000 address three issues
of high priority in the Foundation -- staff training, staff diversity,
and how well advanced technology is being incorporated into NSF business
operations. Five are continuations of goals previously established, with
more stringent performance indicators. The new Management Goal included
this year is reflective of our desire to more fully integrate technology
into the core activities of the Foundation. Four factors are especially
critical to successful management at NSF:

These critical factors are used in developing annual performance goals
in the following performance areas: electronic proposal submission and
processing; staff diversity; technological capability of staff through
training; Y2K compliance; and use of electronic systems for project
reporting. Results for the Management Goals, most of which have
quantitative measures, are prepared and reviewed by NSF staff. They are
presented below by area of performance.

  -------------- ----------- ----------- ----------- ----------- -----------
                 **FY 1997** **FY 1998** **FY 1999** **FY 2000** **FY 2001**

  **Baseline**   4.4%        17%                                 

  **Goal**                               25%         60%         95%

  **Actual**                             44%         81%         
  -------------- ----------- ----------- ----------- ----------- -----------

FastLane is a collection of electronic system modules that allows all
transactions and communications between NSF and its grantees to be
facilitated via the Internet. Under development since 1994, FastLane
plays a major role in NSF's goal of achieving a paperless environment by
the end of FY 2001. This ambitious goal was continued from FY 1999, and
based on real-time results was revised in FY 1999 and FY 2000. The goal
will be continued in FY 2001, with the target level of performance
increased based on expectations and actual performance in FY 1999 and FY
2000.

In FY 2000, a total of 25,160 proposals were received and processed
through FastLane. This is 81% of the full proposal submissions, which
totaled 30,932. The success of this goal can be attributed to an
aggressive outreach strategy combined with the efforts of a *Helpdesk*,
a staffing resource designed to provide external customers with
assistance. More than 35,000 requests for assistance were received by
the *Helpdesk*, of which approximately 90% were related to proposal
preparation and submission.

In September 2000, the NSF Director issued Important Notice 126 to the
presidents of universities and colleges and the heads of other NSF
grantee institutions to reaffirm that effective October 1, 2000,
specified transactions with NSF must be accomplished electronically

via use of the FastLane system. The Important Notice is posted on
http://www.nsf.gov/pubs/2000/iin126/iin126.htm.

## Implications for FY 2001 

FastLane continues to be rapidly accepted among our external customers
for proposal submission. A significant number of program initiatives
required the submission of proposals in FastLane in FY 2000. Virtually
all programs will require FastLane submissions in FY 2001.

For FY 2001, the goal is being raised to 95% of full proposal
submission. This equates to full implementation, and is consistent with
the requirement specified in Important Notice 126 (see above). This
percentage recognizes that some universities, colleges, or persons with
disabilities, may experience difficulties in transmission, and others
may not have the technical capability to submit electronically to NSF.

In order to enhance operational efficiency, NSF instituted requirements
for electronic submission of grant proposals. Upon receipt, proposals
are distributed to the appropriate office for administrative processing
and peer review. Recommendations are prepared by NSF staff, funding
decisions are made and award/declination letters are prepared for the
approximately 30,000 proposals submitted annually. Historically, NSF
required paper submission once grant proposals were submitted
electronically. Efforts to modernize this process have been underway for
several years. The goal is to move to electronic processing for the
entire internal review and award/decline process.

At the start of the year, only four functions within the peer review
process were still paper-based, namely: communications between NSF and
the peer reviewer; electronic panel review system; letters to principal
investigators (PIs) with declined proposals; and release of review
results to PIs. By the end of the year, the technological barriers to a
completely paperless process were removed within NSF, except for one
remaining issue, i.e., the electronic equivalent of a signature for
funding approval by NSF.

## Implications for FY 2001

Two electronic signature pilot projects were initiated during the FY
2000. The results are being evaluated in FY 2001 to determine which
approach will best serve the agency and its customers. Technological,
financial, and legal issues still need to be resolved before electronic
signatures can be fully adopted. NSF will continue to address these
issues in FY 2001. In addition, we will make use of the technological
capabilities established in FY 2000 to initiate pilot projects that
demonstrate the paperless review capability. The FY 2001 goal for NSF is
to conduct 10 pilot paperless projects that manage the review process in
an electronic environment.

### 

### Baseline:

Of S&E hires in 1997, 16 were female and 15 were from under-represented
minority groups.

In order to ensure that the United States maintains its world leadership
role in science and technology, the Nation must maintain a first-class
cadre of scientists, mathematicians, and engineers from all segments of
society. NSF is committed to diversifying its staff of scientists and
engineers both in permanent positions and in the important rotating
scientist positions.

During FY 2000, NSF engaged in a number of activities to increase the
numbers of minorities in the S&E staff. These activities included:

-   Requiring a diversity recruitment plan from each directorate and
    requesting a year-end report on their activities;

-   Advertising specific vacancies in minority-serving magazines,
    institutions and professional associations;

-   Attending job fairs that attract minority and female participants;
    and

-   Requiring written justifications from selecting officials regarding
    their outreach activities and selection process.

Additionally, hiring information is displayed on the NSF GPRA homepage
to assist managers in addressing under-representation. This information
includes demographics of the current S&E workforce, statistics on the
availability of minorities and women in the S&E labor pool, and the
numbers of hires from under-represented groups.

## Implications for FY 2001

NSF will maintain this goal in FY 2001. In addition to increasing
emphasis by the Director's office, NSF will increase its recruitment
presence at major program workshops and seminars, target recruitment
material towards under-represented groups, and create a registry for
minorities interested in serving on NSF advisory committees and panels.
These committees and panels serve as a major resource for recruiting
visiting scientists and engineers for the Foundation. NSF management
will continue to emphasize diversity hiring practices, diversity pool
statistics will be stressed at management sessions, and merit promotions
will be reviewed at the senior executive levels.

  ----------------------- ----------------------- -----------------------
  Orientation             FY 1999                 FY 2000

  **Goal**                100%                    100%

  **Actual**              80%                     100%
  ----------------------- ----------------------- -----------------------

  ----------------------- ----------------------- -----------------------
  Training                FY 1999                 FY 2000

  **Goal**                95%                     80%

  **Actual**              43%                     90%
  ----------------------- ----------------------- -----------------------

By the end of FY 2000, 100% of NSF staff had received an orientation to
FastLane and 90% of program and program support staff had received
practice in using its key modules**.**

As the use of FastLane continues to grow, it is critical that all staff
are oriented to FastLane and other electronic systems. Through a series
of ongoing formal classes, extensive individual and group training,
distribution of informational materials, and the persistent efforts of
NSF staff, NSF achieved this goal this year.

By the end of FY 2000, all 1,239 staff members (100%) on-board as of
July 1, 2000 received an orientation to FastLane. For program and
program support staff, 698 of 777 (90%) received practice in using its
key modules.

The formal FastLane training program, initiated in FY 1998, continued
through FY 2000. Approximately 40 FastLane classes were conducted during
the year, with announcements posted on the training bulletin board and
on the internal electronic Announce channel. Based on user feedback, we
are moving towards new electronic business classes. These are scheduled
to begin in January 2001.

Training on request was also provided to organizational units. Users
were allowed to take training at their workstations through on-line
training services, and informational material on FastLane was developed
and distributed to employees.

Throughout the year, training statistics were posted on the GPRA web
page to help managers monitor their progress. In addition, the NSF
Training System was modified to allow for the entry of short, no-cost
training as a way of capturing some of the required training data. Data
was provided to the directorates to ensure that the information in the
system was accurate and to encourage divisions to schedule employees for
training.

## Implications for FY 2001

Because NSF relies on visiting scientist and engineer positions to
maintain it's portfolio, staff turnover will remain high. Hence,
FastLane orientation will continue to be an on-going process. Moreover,
as existing modules are enhanced or new modules added, the curricula
will be modified to ensure that staff stay current in the use of
FastLane and other electronic systems. Additionally, we will continue
our outreach efforts to increase the proficiency of PI's and grant
administrators in using FastLane. Since existing staff have been fully
trained and procedures have been put in place to ensure that new staff
receive orientation and training, FastLane training will no longer be
reported as a goal.

All activities needed to address the Year 2000 problem were completed
according to plan, on schedule and within budget. Due to inspection and
modification of pre-existing information systems, NSF entered the year
2000 trouble free in regard to the operation of computer and other
critical systems. This activity will no longer be reported as a goal.

  ----------------------- ----------------------- -----------------------
  Training                FY 1999                 FY 2000

  Baseline                59%                     

  **Goal**                70%                     85%

  **Actual**              59%                     92%
  ----------------------- ----------------------- -----------------------

The Project Reporting System (PRS) is part of NSF's effort to use
advanced technology to create a more efficient, paperless work
environment, in which information is exchanged between the Foundation
and its research and education customer community via the Internet. In
its first two years of use, the PRS has provided a wealth of information
that was previously not available electronically. This has lead to
significant changes in how NSF responds to internal as well as external
requests for information on the technical aspects of NSF awards.

An internal search utility allows NSF staff to search the reports based
on a variety of criteria and isolate the award and/or report of
interest. This is leading to profound changes in how NSF can respond to
requests from Committee of Visitors, internal management, and the public
on technical aspects of NSF awards.

During FY 2000, 8,949 final project reports were received, of which
8,269 (92.4%), were submitted through the PRS. The remaining 680 final
project reports were submitted via paper or email.

In addition to final project reports, annual reports are submitted for
those grants that are active. During FY 2000, 9,987 annual reports were
submitted via FastLane. Information on annual project reports submitted
via paper is not maintained in NSF's electronic systems, so data on
annual reports is not included in this Management Goal. However, since
annual and final project reports usually contain the same information
and are submitted by the same Principal Investigators (PIs), we expect
that the percentage of annual reports submitted through the PRS is
comparable to the percentage of final reports.

Two NSF documents that provide guidance to applicants and institutions
were revised to reference the new PRS: the NSF Grant Proposal Guide and
the NSF Grant General Conditions. Both documents now reference the fact
that PIs are required to submit reports electronically via the PRS in
FastLane. Based on feedback received throughout the year, modifications
to the PRS have been made. NSF will continue to enhance the system based
on user feedback and policy changes, as resources allow.

In September 2000, the NSF Director issued Important Notice 126 to the
presidents of universities and colleges and the heads of other NSF
grantee institutions describing NSF's requirements for a paperless
proposal and reporting system. The important notice is posted on
http://www.nsf.gov/pubs/2000/iin126/iin126.htm.

## Implications for FY 2001

During FY 2000, NSF received 92% of final project reports through the
PRS. Recognizing that minor exceptions are allowed for older awards,
this represents nearly full implementation. Since the PRS has been
successfully implemented and is now fully utilized, project reporting
will not be continued as a goal in the future. However, NSF will
continue to emphasize the importance of using the PRS with our external
community.

# C. Investment Process Goals and Results 

## Focus on Investment Process

NSF's key strategy for success is the use of external merit review to
make awards for activities that will impact research and education in
mathematics, science, and engineering, both directly and indirectly. The
heart of the investment process is competitive merit review by external
peers, using two criteria established by the National Science Board. The
scientists and engineers comprising NSF's program staff take NSF
priorities and the advice of external reviewers into account in
developing their portfolio of awards. Critical to the success of the
investment process are the means and strategies for high quality
proposal and award processes that support achievement of the Outcome
Goals and meet customer expectations.

## 

## 

## Investment Process Goals

The Investment Process Goals address various aspects of NSF's awards
process, such as the use of merit review and the need to keep the awards
system open to new people and new ideas. These goals help to establish
customer service standards for the agency. Examples include use of merit
review and improved practices such as the time it takes to process a
proposal. In addition, the facilities oversight performance goals
relevant to the federal science, space and technology agencies, are
included in NSF's set of Investment Process Goals. Results for the
Investment Process Goals, most of which have quantitative measures, are
prepared and reviewed by NSF staff. Investment Process Goal 2 is a
qualitative goal expressed in the alternative form and evaluated by
external experts (COVs and ACs). Results are presented and discussed
according to performance areas: Proposal and Award Processes, Customer
Service, Maintaining Openness in the System, Integration of Research and
Education, Diversity, and Facilities Oversight.

Based on NSF's original goal, which included merit reviewed projects as
a percentage of all NSF funding, the Foundation exceeded its goal of 90%
for FY 2000. As in FY 1999, NSF allocated 95% of its funds to
merit-reviewed projects. This goal was achieved in FY 1999 and
maintained in FY 2000. It will be revised based on OMB revised
definitions for FY 2001.

  ---------------- ---------- ---------- ---------- ---------- ----------
  *Percent of      **FY       **FY       **FY       **FY       **FY
  project funding  1997**     1998**     1999**     2000**     2001**
  subject to merit                                             
  review*                                                      

  **Baseline**     89%        90%                              

  **Goal\***                             90%        90%        85%\*\*

  **Actual\***                           95%        95%        
  ---------------- ---------- ---------- ---------- ---------- ----------

**\*N.B.** Based on old OMB definitions. During FY 2000, the Office of
Management and Budget revised the federal goal, stating that 70-90% of
research and development funds should be awarded to merit reviewed
projects. Under the new definition, federally-funded research and
development centers (FFRDCs) and merit-reviewed scientific research with
competitive selection and internal (program) evaluation will not be
considered merit-reviewed. Taking into account the new definition, NSF
has revised its goal for FY 2001 to 85%.

### Baseline: 

New criteria went into effect in early FY 1999. External expert judgment
is used to assess performance. The assessment process was used for the
first time during FY 1999.

Full performance in achieving this goal requires that both merit review
criteria be addressed by both reviewers and program staff. The results
indicate that NSF was not fully successful as judged by external
evaluators.

For FY 2000 COVs reviewed 78 NSF programs and were asked to judge
whether the programs were successful or not in meeting this performance
goal. A total of 58 out of 64 reports rated programs on their use of
both merit review criteria. NSF was judged successful in achieving this
goal in 20 of the 58 reports.

In most cases where NSF was not successful, reviewers did not fully
address the second merit review criterion regarding the broader impacts
of the proposed activity in their reviews or applicants did not address
broader impacts in their proposals. Most COV assessments noted that NSF
staff addressed both criteria in their decisions.

It is important to note that the two merit review criteria were not
implemented until FY 1998, and the time period covered by COVs
conducting program assessments in FY 2000 included proposals that had
been reviewed before the two criteria were implemented (i.e., proposals
from FY 1997). Since both criteria were not fully implemented during
this time period, full use by reviewers and staff should not be expected
for this assessment. The FY 2001 assessment will include proposals
reviewed in FY 1998 and beyond, which will be the first assessment to
review the full implementation of the two criteria. Full usage should
become more apparent in the FY 2001 and FY 2002 assessments.

## [Comparison]{.smallcaps}: FY 1999 -- FY 2000 

In FY 1999, this goal was stated using two levels of achievement:
*successful* and *minimally effective*, with indicators for each level.
In FY 1999, a majority of reports rated programs as successful on their
use of the merit review criteria. In most cases where programs were not
fully successful it was indicated that reviewers and proposers were not
fully addressing both review criteria. Based on comments from evaluators
in FY 1999, it was determined that the descriptors for the *minimally
effective* level of performance did not provide additional information
in evaluating the programs.

For FY 2000 a single descriptor for the *successful* standard was used
as the target level of performance. In FY 2000 a stricter definition of
allowed success was used in aggregating these results. This required
clear justification of ratings in COV and AC reports. If reports gave
successful ratings but did not mention use of both criteria by both
reviewers and program managers, the goal was judged to be less than
fully successful. It is possible that programs are more successful in
achieving this goal than these results indicate. However, most reports
indicate NSF programs can still improve on use of both criteria.

The issued identified by COVs in FY 2000 are similar to those observed
in FY 1999, even though the evaluation was carried out on a different
subset of NSF's portfolio by a different group of external experts.
Comments from reports indicate that progress is being made.
Nevertheless, improvement is still needed.

## Steps to improve performance results for this goal for FY 2001 and beyond

NSF took steps in FY 2000 to educate reviewers and proposers on the use
of the merit review criteria. NSF clarified the meaning of the criteria
and stressed the importance of using them. Improving results for this
goal depends upon improving information in proposals submitted by
proposers and on motivating reviewers to provide substantive comments on
both criteria. It also depends on the use of both criteria by NSF staff
when making decisions. NSF can encourage proposers and reviewers to
address both criteria, but has limited control over their response. Many
proposals do not contain sufficient information necessary for reviewers
to evaluate the broader impact criterion. To improve this situation, NSF
has modified program announcements to encourage proposers to provide
information on all relevant aspects of the merit review criteria in
their proposals. NSF has recently re-issued guidance to the proposers
and reviewers, stressing the importance of using both criteria in the
preparation and evaluation of proposals submitted to NSF.

To assist reviewers and staff in FY 2001, separate on-screen pages are
available in FastLane - NSF's electronic data system. These provide the
capability for reviewers to address each merit-review criterion
separately. In FY 2001, performance data will be collected from the
FastLane database.

Full implementation of this goal is a priority for NSF in FY 2001 and
beyond. To do so requires information to be included in proposals,
addressed by reviewers, and taken into account by program staff. NSF has
taken steps to ensure that incoming proposals contain adequate
information for reviewers to evaluate. NSF is taking steps to further
implement this goal by developing a system to determine the extent of
program officer use of both criteria in decision making. This process
will be made quantitative upon determination of an appropriate mechanism
and baseline.

In response to a directive by the Senate Appropriations Committee that
NSF review the procedure and criteria for merit review once the new
criteria had been in place for a year, in FY 2000, NSF issued a contract
to the National Academy of Public Administration (NAPA). This contract
was designed to conduct a study of the impact of the new merit review
criteria on the nature of the projects NSF supports. In conducting the
study, NAPA interviewed key personnel and stakeholders from the S&E
community and analyzed a sample of COV reports and proposal documents.
The key finding was that it is too soon to make valid judgements about
the impact and effectiveness of the new criteria. The NAPA report also
highlighted the need to (1) improve the conceptual clarity of the
criteria, (2) better communicate with proposers, reviewers and NSF staff
about how the criteria are to be used, and (3) improve quantitative
measures and performance indicators to track the objectives and
implementation of the new criteria. NSF will act upon these suggestions
beginning in FY 2001.

This goal will be maintained and emphasized in FY 2001. It will appear
as two goals, one addressing use of the criteria by reviewers, and a
second addressing use of the criteria by NSF staff. Improvements to the
COV and AC process and guidelines for evaluating this goal are being
implemented in FY 2001.

In FY 2000, NSF commissioned additional surveys including the ACSI
survey of awardees and informal surveys and focus groups at NSF regional
grants seminars. These were designed to identify the reasons for
Principal Investigator dissatisfaction with the timeliness and
efficiency of the proposal process, the quality and fairness of the
merit review process, and the handling of customer complaints.

The 2000 ACSI survey indicated that NSF improved slightly in two key
areas:

1.  timeliness and efficiency of the proposal process; and

2.  quality and fairness of merit review.

These were the two areas of greatest concern identified in the FY 1999
survey. NSF will continue to address customer service as noted in
Investment Process Goals 3, 6, and 7.

NSF conducted customer service surveys and solicited other forms of
feedback in an effort to pinpoint specific customer issues and to
identify effective practices for handling customer complaints within
NSF. Further, other federal agencies were examined to locate a model
with similar customer interactions, but no appropriate model was
identified. Models of best practices and NSF staff training are still
being developed in FY 2001. NSF continues to place great importance on
these issues and will complete this effort in FY 2001. In addition, NSF
will pilot the best of the models in NSF divisions and provide specific
customer service training to NSF staff.

### Baseline:

57 on a scale of 0-100 in FY 1999

In addition, NSF coordinated a Customer Service Focus Group meeting in
March 2000 as part of an NSF regional grants conference held at
Louisiana State University. This Focus Group was a follow-up activity to
an informal email survey of seminar participants conducted prior to the
seminar. The participants were 32 Principal Investigators and research
administrators. The primary topics addressed by the survey were NSF's
handling of complaints and the timeliness and efficiency of the NSF
proposal process. These informal surveys were continued at the Purdue
University seminar in October 2000 to compare previous data and to
gather additional information concerning customer service.

NSF arranged for another ACSI survey in FY 2000, involving only
grantees, to ascertain possible reasons for customer dissatisfaction
with the merit review system and with NSF's complaint system. This
awardee survey was performed to confirm the results of the ACSI survey
(see Investment Process Goal 5) and to get more detailed information on
specific issues related to merit review and customer interaction. The
University of Michigan conducted the supplementary survey of NSF
awardees in November 2000 using a set of questions developed by the
Foundation.

NSF is striving to improve the time to decision (see Investment Process
Goal 7). Applicants who stated that they had a specific problem or
concern with the quality or fairness of merit review identified two
primary concerns: reviews were inappropriate (i.e., reviews did not seem
to adequately address the proposed project, in the opinion of the
proposer) and reviews were uneven (i.e., the range of review scores
included both high and low scores).

Finally, survey participants in FY 2000 who stated that they had
complained to NSF described the nature of their complaints primarily in
three ways: 1) concern about overall quality or fairness of proposal
merit review process; 2) problem submitting a proposal, review, or
project via FastLane; and 3) problem making timely contact with
appropriate person at NSF.

  ---------------------- ------------ ------------ ------------ ------------
                         **FY 1998**  **FY 1999**  **FY 2000**  **FY 2001**

  **Baseline**           66%                                    

  **Goal**                            95%          95%          95%

  **Actual**                          75%          89%          
  ---------------------- ------------ ------------ ------------ ------------

In FY 2000 89% of program announcements and solicitations were made
available at least three months prior to their deadline/target date.
Approximately 97% of program announcements and solicitations were
available within 5 days of the three-month goal. This is a significant
improvement over FY 1999, when 75% of announcements met the 3-month
standard. The following bar-chart visually demonstrates the number of
program announcements that gave applicants 90 days or more to prepare
proposals (goal achieved) compared with those that missed the goal by a
few days. Ninty-five percent of announcements were posted within 5 days
of the three month goal.

![](media/image3.wmf)The most common reason cited for not achieving this
goal was delay in posting announcements on the NSF web site. In FY 2000
a web-based system for creating program announcements was established.
This system has decreased the time required for an announcement to be
posted on the NSF web site. This should aid the agency in achieving this
goal. However, this was the first year of implementation, and not all
announcements were prepared using the new system. The Foundation intends
to review and revise the timing of clearance procedures, in order to
ensure that web posting of announcements will occur in a timely manner.
NSF is also working to enhance the tracking system that measures the
time available to applicants to prepare proposals in an effort to
improve the accuracy of the data.

The Foundation staff work toward this goal by limiting the number of
special competitions requiring individual program announcements and
solicitations, planning for such competitions as far in advance as
possible, and initiating clearance processes at least six months prior
to the anticipated proposal deadline. NSF expects increased use of the
new systems in FY 2001, and expects to see additional progress toward
meeting this goal next year.

  ---------------------- ------------ ------------ ------------ ------------
                         **FY 1998**  **FY 1999**  **FY 2000**  **FY 2001**

  **Baseline**           59%                                    

  **Goal**                            70%          70%          70%

  **Actual**                          58%          54%          
  ---------------------- ------------ ------------ ------------ ------------

![](media/image4.wmf){width="3.173611111111111in"
height="2.834722222222222in"}In FY 2000 more than half (54%) of all
proposals were processed within six months of receipt, while an
additional 35% of proposals were processed between six and nine months
of receipt. In FY 1999, 58% of proposals were processed within six
months of receipt, somewhat better than the 52% average rate over the
last five years, but nevertheless short of the 70% goal. Data show that
about 71% of proposals were fully processed in less than seven months,
and about 82% of proposals were processed in less than 8 months, as
shown.

One of the most significant issues raised by applicants (see results of
the ACSI customer survey, described under Investment Process Goal 3) is
the amount of time it takes for NSF to process proposals. NSF is
reviewing the steps needed to decrease the processing time of proposals
to find ways to process them more quickly.

One factor leading to delay in processing is that some programs at NSF
prefer to conduct merit

review by mail rather than by panel. Mail reviews often take longer to
complete. Another factor is that some programs tend to hold a few highly
rated proposals until the end of the fiscal year, or even into the next
fiscal year, in anticipation that more funds might become available. In
FY 2000 a few programs reported temporary staffing shortages. This
slowed down their review process. This situation has been corrected.

In addition, the processing of international awards often takes more
time than standard awards. This is because the process of making
international awards necessarily involves additional major steps with
more program units involved, increasing the amount of time required for
processing. For example, in many cases, foreign country approval of a
matching proposal must be obtained, which often results in unpredictable
delays.

In FY 2001 NSF staff will work towards shortening the award processing
time by making more effective use of electronic mechanisms in conducting
the review, working cooperatively to reduce overloads and bottlenecks,
and by carefully tracking the stage of processing and received date of
all proposals. In addition, some internal organizations are
reconsidering the practice of holding over proposals for potential
funding until the next fiscal year. Some have added "performance on
prompt handling of proposals" to the performance evaluation criteria of
their staff.

  ---------------------- ------------ ------------ ------------ ------------
                         **FY 1998**  **FY 1999**  **FY 2000**  **FY 2001**

  **Baseline**           27%                                    

  **Goal**                            30%          30%          30%

  **Actual**                          27%          28%          
  ---------------------- ------------ ------------ ------------ ------------

The percentage of competitive research grants issued to new
investigators was 28% in FY 2000, one percent higher than in FY 1999.
This is a challenging goal for NSF. There continues to be a wide
disparity in the funding rates of "new" Principal Investigators (PIs)
and "prior" PIs -- 24 percent and 40 percent, respectively in FY 2000.

It is important to note that this goal counts "grants" to new
investigators. It does not count *all* new investigators who may be
collaborating on a project -- it counts only new *PIs* - not new
*co-PIs* -- which would be the case if two or more new applicants
collaborating together received an award. Also, the goal does not count
new co-PIs on awards where the PI has had prior NSF support, as is often
the case. If we count both PIs and co-PIs who are new, we find that more
than 32% received support in FY 1999 and more than 33% received support
in FY 2000. The following bar-chart compares the percentage of all
research awards where both new PI and co-PI's are counted (first column)
to the percentage of all research awards where only new PI's are counted
(second column), for fiscal years 1996, 1997, 1999, and 2000. This
result indicates that many new investigators are receiving their first
support as co-PIs on NSF awards.

![](media/image5.wmf)NSF will continue to seek creative and innovative
proposals from new investigators. Program staff will attend scientific
meetings, conferences, and conventions and will conduct site visits to
promote awareness of the research and education opportunities at NSF and
to encourage new investigators to submit proposals. NSF will examine
trends, such as whether the pool of new investigators is smaller than in
previous years or whether they are submitting fewer proposals, and if
needed, use this information to modify targets in the future.

In FY 2000 NSF implemented an electronic Program Announcement Template
(PAT) clearance process that is used by NSF staff to generate
announcements and solicitations. Use of the PAT ensures that PIs are
asked to address the integration of research and education in all
announcements and solicitations. In addition, the Foundation has
included language in the Proposal and Award Manual, the Grant Proposal
Guide, and the FY 2000 Guide to Programs regarding the importance of the
integration of research and education.

In order to verify that PIs are addressing the integration of research
and education, NSF asks Committees of Visitors (COVs) to assess whether
the broader impacts of the proposed activity are being addressed in
proposals and by reviewers and NSF staff as part of the merit review
process. The COV reporting template has been modified in FY 2001 to
explicitly address the use of both merit review criteria by reviewers
and program staff.

During FY 2000 screens were redesigned in FastLane (NSF's electronic
proposal and review system) so reviewers will be able to address each
merit-review criterion separately in FY 2001. This information is used
to aid in the determination of whether NSF has achieved this goal.

NSF modified program announcements to encourage applicants and reviewers
to address these criteria in proposals and reviews. NSF has recently
re-issued guidance to the proposing institutions and reviewers that
stresses the importance of addressing both merit review criteria in the
preparation and evaluation of proposals submitted to NSF. NSF staff
continue to stress the importance of reviewers addressing the "*broader
impacts*" criterion whenever they attend NSF-sponsored seminars, science
meetings, site visits, conferences, and conventions.

NSF is strongly committed to increasing the participation in all NSF
activities of science and engineering researchers, educators, and
students from groups currently under-represented in the science and
engineering enterprise. Congress enacted legislation giving NSF explicit
responsibility for addressing issues of equal opportunity in science and
engineering. This assignment of responsibility reflected the serious
underrepresentation of women, minorities, and persons with disabilities
in the science and engineering workforce, underrepresentation that
persists to this day, although some progress has been made.

NSF is committed to the principle of diversity and deems it central to
the programs, projects, and activities it considers and supports. NSF
continues to work toward increasing diversity in its proposal applicant
pool through the following means:

-   To place the issue on equal footing with the quality of research
    being supported, NSF issued Important Notice No. 125 to presidents
    of universities and colleges encouraging PIs to address the merit
    review criterion -- what are the broader impacts of the proposed
    activity - which embraces integrating diversity into all NSF
    supported activities;

-   Developing and increasing funding for specialized programs designed
    to promote diversity;

-   Recruiting members of under-represented groups for merit review
    panels, COVs, and NSF workshops and conferences; and

-   Strongly encouraging women, minorities, and persons with
    disabilities to compete fully in NSF programs.

NSF is revising this long-term goal to extend its efforts as it
continues to pursue diversity in the applicant pool. A new goal designed
to broaden participation of under-represented groups in FY 2001 will
build on the results of this goal by targeting the reviewer pool.

Every year, in the President's Budget Request to Congress, NSF sets out
a cost plan and schedule for major construction and upgrade projects
currently underway or planned for initiation in the Major Research
Equipment account. NSF has established performance goals and
measurements with respect to these plans and expects each construction
and upgrade activity to meet these performance goals. NSF consults with
other agencies to avoid duplication and to optimize capabilities
available to American researchers and educators, and cooperates with
other agencies in construction of facilities for use across broad
communities of researchers and educators. NSF manages facilities in the
Antarctic that are used by all federal agencies for selected projects.
Many major facilities involve international cooperation.

Facilities must operate efficiently and reliably and must offer
appropriate opportunities if they are to be valuable to those they
serve. NSF program officers work closely with facility directors to
ensure that the facilities have appropriate resources to conduct
operations and to provide maintenance that ensures reliable operations.

In order to report on the government-wide performance goals related to
Facility Operations, and Construction and Upgrade, NSF developed in FY
1999 a new Performance Reporting System (as a module of the existing
FastLane system), to collect information on facility operations and
construction from facilities managers external to NSF. As is the case
with any new data collection effort, we expect the quality of the
information provided to improve in subsequent years as managers gain
experience with gathering and reporting the required data. In FY 1999
NSF developed a general facilities reporting template for use in
collecting information on the construction, upgrade, and operations
goals. This reporting system was linked to the new Project Reporting
System (as a module of the existing FastLane system). The manager of
each facility reports the data to NSF. FY 1999 was the first year that
NSF collected data on these goals.

## 

Of the eleven construction and upgrade projects supported by NSF, all
were within annual expenditure plans; six met the planned annual cost
and five were less than the estimated cost. This goal was achieved in FY
1999. The majority of facilities were within annual spending estimates
of 110%. This goal will be revised in FY 2001 to require that 90% of
NSF-supported facilities keep construction and upgrades within their
annual expenditure plan.

Of the eleven construction and upgrade projects supported by NSF, seven
reported that all of their scheduled milestones were completed within
110 percent of the estimated time for completion. For four projects,
missed milestones were due to circumstances beyond the project manager's
control. For example, one construction project was dependent upon the
research and development of new instrumentation, the results of which
were delayed. In other projects, the missed milestone was due to
difficulty acquiring necessary parts; non-performance of a
sub-contractor; and underestimation of the complexity of the work. One
project did not report. In FY 2001 NSF program managers are working more
closely with project managers to ensure all NSF-supported
construction/upgrade projects achieve this goal. This goal will be
revised in FY 2001 to require that 90% of NSF-supported facilities keep
their planned construction and upgrades within annual schedule.

This goal will be maintained in FY 2001.

Facilities must operate efficiently and reliably and must be available
on schedule if they are to be useful to those they serve. NSF program
officers work closely with facility directors to ensure that facilities
have appropriate resources to operate reliably and schedule necessary
maintenance.

Of the 26 reporting facilities, 22 met the goal of keeping unscheduled
downtime to below 10% of the total scheduled operating time. Four
reported unscheduled downtime greater than 10%.

In FY 2000 NSF reviewed the FY 1999 data collection and reporting effort
and made modifications to the FY 2000 and FY 2001 systems in order to
improve the efficiency, clarity and accuracy of the process. This
included allowing for reporting on construction/upgrade activities at
facilities funded through the Research and Related Activities Account,
refining the clarity of the on-screen language, addressing the
facilities goals more accurately, automating most of the output, and
instituting a stage for collecting estimates. NSF program staff will
work more closely with project managers to ensure that all achieve this
goal in FY 2001.

The *on time* and *on schedule* goals for FY 2001 will be revised
slightly so that when at least 90 percent of facilities meet the federal
standard, the goal is considered achieved. These changes are being made
because NSF places great importance on accurate planning for
construction and upgrade of facilities, but we recognize that the
unique, state-of-art projects being supported stretch the limits of
technological capability. As a result there may be unexpected
construction delays and/or unforeseen expenditures. NSF expects that the
vast majority of its projects will be within budget and on schedule.
However, we do not believe the agency should be considered unsuccessful
overall in these areas if a small percentage of facilities are unable to
meet the goals. Therefore, to provide the flexibility necessary for NSF
to report realistic and achievable goals, we are reestablishing the
target level of success at 90% of the facilities for FY 2001. This
change will be evaluated over time to determine if 90% is the
appropriate level for this goal.

The *operating time* goal will also be revised from 100% to 90% for FY
2001. NSF recognizes that some facilities may have a failure rates
greater than 10%, but that this is balanced overall by facilities that
operate more reliably. NSF expects that the vast majority of facilities
will keep operating time lost due to unscheduled downtime to less than
10% of the operating time. We do not believe the agency should be
considered unsuccessful if a small percentage of the facilities are
unable to meet this goal. Therefore, to provide the flexibility
necessary for NSF to report realistic and achievable goals, we are
reestablishing the target level of achievement at 90% of the facilities
for FY 2001. This change will be evaluated over time to determine if 90%
is the appropriate level for these goals

# 

# D. Table of Evaluations

Table 2 below provides information on the program assessments and
evaluations other than Committee of Visitor and Advisory Committee
assessments - with one exception -- the Major Research Instrumentation
(MRI) program. The MRI program is an agency-wide activity, and is the
first Committee of Visitor (COV) review NSF has contracted to an
external private vendor.

The table lists other types of evaluations, not used in GPRA performance
assessment, that were completed in FY 2000 and for which information was
available at the time this report was prepared. These reports, studies,
and evaluations are frequently used in setting new priorities in a field
or in documenting progress in a particular area. The reader is
encouraged to review the reports for additional information on findings
and recommendations which are beyond the scope of this report. A table
showing the schedule for COV assessments appears in Section XV. A
discussion of results obtained for Outcome Goals based on the COV and
advisory committee assessments is presented in Section V. A.

Reports (other than COV reports) produced by NSF are available online at
http://www.nsf.gov/pubs/start.htm using the NSF's online document system
and the publication number indicated. COV reports will become
electronically available in December, 2001.

Information on obtaining reports produced by the National Research
Council or National Academy of Sciences can be found online by searching
www.nap.edu or from the National Academy Press, 2101 Constitution
Avenue, N.W., Lockbox 285, Washington, D.C. 20055 (1.800.642.6242).

## Table 2

+--------------+---------------------+-------------------+------------+
| #            | ###  {#secti        | ###  {#section    | #          |
| ## Evaluatio | on-21 .Box-Heading} | -22 .Box-Heading} | ##  {#sect |
| ns completed |                     |                   | ion-23 .Bo |
|  in FY 2000  | ### Scope {#        | ###               | x-Heading} |
| {#evaluation | scope .Box-Heading} |  Findings {#findi |            |
| s-completed- |                     | ngs .Box-Heading} | ### A      |
| in-fy-2000 . |                     |                   | vailabilit |
| Box-Heading} |                     |                   | y {#availa |
|              |                     |                   | bility .Bo |
|              |                     |                   | x-Heading} |
+--------------+---------------------+-------------------+------------+
| **Report of  | Initial review of   | Program           | Will be    |
| the          | MRI program for     | effectively uses  | elec       |
| Committee of | period FY 1995-FY   | merit review      | tronically |
| Visitors:    | 1999; program       | process to        | available  |
| Major        | processes and       | generate          | through    |
| Research     | management; program | appropriate       | NSF web    |
| Ins          | results and goals   | portfolio of      | site       |
| trumentation | specific to MRI     | awards based on   | December   |
| Program**    | program.            | quality of        | 2001       |
|              |                     | proposed          |            |
|              |                     | instrument; not   |            |
|              |                     | as effective in   |            |
|              |                     | use of "broader   |            |
|              |                     | impact"           |            |
|              |                     | criterion;        |            |
|              |                     | evaluation of     |            |
|              |                     | progress in       |            |
|              |                     | meeting most      |            |
|              |                     | outcome goals     |            |
|              |                     | difficult because |            |
|              |                     | few results have  |            |
|              |                     | yet been achieved |            |
|              |                     | and some are      |            |
|              |                     | beyond the scope  |            |
|              |                     | of the program.   |            |
+--------------+---------------------+-------------------+------------+
| ###          | ###  {#secti        | ###  {#section    | #          |
|  Evaluations | on-24 .Box-Heading} | -25 .Box-Heading} | ##  {#sect |
|  completed i |                     |                   | ion-26 .Bo |
| n FY 2000 {# | ### Scope {#sc      | ### F             | x-Heading} |
| evaluations- | ope-1 .Box-Heading} | indings {#finding |            |
| completed-in |                     | s-1 .Box-Heading} | ### Ava    |
| -fy-2000-1 . |                     |                   | ilability  |
| Box-Heading} |                     |                   | {#availabi |
|              |                     |                   | lity-1 .Bo |
|              |                     |                   | x-Heading} |
+--------------+---------------------+-------------------+------------+
| **Progress   | Review of           | Coalitions made   | NSF 00-116 |
| of the       | effectiveness and   | important         |            |
| Engineering  | progress in         | contributions and | May 2000   |
| Education    | educational reform  | facilitated the   |            |
| Coalitions** | through engineering | implementation of |            |
|              | coalitions program. | performance-based |            |
|              |                     | accreditation     |            |
|              |                     | standards (ABET   |            |
|              |                     | 2000).            |            |
+--------------+---------------------+-------------------+------------+
| **Measuring  | Review of the SRS   | Recommends        | National   |
| the Science  | portfolio of data   | expansion and     | Research   |
| and          | collection,         | modification of   | Council    |
| Engineering  | acquisition, and    | SRS data          |            |
| Enterprise:  | analysis            | activities such   |            |
| Priorities   | activities.         | as: increased     |            |
| for the      |                     | interaction with  |            |
| Division of  |                     | users and         |            |
| Science      |                     | customers;        |            |
| Resource     |                     | increase          |            |
| Studies in   |                     | timeliness of     |            |
| 2000**       |                     | release data;     |            |
|              |                     | expand data       |            |
|              |                     | collections for   |            |
|              |                     | some areas;       |            |
|              |                     | revise collection |            |
|              |                     | surveys.          |            |
+--------------+---------------------+-------------------+------------+
| **Challenges | Comparison of       | Source of         | Con        |
| in           | numbers reported by | discrepancy is    | gressional |
| Collecting   | the federal         | almost            | Research   |
| and          | agencies as outlays | exclusively with  | Service,   |
| Reporting    | for federal R&D on  | reporting by      | Library of |
| Federal      | National Science    | performers; CRS   | Congress   |
| Research and | Foundation surveys  | suggests further  | Order Code |
| Development  | with those reported | study and         | RL30413    |
| Data**       | by federal R&D      | increased support |            |
|              | performers as       | to improve R&D    |            |
|              | expenditures or     | data collection   |            |
|              | reimbursements from | and reporting.    |            |
|              | federal agencies.   |                   |            |
+--------------+---------------------+-------------------+------------+
| **Na         | Identifies          | Recommends long   | See        |
| notechnology | challenges and      | term fundamental  | reference  |
| Research     | opportunities in    | nanoscience and   | 1          |
| Directions:  | nanotechnology      | engineering       |            |
| IWGN         | field; outlines how | research,         |            |
| Workshop     | advances in field   | synthesis and     |            |
| Report**     | can impact national | processing 'by    |            |
|              | economy, health     | design' of        |            |
|              | care and national   | material building |            |
|              | security.           | blocks, and       |            |
|              |                     | education and     |            |
|              |                     | training of       |            |
|              |                     | future workforce. |            |
+--------------+---------------------+-------------------+------------+
| **Cond       | Scholarly           | Provides advice   | National   |
| ensed-Matter | assessment of field | for support of    | Academy    |
| and          | as part of a new    | the field and     | Press      |
| Materials    | survey of physics,  | what areas should |            |
| Physics --   | *Physics in a New   | receive increased |            |
| Basic        | Era,* that is in    | investment.       |            |
| Research for | progress.           |                   |            |
| Tomorrow's   |                     |                   |            |
| Technology** |                     |                   |            |
+--------------+---------------------+-------------------+------------+

**Reference 1:** http://www.nsf.gov/home/crssprgm/nano/start.htm.

+--------------+---------------------+-------------------+------------+
| ###          | ###  {#secti        | ###  {#section    | #          |
|  Evaluations | on-27 .Box-Heading} | -28 .Box-Heading} | ##  {#sect |
|  completed i |                     |                   | ion-29 .Bo |
| n FY 2000 {# | ### Scope {#sc      | ### F             | x-Heading} |
| evaluations- | ope-2 .Box-Heading} | indings {#finding |            |
| completed-in |                     | s-2 .Box-Heading} | ### Ava    |
| -fy-2000-2 . |                     |                   | ilability  |
| Box-Heading} |                     |                   | {#availabi |
|              |                     |                   | lity-2 .Bo |
|              |                     |                   | x-Heading} |
+--------------+---------------------+-------------------+------------+
| **Astronomy  | Assessment of       | Report identifies | National   |
| and          | field, identifies   | key areas of      | Research   |
| Astrophysics | fundamental         | astronomy and     | Council    |
| in the New   | scientific          | astrophysics for  |            |
| Millennium** | challenges,         | advances to       |            |
|              | assesses            | increase          |            |
|              | infrastructure and  | understanding of  |            |
|              | impact on society,  | the universe.     |            |
|              | international       |                   |            |
|              | activity, and       |                   |            |
|              | balance of national |                   |            |
|              | objectives,         |                   |            |
|              | coordination of     |                   |            |
|              | federal agencies.   |                   |            |
+--------------+---------------------+-------------------+------------+
| **Materials  | Addresses the       | In depth study    | National   |
| Science and  | relationships among | covers three      | Academy    |
| Engineering  | academia,           | sectors:          | Press      |
| -- Forging   | government,         | automotive        |            |
| Stronger     | government          | industry,         |            |
| Links to     | laboratories and    | jet-engine        |            |
| Users**      | industry in the     | industry, and     |            |
|              | materials science   | computer-chip and |            |
|              | and engineering     | in                |            |
|              | field, including    | formation-storage |            |
|              | the relationships   | industries.       |            |
|              | among the producers | Provides advice   |            |
|              | and users of        | for mechanisms to |            |
|              | materials and the   | support           |            |
|              | processes of        | pre-competitive   |            |
|              | innovation.         | research,         |            |
|              |                     | multidisciplinary |            |
|              |                     | research, and the |            |
|              |                     | facilitation of   |            |
|              |                     | un                |            |
|              |                     | iversity-industry |            |
|              |                     | interactions.     |            |
+--------------+---------------------+-------------------+------------+
| *            | To explore possible | U.S. funding      | National   |
| *Cooperative | strategies to       | agencies should   | Academy    |
| Stewardship  | address changing    | adopt a           | Press      |
| -- Managing  | usage of research   | cooperative       |            |
| the Nation's | facilities          | stewardship model |            |
| Multi        | (synchrotron        | for managing      |            |
| disciplinary | radiation, neutron  | facilities.       |            |
| User         | beam, and           |                   |            |
| Facilities   | high-magnetic field |                   |            |
| for Research | facilities) and     |                   |            |
| with         | changing roles of   |                   |            |
| Synchrotron  | the supporting      |                   |            |
| Radiation,   | agencies.           |                   |            |
| Neutrons,    |                     |                   |            |
| and High     |                     |                   |            |
| Magnetic     |                     |                   |            |
| Fields**     |                     |                   |            |
+--------------+---------------------+-------------------+------------+
| **NSF        | A decadal outlook   | The report        | NSF 00-27  |
| Geosciences  | for the geosciences | outlines the      |            |
| Beyond 2000, | evaluating          | scientific        |            |
| U            | opportunities and   | programs needed   |            |
| nderstanding | requirements for    | to continue the   |            |
| and          | research, education | expansion of the  |            |
| Predicting   | and infrastructure. | basic knowledge   |            |
| Earth's      |                     | of Earth systems. |            |
| Environment  |                     |                   |            |
| and          |                     |                   |            |
| Ha           |                     |                   |            |
| bitability** |                     |                   |            |
+--------------+---------------------+-------------------+------------+

+--------------+---------------------+-------------------+------------+
| ###          | ###  {#secti        | ###  {#section    | #          |
|  Evaluations | on-30 .Box-Heading} | -31 .Box-Heading} | ##  {#sect |
|  completed i |                     |                   | ion-32 .Bo |
| n FY 2000 {# | ### Scope {#sc      | ### F             | x-Heading} |
| evaluations- | ope-3 .Box-Heading} | indings {#finding |            |
| completed-in |                     | s-3 .Box-Heading} | ### Ava    |
| -fy-2000-3 . |                     |                   | ilability  |
| Box-Heading} |                     |                   | {#availabi |
|              |                     |                   | lity-3 .Bo |
|              |                     |                   | x-Heading} |
+--------------+---------------------+-------------------+------------+
| **National   | Review of merit of  | Planning and      | National   |
| Research     | seafloor            | implementation of | Research   |
| Coun         | observatories.      | a seafloor        | Council/   |
| cil/National |                     | observatory       | National   |
| Academy of   |                     | program should    | Academy of |
| Sciences:    |                     | move forward.     | Sciences   |
| Illuminating |                     |                   |            |
| the Hidden   |                     |                   | July 2000  |
| Planet: The  |                     |                   |            |
| Future of    |                     |                   |            |
| Seafloor     |                     |                   |            |
| Observatory  |                     |                   |            |
| Science**    |                     |                   |            |
+--------------+---------------------+-------------------+------------+
| **National   | Review of program   | Recommendations   | National   |
| Research     | balance and         | address new       | Research   |
| Coun         | research            | mechanisms to     | Council/   |
| cil/National | opportunities.      | exploit research  | National   |
| Academy of   |                     | opportunities.    | Academy    |
| Sciences:    |                     |                   | Press      |
| Basic        |                     |                   |            |
| Research     |                     |                   | 2000       |
| O            |                     |                   |            |
| pportunities |                     |                   |            |
| in Earth     |                     |                   |            |
| Science**    |                     |                   |            |
+--------------+---------------------+-------------------+------------+
| **The        | GRT projects        | As of 1998,       | Available  |
| Graduate     | evaluated on the    | almost half of    | from NSF   |
| Research     | number of students  | the nearly 200    | in FY 2001 |
| Traineeships | reached and on      | students          |            |
| (GRT)        | processes carried   | receiving a Ph.D. |            |
| Program**    | out to meet goals.  | with partial      |            |
|              |                     | support through   |            |
|              |                     | the GRT program   |            |
|              |                     | had obtained      |            |
|              |                     | postdoctoral      |            |
|              |                     | positions and     |            |
|              |                     | half were working |            |
|              |                     | in education,     |            |
|              |                     | government, or    |            |
|              |                     | private           |            |
|              |                     | employment.       |            |
+--------------+---------------------+-------------------+------------+
| **Co         | Review of changes   | Too early for     | Available  |
| llaboratives | in learning         | information on    | from NSF   |
| for          | infrastructure,     | students; program | in FY 2001 |
| Excellence   | faculty             | is meeting        |            |
| in Teacher   | involvement, and    | objectives in     |            |
| Preparation  | student outcomes.   | areas of learning |            |
| Program      |                     | infrastructure    |            |
| (CETP)**     |                     | and faculty       |            |
|              |                     | involvement.      |            |
+--------------+---------------------+-------------------+------------+
| **           | To determine        | Most initiatives  | NSF 00-73  |
| Mathematical | whether curricula   | successful in     |            |
| Sciences and | for undergraduates  | developing and    |            |
| Their        | was developed and   | disseminating     |            |
| Applications | new partnerships    | materials; less   |            |
| Throughout   | among higher        | success in        |            |
| the          | education           | developing and    |            |
| Curriculum:  | institutions were   | maintaining       |            |
| Final        | created by          | institutional     |            |
| Report**     | initiatives.        | partnerships.     |            |
+--------------+---------------------+-------------------+------------+
| **Best       | To determine best   | All programs had  | Available  |
| Practices    | practices in        | recognized        | from NSF   |
| Study of     | programs for        | strengths; NSF    | in FY 2001 |
| Federal      | undergraduate       | program focused   |            |
| Minority     | minority programs   | on less           |            |
| U            | across NSF, NASA,   | well-prepared     |            |
| ndergraduate | and HHS.            | students.         |            |
| SMET         |                     |                   |            |
| Programs**   |                     |                   |            |
+--------------+---------------------+-------------------+------------+
| ###          | ###  {#secti        | ###  {#section    | #          |
|  Evaluations | on-33 .Box-Heading} | -34 .Box-Heading} | ##  {#sect |
|  completed i |                     |                   | ion-35 .Bo |
| n FY 2000 {# | ### Scope {#sc      | ### F             | x-Heading} |
| evaluations- | ope-4 .Box-Heading} | indings {#finding |            |
| completed-in |                     | s-4 .Box-Heading} | ### Ava    |
| -fy-2000-4 . |                     |                   | ilability  |
| Box-Heading} |                     |                   | {#availabi |
|              |                     |                   | lity-4 .Bo |
|              |                     |                   | x-Heading} |
+--------------+---------------------+-------------------+------------+
| **Program    | Review of           | Program           | Available  |
| for Gender   | collaborations      | successful in all | from NSF   |
| Equity       | developed among     | areas: most       | in FY 2001 |
| (PGE)**      | educational         | projects          |            |
|              | organizations,      | replicated or     |            |
|              | number of           | i                 |            |
|              | individuals         | nstitutionalized; |            |
|              | impacted by         | nearly 85,000     |            |
|              | projects, findings  | participants      |            |
|              | on gender equity.   | served.           |            |
+--------------+---------------------+-------------------+------------+
| **Faculty    | Study of first      | CAREER awardees   | Available  |
| Early Career | three years of      | reported more     | from NSF   |
| Development  | award impact and    | rapid advancement | in FY 2001 |
| (CAREER)     | value to awardees,  | in professional   |            |
| Program**    | to determine if     | careers than      |            |
|              | CAREER awardees     | non-CAREER        |            |
|              | demonstrated        | awardees.         |            |
|              | greater career      |                   |            |
|              | advancement than    |                   |            |
|              | non-CAREER          |                   |            |
|              | awardees.           |                   |            |
+--------------+---------------------+-------------------+------------+

# 

# VI. Verification and Validation

## 

## A. Quality of the Reported Performance Information

In FY 1999 concerns were expressed by the General Accounting Office
(GAO) with regard to the quality of reported performance information
used by NSF. To address these concerns in FY 2000, NSF engaged an
external third party, Price-waterhouseCoopers LLP (PwC), to verify and
validate selected FY 2000 GPRA performance data as well as the process
used in collecting and compiling data and information. In their final
reports, PwC concluded that NSF was reporting its GPRA measures with
"sufficient accuracy such that any errors, should they exist, would not
be significant enough to change the reader's interpretation as to the
Foundation's success in meeting the supporting performance goal . . . ."
Furthermore, PwC concluded that NSF "relies on sound business processes,
system and application controls, and manual checks of system queries to
confirm the accuracy of reported data. We believe that these processes
are valid and verifiable."

KPMG LLP, an independent certified public accounting firm, was selected
by the NSF Inspector General to perform an audit of NSF's FY 2000
financial statements. Their review included a review of the collection
process and maintenance of data and information for NSF's GPRA goals.
NSF received an unqualified opinion stating that the principal financial
statements were fairly stated in all material respects. The independent
auditors did not report any material weaknesses in internal control or
material noncompliance with laws or regulations.

All data are imperfect in some way. Establishing responsible and
reasonable verification and validation procedures and understanding data
limitations requires a balanced approach. NSF acknowledges the need to
improve data systems for collecting and maintaining performance
information and data as budget and time allow, and regards this as an
evolutionary process which will continue to improve with time. NSF is
comfortable with the quality of the data it uses in assessing the
overall progress of the agency in meeting performance goals and makes
use of the information it gains through performance reporting to improve
policies, practices and management of the agency. Implementing GPRA has
enabled NSF to gather information in a structured way, and to address
issues in a more formal, focused way than in the past.

Because basic research and education projects rarely produce results in
less than three to five years, it is difficult to compare the outcomes
reported in the fiscal year with the funds that were obligated in that
year. In some cases, the results of NSF support may not be recognized
and reported for twenty years or more. Because the GPRA reporting
schedule is annual, NSF conducts an annual assessment or evaluation of
results submitted to the agency in the fiscal year, which is a
retrospective evaluation, carried out by external experts. This
retrospective evaluation makes use of the alternative form for
reporting, to cover about thirty percent of NSF's total portfolio in one
year. This makes sense for NSF's Outcome Goals, which are long-term
goals, and are not expected to be achieved in a short time period.
Nonetheless, we are concerned that the current form of reporting goal
achievement under GPRA does not convey the accomplishments of NSF or the
full value of the NSF investment. To help the reader understand the
level and accomplishment of performance for the Outcome Goals, examples
are included to illustrate achievements reported during the fiscal year.
While they may appear to be anecdotal, they can be traced to
NSF-supported awards.

## B. Data Verification and Validation Activities

For reporting goal achievment, all of NSF's goals are aggregated across
the agency. To enable aggregation, NSF developed reporting templates in
FY 1998, and data modules to collect data uniformly across the agency.
These modules and templates were revised and refined in FY 2000 and were
based on information gained in using the templates and systems in FY
1999. In FY 1999 NSF established a Data Quality Program to assess and
improve the quality of data within the Foundation. NSF will continue to
further refine data collection methods and systems to address areas in
need of improvement as time and funds allow.

During FY 1999 NSF staff implemented a Data Quality Project for the
quantitative Investment Process and Management goals. The objectives of
the project were to:

1.  Evaluate the quality of the data in the central databases.

2.  Ensure the paper documents and the NSF central databases are
    synchronized.

3.  Identify inconsistencies so that methods for correcting the cause of
    the inconsistencies can be developed.

4.  Ascertain the causes of the data quality problems and develop
    systematic methods for correction.

5.  Develop a comprehensive data dictionary.

6.  Promulgate data quality policies and procedures NSF-wide.

In FY 2000, NSF increased the expected quality of information for the
Outcome Goals in two ways:

1.  NSF changed the two-level standard from *successful/minimally
    effective* to *successful.*

2.  NSF required thorough justification for "grades."

NSF staff update and revise guidelines and reporting procedures for
collecting data for the Outcome Goals annually. The Committee of Visitor
(COV) guidelines were revised in FY 1999 and 2000 to incorporate the
GPRA-related reporting requirements. COVs address a common set of
questions for all programs reviewed in a fiscal year. Reporting
guidelines were also developed for Advisory Committees to enable
uniform, systematic aggregation of information. The results of using the
new procedures helped to identify areas for improvement to the
guidelines. These were incorporated for FY 2000 reporting and guidelines
will be revised in FY 2001 based on experiences in FY 2000. The
experience gained while conducting these assessments has also been used
in revising the FY 2001 and FY 2002 Performance Plans and goals, and the
updated Strategic Plan.

In addition, for FY 2000 NSF established parameters to define the
acceptability and reliability of the qualitative information it uses.
NSF defined the quality of the information it uses to insure uniform
quality of results and applied stricter definitions of success in
determining whether Outcome Goals had been met. NSF used a confidence
limit to identify non-substantive information. Information falling
outside the confidence limit was excluded from use.

The overall effect of applying these stricter definitions was to raise
the expected performance level and reduce the aggregated success rate
for NSF in FY 2000. However, the performance of the agency as a whole in
FY 2000 was very much the same as in FY 1999 and positive trends are
beginning to emerge. Many of the same issues identified by external
groups in FY 1999 were identified in FY 2000. This is an interesting
result in itself, since the Outcome Goals make use of judgement by
different groups of external experts each year, and one might expect the
result to be different if done by different groups, but this was not the
case. Thus, this second year of reporting validated results obtained in
the first year. A more complete picture will be obtained when results
for the third year of reporting are known.

Information gathered from external sources for use in measuring
performance related to the Outcome Goals is checked by NSF staff,
reviewed by groups of external evaluators, and is subject to audit and
tracking by association with grant numbers. In assessing its performance
NSF makes use of reports generated by COVs who provide judgements. The
scores and comments are compiled and aggregated to determine the success
of the agency in meeting the Outcome Goals. This process was reviewed by
PwC, who noted that the "approach NSF uses to assess its performance
under these specific qualitative measures is reasonable"\... and that in
comparing NSF's results with PwC's results using the process established
by NSF, "overall conclusions regarding program success or lack thereof
in respect to individual goals remained largely unchanged."

It is likely that NSF will continue to make use of external third
parties on an appropriate schedule, to verify and validate data used in
reporting performance goals as funds are available for this purpose.

## C. Types and Sources of Performance Data [and Information]{.smallcaps}

The data used in measuring performance are developed by and come from a
variety of sources. Much of the data originate outside the agency, and
quality is beyond the control of the NSF. Data come from administrative
records, awardee reports, external committee reports, and internal data
systems. Additional information can be found in the FY 2001 Performance
Plan.

Quantitative data is used primarily in assessing the Investment Process
and Management Goals. Most quantitative data used in assessing
performance is collected using internal data systems and is reviewed by
staff on a quarterly schedule.

Most of the qualitative information used in assessing Outcome Goal
performance is provided to NSF by external evaluators (COVs) near the
end of the fiscal year, and is reviewed by NSF senior management as it
becomes available.

Collection of data is dependent on the type of data/information.
Collection of data for all goals takes place throughout the year and is
completed near the end of the fiscal year. Data are collected into
reports for each goal by a group staff having reporting responsibility
for the particular goal. The data obtained are reviewed on a continuing
basis by senior management throughout the year, in order to observe
whether the results are as expected, whether performance needs to be
improved, whether targets need adjustment, or whether the information
being obtained is useful to the agency. Data collection systems are also
under constant observance and refinement.

## D. Data Limitations

Specific data limitation issues are discussed below. The NSF FY 2001
Performance Plan contains additional information on data sources and
limitations.

This is the second year in which reports were collected, tabulated, and
an assessment of NSF's performance was completed. Several data
quality/limitation issues were identified in the first year of
reporting. The agency worked to address these issues during FY 2000. The
issues included: incomplete data collection systems related to some of
the quantitative goals (such as the goals related to Facility Operations
-- Investment Goals 12-15); the need to improve report templates to
ensure that the performance information provided by external groups is
more complete and consistent for the qualitative Outcome Goals; and
explanations for goals that were missed.

Steps were taken to improve the quality and value of performance data
for the Outcome Goals. They included improved reporting templates for
collecting program performance information from external committees by
asking for more complete justifications for ratings. We note an improved
quality and consistency of COV reports for FY 2000, but note they are
still not optimal. NSF has modified the COV reporting template
guidelines for FY 2001 to further improve consistency and completeness.
NSF staff will work more closely with COV members to ensure improved
reporting. This will aid NSF in aggregating qualitative information for
measuring progress in achieving the Outcome Goals.

NSF employs an alternative form for determining progress made in
achieving its Outcome Goals for research and education. In FY 1999 NSF
made use of the alternative form using the two-standard approach
required by the Act (*successful* or *minimally effective*). In doing
so, NSF learned that there was little to be gained in using *minimally
effective* standard, and that in many instances it was confusing to the
evaluators. Therefore, for FY 2000, NSF defined one standard only: the
*successful* standard. The effect of this change was to increase the
level of expected performance for the Outcome Goals.

When NSF collects performance information it asks COVs to indicate when
data is not adequate or available to evaluate progress toward meeting an
Outcome Goal. NSF found in FY 1999 and FY 2000 that external evaluators
did not always have adequate information available to judge each program
in use of the merit review criteria; in achieving increased
participation from under-represented groups; and, in achieving science
and math skills for ***all*** Americans. NSF management is reviewing
means that will help NSF staff to provide this information for FY 2001
assessments. In some instances, data is difficult to obtain. An examples
is complete data describing the participation of under-represented
groups, which is voluntary.

NSF is also reviewing the wording of goals to correct issues which
created difficulty when aggregating results in FY 1999 and FY 2000. For
example, Outcome Goal 3 combines achieving increased diversity with
achieving a globally-oriented workforce. While NSF was judged successful
in achieving a globally-oriented workforce in most programs, it was
judged less successful in achieving increased diversity. Consequently,
NSF is not able to indicate success across the entire agency in FY 2000
for this goal, although some aspects of the goal were realized by
programs, in particular those programs with funds targeted directly to
meet these goals.

A similar situation arose in evaluating Outcome Goal 4, which targets
improved math and science skills for ***all*** Americans. NSF programs
were successful when they had clearly invested funds to support
activities relevant to achieving this goal. It was less apparent to
external groups whether success had been achieved for programs not
designed to specifically address this goal, and the resulting COV
reports did not provide clear evidence of success at the aggregate
level. Therefore, for this goal, we are unable to indicate successful
performance for the agency.

A new format has been adopted for NSF's goals in FY 2001 which we hope
will help to alleviate some of these issues: the five Outcomes Goals
have been organized under three headings, each with independent
indicators. This will aid assessment by COVs to address the indicators
separately when they are relevant to the program being evaluated.

Another limitation noted was for Investment Process Goal 8, Maintaining
Openness in the System. It was found that the identification of new PIs
was inaccurate on occasion, and steps were taken to identify such
individuals in the NSF PI system more carefully. For Investment Process
Goals 12-15 on Facility Oversight, the reporting system was revised and
implemented in FY 2000. This system was provided to facility managers
located at universities who must use the NSF developed system to report
data that support this goal. There was lack of agreement in FY 1999 on
how the required data were defined, which led to different
interpretations. This deficiency was addressed in FY 2000. However,
facility managers are still gaining experience in collecting and
providing information needed for reporting these goals.

For the quantitative Management Goal 3 - Staff Diversity - a reported
data limitation for this goal is that an applicant by law cannot be
required to provide gender and ethnicity information. Thus, it is
certain that the results for this measure are not accurate. Goals which
require voluntary self-reporting are also subject to being incomplete or
inaccurate.

Finally, in reporting results for all goals in FY 1999, NSF did not
always have a complete understanding of why some goals were not met. For
FY 2000 NSF revised report templates for collecting information across
directorates and offices within NSF. Each reporting organization within
NSF was asked to provide a summary of performance at a lower
organizational level, and to provide explanations when agency goals were
not met along with plans to meet those goals. This provided the agency
with more complete information on goal performance in FY 2000 and has
helped to identify several important obstacles critical to achieving
some goals. This information is being used to develop implementation
strategies for meeting targets in future fiscal years.

## E. Other Issues - Timing

One of the most significant challenges for NSF is that results of
research and education investments do not appear annually or on
schedule. NSF faces other timing issues in preparing this report. Such
timing issues may be shared by other agencies. One timing issue is
related to NSF's reliance on external committees to conduct assessments
after the close of the fiscal year. Materials are prepared in advance,
but there is a narrow window of time between the end of the fiscal year
and the start of the calendar year for the assessment to be conducted by
external groups and the results to be finalized and written by the
external committees. The committee reports must be submitted to the
agency, and reviewed by the agency. NSF relies on the availability and
cooperation of the external community and their ability to deliver their
assessments on a tight schedule.

In addition, the timing and phasing of the annual plan, collection of
information, and data for reporting have been difficult to coordinate
with the budget process. To optimize goals and plans for the new fiscal
year, NSF must review progress from the prior fiscal year, and make
revisions to the annual plan for the upcoming year. However, the
Performance Plan is typically due well before the results of the prior
fiscal year are known. This creates an awkward situation, in that an
early Performance Plan may need significant revisions to best serve the
agency.

In FY 1999 and FY 2000 NSF found that the time needed to collect and
review the annual performance data, and incorporate changes into the FY
2001 and FY 2002 annual performance plans in a way which we believe
benefits the process and lead to desired results, was insufficient to
meet the current schedule set by law. We are reviewing staffing and
procedural mechanisms to accelerate the process for aggregating
performance results. However, this is unlikely to yield an agency result
at the right time to develop a plan appropriate for the upcoming year.

# 

# VII. Major Management Challenges

The United States Senate Committee on Governmental Affairs annually
conducts oversight of each agency to ensure that the federal government
delivers better results to its citizens and taxpayers. The Senate
Committee works through the Office of the Inspector General (OIG),
annually requesting that each agency's OIG identify the ten most serious
long-term management challenges facing their respective agency.

In an FY 1999 report to the Senate Committee (letter dated 1 December
1998), the NSF OIG identified ten significant NSF management challenges:

NSF responded to the Senate Committee in a letter dated September 28,
1999, noting that the NSF IG had stated overall that the Foundation is
well managed, and accordingly found these issues to be challenges,
rather than managerial "deficiencies".

In FY 2000 the NSF OIG report to the Senate Committee identified 10
management challenges which it considered to be the most important for
NSF. The NSF OIG found that the overall NSF investment portfolio was
healthy and the Foundation did not have significant management
deficiencies. The OIG identified the same list of challenges for NSF in
FY 2000 as it did in FY 1999. The OIG also stated that the Foundation
management had made continuous progress on each of these items in FY
2000.

In its FY 1999 Performance Report to Congress NSF did not discuss each
of the major management challenges identified by the NSF OIG. In
discussions with GAO the OIG indicated that NSF is taking effective
steps to respond to these challenges, and did not need to include each
of them in performance reports or future performance plans. However, the
OIG did express concerns about areas related to effective oversight and
the need for NSF to be alert to emerging situations that could result in
problems. NSF senior management, responding to the concerns of the NSF
OIG, continually review and monitor each of these areas and continue to
include select areas for assessment in annual performance plans.

For FY 2001, the NSF OIG has identified 10 areas (see below) including
some areas identified in previous years, and several new areas for NSF
to monitor:

NSF continues to maintain performance goals in annual performance plans
for FY 2001 and 2002 related to the use of merit review, use of
FastLane, workforce training, and increasing the diversity of the
scientific workforce. NSF has internal management controls which
continually monitor award administration, cost sharing by awardees, data
security and quality, and project management. NSF is committed to
achieving and maintaining the highest standards with integrity to
produce high quality outputs and outcomes, and to improving it's
performance overall in these areas and others on a continuing basis.

## 

# VIII. Transition to FY 2001 and Beyond

The NSF FY 2001 Performance Plan is based on NSF's updated GPRA
*Strategic Plan FY 2001* -- 2006, finalized in September 2000, and upon
newly developed Strategic Outcomes included therein. The chart below
clarifies the linkage between the new goals and those described in
earlier NSF GPRA documents. The Strategic Outcome Goal areas of
developing People, enabling Ideas, and providing Tools serve as the
linkage between NSF's mission and annual performance goals. The FY 2001
Performance Plan goals take into account lessons learned in FY 1999 and
FY 2000, strengths and weaknesses identified, recommendations from the
NSF Strategic Planning Integration Group, and input from the research
community, auditors, Congressional groups, and stakeholders. However, we
have since learned that our performance indicators for the Outcome Goals
may be too broadly-stated. Additional discussion of annual performance
goals and indicators pertaining to these Outcome areas may be found in
the NSF FY 2001 Performance Plan.

# IX. Information on use of non-Federal parties

This GPRA performance report was written and prepared solely by NSF
staff.

Non-Federal external sources of information used in preparing this
report include:

-   Reports from awardees demonstrating results.

-   Reports prepared by evaluators -- Committees of Visitors and
    Advisory Committees -- in assessing NSF programs for progress in
    achieving Outcome Goals.

-   Reports prepared by a consulting firm to assess the procedures the
    Foundation uses to collect, process, maintain, and report
    performance goals and measures.

Specific examples:

Highlights or sources of examples shown as results may be provided by
principal investigators who received support from NSF.

NSF uses external committees to assess the progress of programs toward
Outcome Goal achievement. External evaluators provide NSF with reports
of programs, and provide feedback to NSF on a report template prepared
by NSF. NSF makes use of these committee reports when assessing progress
the Foundation is making towards achieving its goals. Examples are COV
and Advisory Committee reports that provide an independent external
assessment of NSF's performance.

NSF engaged an independent third-party (PricewaterhouseCoopers LLP) to
conduct a review of data and information used in performance reporting.
PwC reviewed NSF's performance data and information pertaining to its
Outcome Goals, Management Goals, and Investment Process Goals. This
additional independent review helped to eliminate potential reporting
bias that can develop in self-assessments. It also provides assurance of
the credibility of performance reporting information and results.

# X. Budget Information

NSF obligated \$3.9 billion in FY 2000. Administrative support for the
Foundation was approximately 5% of the total NSF budget.

## **[Classified Appendices not Available to the Public]{.smallcaps}** 

None to report

## **[XII. Analysis of Tax Expenditures]{.smallcaps}** 

None to report

## **[XIII. Waivers of Administrative Requirements]{.smallcaps}**

None to report

# XIV. Appendix of Additional Examples Illustrating Outcomes of NSF Investments

## FY 2000 Examples of Achievements Cited by External Evaluators

## Outcome Goal 1

### Discoveries At and Across the Frontier of Science and Engineering

External evaluators cited the following examples of results from NSF
awards as demonstrating success in support of Outcome Goal 1. These
examples illustrate important discoveries, new knowledge and techniques,
both expected and unexpected, within and across traditional boundaries;
and high-potential links across these boundaries.

The examples shown illustrate NSF-supported results reported in the FY
2000 areas of emphasis for this outcome goal: balance of innovative,
risky, interdisciplinary research; new types of scientific databases and
tools to use them; life in extreme environments; biocomplexity; and
nanoscience and engineering. It is interesting to note that many results
cross the boundaries between discoveries, new knowledge,
interdisciplinary research, biocomplexity, and nanoscience. The diverse
portfolios of awards show potential for significant impact in these
areas.

-   **[Tracking Turbulence]{.smallcaps}** Turbulent flow of fluids is
    important in many fields, from atmospheric sciences to combustion
    science to fundamental fluid physics. For example, in the design of
    engines, fuel combustion leads to the production of hot gases that
    are very turbulent. This turbulence affects the amount of fuel that
    is actually burned and significantly affects the efficiency of the
    engine and the level of pollutants it produces. Being able to
    measure turbulence can lead to the design of more efficient engines.
    A major problem is to monitor the fluid flow, which is characterized
    by high \"Reynolds Numbers\". A low Reynolds Number indicates smooth
    flow, and a high Reynolds Number indicates chaotic flow. Recent
    innovative work has begun to apply particle physics techniques and
    particle detectors to follow the motion of numerous small buoyant
    \'particles\' to map out the complex turbulence patterns. In other
    recent work, seminal studies of turbulence over an unprecedented
    range of Reynolds Numbers was achieved using cryogenic helium, near
    its critical point of 4.5 Kelvin, a very low temperature. (A Kelvin
    is a measure of absolute temperature. A temperature of 0 Kelvin,
    equal to absolute 'zero' at which all motion of atoms and molecules
    would theoretically cease, would be colder than --273 degrees. Most
    materials would be frozen solid at much higher temperatures.) Liquid
    helium is the fluid of choice for ultra-high Reynolds studies
    because its physical properties lead to the ability of scientists to
    create turbulent flow at possibly the highest Reynolds Numbers in
    the universe, right in an earth-bound cryostat. Turbulence studies
    done at very low temperatures will impact aeronautical, chemical,
    and mechanical engineering fields.

-   Innovative adaptation of retired undersea telephone cables to
    establish a sea bed seismological observatory, called H2O, under 5
    km (about 4.6 miles) of water in the Pacific Ocean halfway between
    California and Hawaii. This resource will enable detailed
    characterization of the Earth's mantle under the northeastern
    Pacific Ocean.

-   In the Antarctic a high-risk, international project netted
    impressive results about the long-term evolution of the Antarctic
    climate and ice sheets, marine life, and also topography and
    tectonics. The recorded Antarctic changes help to explain some of
    the puzzling major changes in global climate on timescales of
    millions of years.

-   The tundra regions on the Alaskan North Slope were found to have
    recently shifted from being sinks for greenhouse gases to being
    sources for these gases in winter This is important because the
    large regions of Arctic tundra now represent a potentially
    significant addition to the global greenhouse gas budget. In the
    Antarctic, discoveries of rapid and episodic algal blooms in the
    Ross Sea were linked to fluctuations in the export of carbon from
    the ocean surface. When extrapolated to larger spatial scales, such
    blooms have the potential to explain major fluctuations in
    atmospheric carbon dioxide concentration.

-   NSF recently funded the continued development of a small,
    long-range, long- endurance robotic aerosonde for use in cold
    regions. Typically, aircraft of this type are not designed with
    anti-icing capabilities in order to maintain their weight and
    endurance characteristics. In August 2000 two weeks of flights were
    conducted to test new instruments, ice detectors and anti-icing
    coatings. The tests were highly successful and will continue with
    more miniature instruments, an upgraded aerosonde design and a new
    catapult launch device.

-   An interesting example of research focused on Life in Extreme
    Environments is the discovery of bacteria actively metabolizing at
    --17^o^C in snow at the South Pole. This unexpected result reduces
    the lower temperature limit for life. This evidence for the
    resilience of life exposed to heavy doses of damaging ultra-violet
    radiation, extreme cold, and darkness has important implications for
    the possibility of life existing elsewhere in the Solar System.

-   **[New Types of Ocean Bacteria that Convert Light Into
    Energy]{.smallcaps}** A new strain of bacteria was recently
    discovered that employs bacteriorhodopsin, a protein that
    demonstrates light-harnessing abilities previously known only to
    exist in fungi and archaea. These organisms thrive in hostile
    environments where sustenance is scarce---like the open ocean.
    NSF-supported researchers found a bacteria that uses a type of
    chlorophyll never before seen in open ocean bacteria. Two teams
    isolated DNA directly from seawater samples and compared many of the
    fragments to those already on file in public databases. The gene
    that stood out was that which codes for bacteriorhodopsin. They
    developed a fluorometer to search for bacteria that might use dim
    infrared light emitted from deep sea vents. They found none at the
    vents, but when scanning the surface waters they picked up many
    positive signals. The types of bacteria discovered by these two
    teams employ different mechanisms to harness light and convert it
    into energy. What they have in common is the ability to thrive in
    the open ocean where nutrients are limited and in turn provide
    nourishment for other organisms higher up the food chain.

-   NSF-supported collaborators have developed a new biosensor based on
    fiber optic technology that can directly monitor microbial community
    structure and activities in coastal and estuarine waters and
    sediments. The biosensor design will allow multiple assays in the
    future. Such a sensor should lead to new knowledge regarding
    biogeochemical processes in these environments including the
    \"relationships between cell abundance, gene expression, and actual
    microbial metabolic activities under different environmental
    conditions\".

-   Invaluable insights into the evolution of vertebrates have been
    provided by NSF- supported groups working to understand important
    evolutionary steps in the transition from invertebrates to
    vertebrates. Using comparisons of gene expression, and model
    vertebrates such as mice and zebrafish, these labs have provided
    important insights into the development of the brain, spinal cord,
    and neural crest. In addition, the researchers have been able to
    develop methods to perturb gene expression in non-model systems and
    have paved the way for important future studies by all researchers
    in the field.

-   **[New Nanowires]{.smallcaps}** Nanoporous templates were created by
    exploiting the self-assembling nanoscale structure of cylindrical
    phase diblock-copolymers. Functional materials are deposited into
    the nanopores to create the final functional nanostructure. This
    fast and easy-to-use method was used successfully to fabricate an
    array of magnetic cobalt \'pillars\' (or nanowires) of very high
    density. These may find use in new magnetic hard disks that can have
    200 times the storage capacity of present commercially available
    disks.

## FY 2000 Examples of Achievements - 

## Outcome Goal 2

### Connections between discoveries and their use in service to society

External evaluators cited the following examples of results from NSF
awards as demonstrating success in support of Outcome Goal 2. These
examples made the connections between discoveries and their use in
society and were rapidly and readily available and used as appropriate
in education, policy development, or by other federal agencies or the
private sector.

-   **[Fire-Weather]{.smallcaps}** Ongoing wildfire research supported
    by NSF is directed at improving the understanding of fire behavior
    and dynamics. This research involves collaboration with a growing
    number of colleagues including the U.S. Forest Service, Monash
    University, University of Colorado, the Country Fire Authority (CFA)
    of Victoria, and Australia\'s Northern Territories Bushier Council.
    The groups combine models, instrument development, and observations
    of forest fires and grass fires in field experiments conducted in
    the Northwest Territories of Canada, Australia, and the western
    United States to gain a better understanding of fire behavior and
    spread. This new knowledge helps the fire captain to better position
    his firefighters and equipment to fight a fire. The increased
    understanding of fire dynamics can help prevent firefighters from
    being overrun by fire and can save lives.

-   **[Lighter-Weight Electronics]{.smallcaps}** Consumer demand for
    smaller, more-reliable, lighter-weight electronic devices is a
    strong contributor to our nation's economic growth. One center
    located at Georgia Institute of Technology is helping to meet that
    demand. Flip-chip microelectronic packaging, the technology of
    attaching semiconductor chips directly to circuit cards, can deliver
    the needed functionality. However, costly manufacturing processes
    and materials have hindered wide-scale proliferation of flip-chip
    technology. The center has created a suite of next-generation
    flip-chip manufacturing processes, advanced materials, and
    manufacturing equipment in cooperation with their 20 industrial
    partners. These new methods could reduce flip-chip costs by 50-80
    percent. The new materials are now commercially available from
    organizations such as Dexter, National Starch & Chemical, Emerson
    and Cuming, Loctite, and Alpha Metals. Manufacturing equipment for
    using these new materials is available from Cookson and Siemens.
    Seagate has already integrated some of the new technologies,
    materials, and equipment into their manufacturing lines. Large
    industrial organizations, whose products span the spectrum of
    commercial and military electronic devices, such as Advanced Micro
    Devices, Chrysler, Ericsson, Honeywell, Texas Instruments, Nokia,
    Lucent Technologies, and Northrop Grumman now are poised to
    implement flip-chip technologies into their next-generation of
    products and systems.

-   **[Cracks Along the Continental Shelf]{.smallcaps}** NSF-supported
    researchers discovered cracks along the edge of the continental
    shelf off the coast of Southern Virginia. The researchers suggest
    that the cracks could be the start of underwater landslides that
    could, in turn, create tsunamis. The investigators received
    additional support to carry out a detailed geological and
    geophysical investigation of these features. Initially thought to be
    caused by faults. The cracks appear to be depressions formed by
    continuous and massive blowouts of gas. The researchers maintain
    that such blowouts could trigger landslides and tsunamis and note
    that similar gas blowouts have damaged or destroyed oil rigs in the
    Gulf of Mexico and the North Sea. The implications of these findings
    are important for identifying geo-hazards on the east coast of the
    United States.

-   **[In the area of design and manufacturing]{.smallcaps}**
    NSF-supported researchers have been very successful in disseminating
    their research results to the private sector, where they have become
    useful tools. For example, in the areas of rapid prototyping and
    rapid fabrication, there have been spin-offs significant for
    commercial ventures. NSF research resulted in patented technology
    licensed to DTM Corporation. Subsequently, DTM developed and
    manufactured the Sinterstation 2500^®^ System. The Sinterstation
    2500^®^ is a commercially successful rapid prototyping technology
    used in US European university and government research laboratories,
    including Sandia and Los Alamos National Laboratories and the Jet
    Propulsion Laboratory, and in several large corporations, including
    The Gillette Company. A 3D Printing technology was licensed to
    several companies, all of whom have used the technology to
    manufacture a wide variety of products. Several large companies have
    options for licenses in place. Both of these technologies have led
    to significant reductions in the time from design to manufacture. A
    further effect of this reduction in lead time is the potential to
    diminish the need to maintain large spare-parts inventories,
    particularly in Department of Defense applications.

-   **[NSF-supported projects on the NSFNET]{.smallcaps}** developed new
    techniques in cross vendor route registry and its management. The
    major activities include advancement of Internet routing algorithms
    with respect to scaling and stability issues, routing information
    registration and dissemination for the network service providers
    serving the Internet, deployment of route servers to aid in the
    dissemination and real time maintenance of the global Internet
    routing system, and coordination and sharing of technical
    information in support of the Internet operation community. Several
    leading vendors have licensed routing algorithms developed by this
    project.

-   **[Global Sharing of Science Results]{.smallcaps}** E-print
    Archives, the electronic-print system, allows physicists to post the
    results of their research in a timely manner on the web, allowing
    other scientists to have rapid access to their work as important
    discoveries are made.

-   [**Improving efficiency of Network Design to save time and
    money**]{.smallcaps} Solving problems in local access and network
    layout are fundamental to optimizing computation. NSF-supported work
    addresses some fundamental issues in the area of network design with
    practical relevance since small improvements in network layout can
    translate into savings of many millions of dollars. The
    NSF-supported research has resulted in solutions and associated
    software packages capable of solving network design
    orders-of-magnitude faster on available computers. The work applies
    methods from operations research to networking problems, and has
    already received recognition from the operations research community.
    It will have substantial impact on network service providers.

NSF awards produced a wide variety of important discoveries in both the
Arctic and Antarctic. Many discoveries concern regional environmental
change which have implications for global climate change.

-   Greenland ice core studies have produced evidence that rapid climate
    change -- 8 degrees C in less than a decade -- has occurred.

-   Studies have shown that the sea ice cover of the Arctic Ocean has
    undergone a major decrease in the past decade as a result of melting
    and redistribution due to atmospheric circulation changes.

Protection of the environment and human health in a context of continual
development and economic growth is perhaps the most significant global
challenge for the next millennium. Issues such as sustainability,
pollutant avoidance, and remediation drive the direction of many NSF
investments. NSF programs continue to participate substantially in the
NSF/EPA initiative, "Technology for a Sustainable Environment". Other
notable NSF-supported programs include studies on environmentally-benign
processing, development of environmentally-safe products, destruction
methods for pollutants, and diagnostics of known carcinogens. Recent
fundamental breakthroughs under NSF support in clean chemistry,
diagnostics, and new concept development show the promise of
minimizing/controlling emissions and improving our quality of life.

-   NSF supported researchers have demonstrated the use of high pressure
    carbon dioxide (super critical conditions) as a solvent for cleaning
    clothes, computer parts, and textiles. This technology has the
    potential for replacing organic and halo-carbon solvents. They are
    the principal pollutants in the cleaning industry. The NSF-
    supported fundamental science has led to the creation of a small
    business and a center of study at North Carolina universities.

-   Simultaneous reduction of nitric and nitrous oxide (NOx) and soot
    emissions from practical combustion systems to meet tight emission
    standards is an extreme challenge due to competing
    formation/destruction processes. NSF-supported researchers have
    developed a catalytic filter for diesel engines that captures soot
    particulates formed during low temperature operation and then
    oxidizes this soot to reduce NOx emissions under high temperature
    conditions.

-   **[Impact on the Development and Manufacture of Specialty Chemicals,
    Novel Materials and Bio-based Agents]{.smallcaps}** Our ever-rising
    standard of living is accompanied by an ever-increasing demand for
    better and more affordable specialty chemicals, novel materials, and
    chemical/biological therapeutic agents. The production of
    these chemicals and materials requires highly sophisticated, and
    often prohibitively expensive, reaction processes and/or separation
    and purification steps. Advances made with support from NSF have
    greatly contributed to the technological developments in these
    areas.

```{=html}
<!-- -->
```
-   NSF-supported development of a low-cost technique to use immobilized
    affinity chromatography to separate valuable blood proteins such as
    protein C, prothrombin and vitamin K-dependent proteins from plasma
    was developed. These natural plasma proteins are much more effective
    than those derived from currently available recombinant DNA
    technology. This work is closely coordinated with and done in
    collaboration with the American Red Cross.

```{=html}
<!-- -->
```
-   NSF awards frequently result in application software, patents, and
    an educational exchange between academic researchers and industry
    engineers. Researchers learn about the domain and adapt their
    research agendas to industrial needs. Industry engineers have access
    to recent research results and course materials and, as the
    relationship progresses, access to a pool of potential employees
    skilled in the domain. A few examples that highlight these
    connections are:

```{=html}
<!-- -->
```
-   NSF engineers in partnership with apparel manufacturer Levi-Strauss,
    have developed and implemented an adaptive, closed-loop production
    control system. Thanks to a donation of equipment from a factory
    being closed by Levi-Strauss, a production line identical to an
    operating line at another factory was installed in the engineer's
    laboratory space, allowing a rigorous comparison of the traditional
    and proposed lines. The research has helped the manufacturer to
    supply quantities of a large mix of products rapidly and
    economically while maintaining minimal surge capacity to meet
    occasional peak demands.

-   A tricycle "cobot" for removing doors from newly painted automobiles
    prior to assembly of the cabin has been developed with NSF support.
    On the basis of this research prototype, General Motors developed a
    rugged, highly maneuverable device for this task. The cobot is
    patented. Various educational courses were provided to GM engineers.

***Science and Technology Centers (STCs):*** Integrative Partnerships
have responsibilities in the area of knowledge transfer that go beyond
those of individual investigators by design. The first new set of STC
awards in almost a decade were made in FY 2000. Assessments address both
the effectiveness of existing centers in establishing connections and
the potential for strong performance in the new class.

-   One Science and Technology Center is dedicated to exploiting the
    atmospheric conditions at the South Pole to do astronomy and
    cosmology. The Center also has an active program of education and
    outreach. The core program, Space Explorers, targets
    African-American students at inner-city high school students in
    Chicago to enhance their science abilities before they enter
    college. Each August, thirty students attend a week-long summer
    residential institute at Yerkes Observatory. A 2-day version of the
    summer institute is given each December. The Space Explorers teach
    in grammar schools during the academic year. Space Explorers and
    Adler Planetarium astronomers present programs using a portable
    planetarium to 30 schools, reaching nearly 3,000 students, annually.

-   NSF-supported research in cognitive science has focused on the
    cognitive aspects of language acquisition, structure and processing,
    logic and computation, and perception and action. Recent
    accomplishments include progress on the integration of research on
    language acquisition with statistical learning theory, simulations
    of language learning, and online methods to study children's
    language acquisition.

## FY 2000 Examples of Achievements - 

## Outcome Goal 3

### A Diverse, Globally-oriented Workforce of Scientists and Engineers

External evaluators cited the following examples of results from NSF
awards as demonstrating success in support of Outcome Goal 3. Noteworthy
examples taken from committee reports have been selected to demonstrate
results in FY 2000 areas of emphasis that include integrative research
and education opportunities, and participation of under-represented
groups in integrative research and education.

-   **[Living in a Global World]{.smallcaps}** NSF programs are at the
    forefront of study of the emerging area of globalization. Increased
    economic, technological, and social interconnections among nations
    have resulted in new transnational, international, and supranational
    legal practices. Our increasingly global world calls for continued
    development and innovation of legal theories. Several NSF funded
    projects have already made significant findings on new forms of
    global law and global legal processes. Studies have examined and
    explored

```{=html}
<!-- -->
```
-   the quality of the practice of law internationally by looking at
    lawyers' roles in social movements and human rights campaigns;

-   international networks of lawyers to understand how the practice of
    law and experience of practitioners have changed in response to
    global processes;

-   the new culture of finance, which draws on innovative methodology
    such as monitoring an internet discussion group.

```{=html}
<!-- -->
```
-   **[Increasing Students' Exposure to Quantitative Data and Empirical
    Analysis]{.smallcaps}** The principle purpose of the American
    National Election Study (ANES), has been the development of
    high-quality data on public opinion and political choice in American
    national elections. With a history of support spanning 25 years, the
    data generated by this enterprise increasingly is used in college
    and even in high school texts, not only to inform students about
    elections and voting, but also to teach them the rudiments of data
    analysis and statistics. Because American government is a required
    course at the college level in many state universities, all
    students, not just those interested in Political Science, are
    exposed to quantitative data and empirical analysis. Frequently ANES
    data on diskette are now included in introductory American
    government textbooks, along with basic statistical software, and
    lesson plans include suggestions for the effective integration of
    these data in courses.

-   NSF support at the University of Hawaii provides widely distributed
    educational kits that undergraduate students use to effectively
    learn how to design low cost, complex systems. The kits consist of a
    prototyping card, with network capability, in a book with companion
    materials. There is a strong team involving researchers at the
    University of Hawaii, several industrial organizations, and a
    publisher. The kit is in accordance with the emphasis on design in
    the new Accreditation Board for Engineering and Technology (ABET)
    accreditation requirements and is expected to make a significant
    impact in undergraduate education in engineering and science.

-   NSF support to a researcher at the University of Maine has led to
    the development of an outreach program that supplements research on
    geospatial databases. Work covered by this project is part of a high
    school outreach program called "Spatial Horizons". The program is
    typically attended by more than 100 high school students per year.
    Another high school outreach program affected by this project is the
    University of Maine organized "Expanding your Horizons" program. It
    is targeted at female high school students attempting to increase
    their participation in science and engineering disciplines. The lead
    researcher on this project won the 1998 Outstanding Young Faculty
    Research Award at the University of Maine and the 1999 Presidential
    Citation of the American Society for Photogrammetry & Remote
    Sensing.

-   **[Developing an environment for the workforce]{.smallcaps}** NSF
    support to Mississippi State University has led to the development
    of methodologies for enabling industry to better accommodate
    disabled employees. Specifically this work targets persons with
    paraplegic and/or visual disabilities. It should lead to the
    development of an intelligent computer system to assist in making
    decisions associated with designing and retrofitting work tasks and
    the workplace to accommodate persons with disabilities. It may also
    address some of the major problems associated with the high
    unemployment rate of persons with disabilities.

-   **[Research Experiences for Undergraduates (REU)]{.smallcaps}** One
    mechanism NSF uses to increase support for under-represented groups
    is the REU site award. Participation in REU sites frequently
    includes high percentages of under-represented groups. For example,
    one NSF organization has invested in over 200 REU sites. One NSF
    division provides resources for 60 REU sites in 33 states and Puerto
    Rico, with about 550 students supported at these sites. Typically,
    half the participants are female, and over 16% are from
    under-represented minority groups.

-   With funding from some 24 NSF-supported centers, about 370
    undergraduates participated in research activities. The
    undergraduates included 145 females and more than 100 minority
    students. Outreach to Native Americans is a focus of two of the
    centers.

-   NSF administers the four-year undergraduate and graduate program
    Significant Opportunities in Atmospheric Research and Science
    (***SOARS***) to provide education and research opportunities in the
    atmospheric sciences to students from under-represented groups.
    Typically, about 20 students worked with scientific mentors from NSF
    supported labs, DOE and NASA laboratories, the University of
    Colorado, and other national and international universities.

-   An NSF program funds a collaboration between the University of
    Pittsburgh and the Carnegie Museum of Natural History, in which
    minority high school students serve as interns, working on web-site
    development, exhibit and seminar development, and as museum docents.
    All of this is in connection with Earth science museum exhibits. The
    internships target African-American students interested in math and
    science in the Pittsburgh area.

## FY 2000 Examples of Achievements - 

## Outcome Goal 4

### Improved achievement in mathematics and science skills needed by all Americans

External evaluators cited the following examples of results from NSF as
demonstrating success for Outcome Goal 4. Noteworthy examples taken from
committee reports have been selected to demonstrate results in FY 2000
areas of emphasis, which include K-12 systemic activities; research on
learning and education; graduate teaching fellows in K-12 education; and
K-16 digital libraries.

NSF considers many of the K-12/16 activities listed to be of interest to
students in order to engage them at an early stage in their education in
science, mathematics and computer science. Early involvement is
extremely important to retaining students in science and engineering in
the future. Educating in science is educating for the future.

-   Research on learning has provided important findings for middle and
    high school. At Rutgers University a longitudinal study of the
    development of proof-making in students has found that students at
    the middle school and high school level are capable of much more
    advanced mathematical thinking than expected. For example, although
    high school students did not use the symbolic representation or
    procedures of college students studying calculus, they developed
    powerful and correct solutions to calculus-type problems.

-   Hampshire College found that students enrolled in inquiry-based
    classes performed better on essay style assessments of general
    scientific reasoning in comparison to students in more traditional
    classes.

-   NSF supports a focused project on "at-risk" students at the Kieffer
    Institute for Development of Science-based Education, entitled
    "Science: Day-by-Day, Life-by-Life, Community-by-Community\". The
    goal of the project is to formulate an Earth-science-based
    curriculum for K-12 education for at-risk students. At-risk students
    are defined as any group of students who are not able to participate
    in a continuous K-12 curriculum. The curriculum uses the Earth
    sciences to capture students\' interests and to stimulate learning
    in other fields such as mathematics, social sciences, and language
    arts.

-   Pre-service and in-service science teachers work together in
    NSF-supported research projects at Towson State. This teaming of
    experienced teachers with teachers-in-training facilitates the
    acquisition of important new expertise, the development of a deeper
    understanding of research, and the unique opportunity to share
    invaluable experience. By targeting both active and developing
    science teachers, the benefits of this experience will be rapidly
    and widely distributed to K-12 students.

-   Critical to the goal of science education for all Americans is the
    development of creative ways of reaching the public. Support for
    museum collections promotes this goal in a number of ways. Museum
    displays and the computerization of collections and distribution of
    data over the World Wide Web provide broad accessibility to the
    American public. Equipment supported by NSF and placed in museums
    often becomes the focus of education-related activities. Museums are
    also involved in web-based science and education programs that are
    directly related to supported collections.

-   NSF-supported researchers at the University of Massachusetts are
    studying ways to improve the abilities of K-12 students to find,
    evaluate, and organize information available on the Internet. These
    skills comprise a significant subset of the Information Literacy
    skills that Library Science teaches. The approach consists of
    building a Web search interface in which Information Literacy skills
    are matched to Information Retrieval (IR) tools in a way that
    teaches skills while helping students locate information on the
    Internet. Improved queries are created from the student\'s
    information need, supporting information from the surrounding
    educational environment, and query expansion from
    educationally-focused databases. Information filtering techniques
    identify, and if desired eliminate, retrieved information at the
    wrong grade-level or containing inappropriate content. One of the
    goals is to establish a long-term research relationship to address
    the use of information technology and the Internet in K-12
    education.

-   NSF supported collaborators at Carnegie Mellon have supported
    development of a computer based reading tutor for elementary school
    students. An early evaluation showed rapid improvement in reading
    skills among poor readers.

-   **[Quarknet]{.smallcaps}** partners high school physics teachers and
    their students with particle physics research groups at 60 U.S.
    universities and laboratories. Students learn fundamental physics,
    investigate particle physics through live, online data and
    collaborate with other students worldwide. About 25 teachers
    complete summer research appointments and these teachers go on to
    offer workshops to another hundred teachers. Each teacher who has
    skills and knowledge enhanced in such research experiences then
    communicates that knowledge and excitement to all of the students in
    his or her classroom.

-   Recent results from research on the learning of science and
    mathematics have shown that elementary school children are capable
    of more sophisticated forms of reasoning, modeling, and higher order
    learning that previously thought or that are currently embedded in
    teaching materials and teaching practice:

```{=html}
<!-- -->
```
-   Homeless students and Latino students for whom English is a second
    language, or whose command of English is still limited, do learn to
    high national standards when properly taught.

-   Fourth and fifth graders can talk appropriately about sampling and
    distributions and how these ideas can help explain the growth of
    organisms and populations of organisms.

-   Research projects constructed by elementary school students reveal
    understanding of experimental controls and extraneous variables even
    at the first grade level.

```{=html}
<!-- -->
```
-   NSF has actively supported research in Digital Libraries since 1994.
    The research has made major strides in developing techniques to
    advance digital technologies for searching, indexing and storing
    objects beyond traditional text. For example:

```{=html}
<!-- -->
```
-   The e-skeletons project enables students to study comparative
    anatomy of humans and baboons (gorillas are coming soon!). With low
    cost 3-D printing equipment schools can supplement high resolution
    images with 3-D copies of bones that otherwise would be unavailable
    for most schools and colleges. (See http://www.eskeletons.org)

-   Digital library research in the humanities for the Perseus Digital
    Library extends access to a wide range of unique museum materials to
    students and scholars. A timely presentation on the Greek Olympics
    showed vases and also referenced text material.

```{=html}
<!-- -->
```
-   School children, with the help of staff at the University of
    Colorado, are able to sample current and fossil remains of a
    particular gastropod throughout its historical range. They do this
    in order to explain outstanding problems in predator/prey
    relationships through geologic history.

-   Several hundred volunteer observers, ranging in age from 6 to 80,
    have been trained to make rain and hail observations across the
    state of Colorado. Rain and hail patterns are mapped daily,
    disseminated to students, businesses, government and scientists, and
    used by the National Weather Service, the local mosquito control
    program, and community water conservation programs.

-   The Lawrence Hall of Science has developed Student Radon Research
    Kits, that contain all of the equipment necessary for secondary
    school students to conduct radon and meteorological research at
    school sites.

-   UCLA geoscientists have developed "Geoscience Interactive
    Simulations for Teaching (GIST),\" student-controlled numerical
    simulations of Sun-Earth interactions.

-   The IRIS Consortium has developed a museum display and educational
    materials that bring research quality seismograph data to the public
    to help understand earthquakes and the role they play in shaping our
    dynamic Earth.

-   **[The *WorldWatcher* Curriculum: Integrating Visualization into
    Inquiry-Based Science Learning]{.smallcaps}** explores use of
    cutting-edge scientific visualization as a teaching tool in middle
    school and high school classrooms. This effort has received an A+ by
    *Education World*. The *WorldWatcher* website includes revolutionary
    and downloadable scientific visualization environment software.
    Students engage in inquiry-based learning, exploring, creating, and
    analyzing scientific data.

-   [**The** **Alternatives for Rebuilding Curricula**]{.smallcaps}
    (ARC) is a center promoting awareness and effective use of the
    elementary mathematics curricula: *Math Trailblazers*,
    *Investigations in Number, Data, and Space*, and *Everyday
    Mathematics.* The release of these curricula is relatively recent,
    but they are making determined inroads to the market. In 1998, these
    NSF-supported programs were used in about 3% of the nation's school
    districts; adoptions and large pilot tests have nearly doubled each
    year. Currently, the curricula are being used by almost 2,600,000
    students in about 11% of the 14,000 school systems in the country.

# XV. Schedule of Program Evaluations

The following table provides information on the scheduling of meetings
for Committees of Visitors (COVs) for FY 1999, FY 2000, and FY 2001.
Note that the schedule and program titles shown are actual for FY 1999,
FY 2000, and tentative for FY 2001. A COV may not be scheduled in this
3-year period of the program was new in FY 2000. Programs that are new
in FY 2001 will not appear on this list. Committee of Visitor reports
will be made available to the public electronically in December, 2001.
NSF anticipates that COV reports will become electronically available in
July, of subsequent years.

### Committee-of-Visitor Schedule for FY 1999-2000-2001

###### [Division (Program)]{.underline} FY FY FY 

######  1999 2000 2001

###### Biological Infrastructure 

Instrument Related Activities X

Research Resources X X

Training X X

Plant Genome X

###### Environmental Biology 

Ecological Studies X

Systematic and Population Biology X X

###### Integrative Biology and Neuroscience 

Neuroscience X

Developmental Mechanisms X

Physiology and Ethology

###### Molecular and Cellular Biosciences

Genetics X

Biomolecular Structure and Function X

Cell Biology X

Biomolecular Processes X

###### Advanced Computational Infrastructure and Research

Advanced Computational Research X

Partnerships for Advanced Computational Infrastructure (PACI)

###### Computer -- Communications Research

Theory of Computing X

Computer Systems Architecture X

Numeric, Symbolic, & Geometric Computation X

Software Engineering and Languages X

Operating Systems and Compilers X

Design Automation X

Communications X

Signal Processing Systems X

### COV Schedule for FY 1999-2000-2001

###### [Division (Program)]{.underline} FY FY FY 

######  1999 2000 2001

###### Advanced Networking Infrastructure and Research

Advanced Networking Infrastructure X

Special Projects in Networking Research X

Networking Research X

###### Information and Intelligent Systems

Computational and Social Systems X

Information and Data Management X

Robotics and Human Augmentation X

Human Computer Interaction X

Knowledge and Cognitive Systems X

###### Experimental and Integrative Activities

###### *Infrastructure*

Minority Institutions Infrastructure X

Advanced Distributed Resources for Experiments X

Instrumentation Grants for Research X X

Research Infrastructure X

###### *Research*

Experimental Partnerships X

Digital Government X

Next Generation Software X

###### *Education*

CISE Educational Innovation X

###### Education System Reform

Statewide Systemic Initiatives X

Urban Systemic Initiatives X X

Rural Systemic Initiatives X

Urban Systemic Program

**EPSCoR** X

###### Elementary, Secondary, & Informal Education

Informal Sci Education X

Teacher Enchancement X

Presidential Awards for Excellence in Mathematics X

Instructional Materials Development

###### Undergraduate Education

Course, Curriculum, and Laboratory Improvement X

Teacher Preparation X

Advanced Technological Education X

### COV Schedule FY 1999-2000-2001 (cont.)

###### [Division (Program)]{.underline} FY FY FY 

######  1999 2000 2001

###### Graduate Education

Graduate Research Fellowships X

NATO Postdoc Fellowships X

IGERT

PFSMETE

GK-12 Fellows

###### Human Resource Development

The Louis Stokes Alliances for Minority Participation X

CREST X

Program for Gender Equity in SMET X

Programs for Persons with Disabilities X

Alliances for Grad. Ed. and the Professoriate (AGEP) X

HBCU X

###### Research, Evaluation & Communications

REPP/ROLE

Evaluation X

###### Bioengineering and Environmental Systems

Biochemical Engineering X

Biotechnology X

Biomedical Engineering X

Research to Aid the Disabled X

Environmental Engineering

Environmental Technology

Ocean Systems

###### Civil and Mechanical Systems

Dynamic System Modeling, Sensing, and Control X X

Geotechnical and GeoHazard Systems X X

Infrastructure and Information Systems X X

Solid Mechanics and Materials Engineering X X

Structural Systems and Engineering X X

Network for Earthquake Engineering Simulation X

###### Chemical and Transport Systems

Chemical Reaction Processes X

Interfacial, Transport, & Separation Processes X

Fluid and Particle Processes X

Thermal Systems X

### COV Schedule for FY 1999-2000-2001 (cont.)

###### [Division (Program)]{.underline} FY FY FY 

######  1999 2000 2001

###### Design, Manufacture & Industrial Innovation

Operations Research & Production Systems X

Design and Integration Engineering X

Manufacturing Processes & Equipment X\
Innovation and Organizational Change X

Grant Opportunities for Academic Liason w/Industry X

SBIR X

Small Business Technology Transfer X X

### 

###### Electrical & Communication Systems

Electronics, Photonics, and Device Technologies X

Control, Networks, and Computational Intelligence X

###### Engineering Education & Centers

Engineering Education X

Human Resource Development X

Engineering Research Centers X

Earthquake Engineering Research Centers X

Industry/Univ Coop Res Centers X

Combined Research-Curriculum Development X

Supplement Support for Women, Minorities and Phys. Disabled X

**Geo-wide Education & Human Resources** X

###### Atmospheric Sciences

Atmospheric Chemistry X

Climate Dynamics X

Meoscale Dynamic Meteorology X

Large-Scale Dynamic Meteorology X

Physical Meteorology X

Paleoclimate X

Magnetospheric Physics X

Aeronomy X

Solar Terrestrial Research X

Upper Atmospheric Research Facilities X

NCAR/UCAR X

Lower Atmospheric Observing Facilities X

UNIDATA X

### COV Schedule FY 1999-2000-2001 (cont.)

# 

###### [Division (Program)]{.underline} FY FY FY 

######  1999 2000 2001

###### Earth Sciences

Instrumentation and Facilities X

Tectonics

Geology and Paleontology

Hydrological Sciences

Petrology and Geochemistry

Continental Dynamics

Geophysics

###### Ocean Sciences

Oceanographic Technical Services

Ship Operations

Oceanographic Facilities

Ocean Drilling

Marine Geology and Geophysics X

Biological Oceanography X

Chemical Oceanography X

Physical Oceanography X

Oceanographic Tech. & Interdisciplinary Coordination X

###### Astronomical Sciences

Extragalactic Astronomy and Cosmology X

Advanced Technologies & Instrumentation X

Planetary Astronomy X

Stellar Astronomy & Astrophysics X

Electromagnetic Spectrum Management X

Galactic Astronomy X

Education, Human Resources & Special Programs X

National Astronomy & Ionosphere Center (NAIC) X

National Optical Astronomy Observatories (NOAO) X

National Radio Astronomy Observatory (NRAO) X

Gemini 8-Meter Telescopes X

University Radio Facilities X

#  

### COV Schedule FY 1999-2000-2001 (cont.)

###### [Division (Program)]{.underline} FY FY FY 

**1999 2000 2001**

###### Chemistry

Analytical & Surface Chemistry X

Chemistry Research Instrumentation & Facilities (CRIF) X

Inorg, Bioirnorg, & Organometallic Chemistry X

Organic & Macromolecular Chemistry X

Experimental Physical Chemistry X

Office of Special Projects X

Organic Chemical Dynamics X

Organic Synthesis X

Theoretical and Computational Chemistry X

Chemistry of Materials X

###### Materials Research

Condensed Matter Physics X

Materials Theory X

Metals, Ceramics, & Elect. Materials X

Solid-state Chemistry and Polymers X

National Facilities & Instrumentation X

Materials Research Sci. & Eng. Centers X

###### Mathematical Sciences

Statistics and Probability X

Geometric Analysis X

Algebra & Number Theory X

Applied Mathematics X

Infrastructure X

Analysis X

Computational Mathematics X

Topology and Foundations X

###### Physics

Atomic, Molecular, Optical and Plasma Physics X

Elementary Particle Physics X

Gravitational Physics X

Nuclear Physics X

Theoretical Physics X

Particle and Nuclear Astrophysics

Education and Interdisciplinary Research

Major Research Instrumentation X

Science and Technology Centers

### COV Schedule for FY 1999-2000-2001 (cont.)

###### [Division (Program)]{.underline} FY FY FY 

**1999 2000 2001**

###### Polar Research Support X

###### Antarctic Sciences

Antarctic Aeronomy and Astrophysics X

Antarctic Biology and Medicine X

Antarctic Geology and Geophysics X

Antarctic Ocean and Climate Systems X

Antarctic Glaciology X

###### Arctic Sciences

Arctic System Sciences X

Arctic Social Sciences X

Arctic Natural Sciences X

Arctic Research Support and Logistics X

###### International Programs X

###### Social, Behavioral, and Economic Research

Economics X

Decision, Risk, and Mgt. Sciences X

Innovation and Organizational Change X

Cultural Anthropology X

Physical Anthropology X

Archeology & Archaeometry X

Geography & Regional Science X

######  Sociology X

Political Science X

Law & Social Issues X

Methodology, Measurement & Statistics X

Society Dimensions of Engineering, Science, and Technology X

Science, Technology, and Society X

Linguistics X

Human Cognition & Perception X

Social Psychology X

# 

# XVI. Table of Acronyms

# 

## Acronym Definition

ABET Accreditation Board for Engineering and Technology

AC Advisory Committee

ACM Association for Computing Machinery

ACSI American Customer Satisfaction Index

AGI Arabidopsis Genome Initiative

ARPANET Advanced Research Projects Agency Network

ATM Atmospheric Sciences

BART Bay Area Rapid Transit

BBC British Broadcasting Corporation

BGE Baltimore Gas and Electric?

BIO Biological Sciences

CARA Center for Astrophysics Research in Antarctica

CAREER Faculty Early Career Development Program

CCR Computer-Communications Research

CETP Collaboratives for Excellence in Teacher Preparation

CFA County Fire Authority

CHE Division of Chemistry

CIA Central Intelligence Agency

CISE Computer and Information Science and Engineering

CMB Cosmic Microwave Background

CMU Carnegie Mellon University

COV Committee of Visitors

CREST Centers of Research Excellence in Science and Technology

CRS Congressional Research Service

CS&E Computer Science and Engineering

CSNET Computer Science Network

CTS Chemical and Transport Systems

DARPA Defense Advanced Research Projects Agency

DBI Biological Infrastructure

DMII Design, Manufacture, & Industrial Innovation

DMP Distributed Mentor Project

DOE Department of Energy

ECS Electrical and Communications Systems\
EEC Engineering Education and Centers

EHR Education and Human Resources

EIA Experimental and Integrative Activities\
ENG Engineering

EPA Environmental Protection Agency

EPSCoR Experimental Program to Stimulate Competitive Research

ERC Engineering Research Centers

ESIE Elementary, Secondary and Informal Education

ESR Educational System Reform

FBI Federal Bureau of Investigation

FCSM Federal Committee of Statistical Methodology

FEC Forward Error Correction

FFRDC Federally-Funded Research and Development Center

FY Fiscal Year

GAO General Accounting Office

GEO Geosciences

GIST Geoscience Interactive Simulations for Teaching

GPA Grade Point Average

GPRA Government Performance and Results Act (of 1993)

GRT Graduate Research Traineeships

HBCU Historically Black Colleges and Universities

HHS Health and Human Services

HRD Human Resource Development

I/UCRC Industry University Cooperative Research Centers

IBM International Business Machines

IBN Integrative Biology and Neuroscience

IEEE Institute of Electrical and Electronics Engineers\
IG Inspector General

IGERT Integrative Graduate Education and Research Training

IIS Information and Intelligent Systems

IR Information Retrieval

IRIS Information, Robotics, and Intelligence Systems

ITR Information Technology Research

K-12 Kindergarten through twelfth grade

K-16 Kindergarten through college

KMPG LLC (accounting firm)

LEARN Laboratory Experience in Atmospheric Research

MCB Molecular and Cellular Bioscience

MIT Massachusetts Institute of Technology

MPS Mathematical and Physical Sciences

MRI Major Research Instrumentation Program

NAEP National Assessment of Educational Process

NAPA National Academy of Public Administration

NASA National Aeronautics and Space Administration

NCAR National Center for Atmospheric Research

NEC Nippon Electric Corporation?

NIST National Institute for Science and Technologies

NOAA National Oceanic and Atmospheric Administration

NSB National Science Board

NSF National Science Foundation

NSFNET National Science Foundation Network

OCE Ocean Sciences

OIA Office of Integrative Activities

OIG Office of the Inspector General

OLPA Office of Legislative and Public Affairs

OMB Office of Management and Budget

ONR Office of Naval Research

OPP Office of Polar Programs

PACI Partnerships for Advanced Computational Infrastructure

PAT Program Announcement Template

PFSMETE Postdoctoral Fellowships in Science, Mathematics, Engineering

> and Technology Education

PI Principal Investigator

POL SCI Political Sciences

PRS Project Reporting System

PwC PricewaterhouseCoopers

R&D Research and Development

REU Research Experiences for Undergraduates\
RUI Research in Undergraduate Institutions

S&E Salary and Expenses

SBE Social, Behavioral, and Economic Sciences

SBIR Small Business Innovation Research Program

SGER Small Grants for Exploratory Research

SHEBA Surface Heat Budget of the Arctic Ocean Project

SMET Science, Mathematics, Engineering and Technology

SOARS Significant Opportunities in Atmospheric Research and Science

SOC Sociology

SRC Semiconductor Research Corporation

SRS Science Resource Studies

SSI Statewide Systemic Initiative Program

STC Science and Technology Center

TAAS Texas Assessment of Academic Skills

TCP Transmission Control Protocol

TCP/IP Transmission Control Protocol/Internet Protocol

TE Teacher Enhancement

UCAR University Corporation for Atmospheric Research

UCLA University of California at Los Angeles

UCSC University of California at Santa Cruz

UCSD University of California at San Diego

UNIDATA UNIDATA (previously University Data Systems)

USGS U.S. Geological Survey

USI Urban Systemic Initiative

VBNS Very High Speed Backbone Network Service

Y2K Year 2000

[^1]: ^1^*Funding a Revolution - Government Support for Computing
    Research*, Computer Science Telecommunications Board of the National
    Research Council 1999, National Academy Press, 286 pages. This
    extended study has a detailed history of the ARPANET and Internet
    and places the development in a broader context of networking
    research and development.

[^2]: ^2^ In several instances, a single COV report evaluates more than
    one program.

[^3]: ^4^ Additional examples may be found in Appendix XIV.

[^4]: ^5^ Additional examples may be found in Appendix XIV.

[^5]: ^6^ Additional examples may be found in Appendix XIV.

[^6]: ^7^ Additional examples may be found in Appendix XIV.