**FY 2001 GPRA**

**PERFORMANCE REPORT**

**March 29, 2002**

## 

## 

## ![](media/image1.wmf)**NATIONAL SCIENCE FOUNDATION ACT OF 1950**

**"AN ACT to promote the progress of science; to advance the national
health, prosperity, and welfare; to secure the national defense; and for
other purposes."**

## ***Enabling the Nation's future through*** 

## ***discovery, learning and innovation.***

> **[Realizing the promise of the 21^st^ century depends in large
> measure on today's investments]{.smallcaps}**
>
> **[in science, engineering and]{.smallcaps}**
>
> **[mathematics research and education.]{.smallcaps}**
>
> **[NSF investments -- in people, in their ideas, and in the tools they
> use -- will catalyze the strong progress in science and engineering
> needed to secure the Nation's future.]{.smallcaps}**

## **Table of Contents**

PAGE

**Overview**

Table of Contents i

Message from the Director ii

Executive Summary iv

I.  About NSF 1

II. Some NSF Achievements 5

III. Summary of Performance Results 11

**Supporting Information for FY 2001 GPRA Reporting**

Performance Reporting Requirements 22

IV. NSF Outcome Goals 24

A. People 24

Indicators and Examples 25

Areas of Emphasis and Examples 34

B. Ideas 41

Indicators and Examples 42

Areas of Emphasis and Examples 51

C. Tools 58

Indicators and Examples 60

Areas of Emphasis and Examples 66

V.  Management Goals 71

A. Business Practices 72

B. NSF Staff 76

VI. NSF Investment Process Goals 80

A. Merit Review 81

B. Customer Service 88

C. Awards 92

D. Broadening Participation 97

E. Facilities 99

VII. Assessment and Evaluation Process 108

VIII. Verification and Validation (V&V) 112

IX. Transition from FY 2000 to FY 2001 115

X.  Transition from FY 2001 to FY 2002 119

XI. Other Features 123

XII. Management Challenges 125

**Appendices**

Appendix I. Table of External Evaluations 137

Appendix II. Schedule of Program Evaluations 148

Appendix III. Table of Acronyms 156

Appendix IV. PricewaterhouseCoopers LLP Executive Summary 159

> **Message from the Director**

## 

![](media/image2.jpeg){width="1.5763888888888888in"
height="2.045138888888889in"}*I am pleased to present the National
Science Foundation's Annual Performance Report for Fiscal Year 2001, as
required by the Government Performance and Results Act of 1993 (GPRA).*

*That law requires federal agencies to set clear goals and to explain
what they have done to meet them. Again this year, NSF demonstrates a
record of solid progress toward some of the most formidable challenges a
government organization has ever set for itself. These include raising
nationwide achievement in science and engineering, achieving processing
electronically all proposals, and ensuring that America has a globally
competitive and diverse workforce.*

*In the aftermath of the terrorist attacks on September 11, the stakes
for our investments could not be higher. The future of America -- indeed
the future of the world -- is more dependent upon advances in science
and technology than ever before. An inspired science and engineering
community is focused on ensuring not just our security, but also our
very quality of life. We well remember that our national security
includes the condition of our spirit as much as the size of our arsenal,
and we are heartened by the echo of President Franklin D. Roosevelt's
words in his secret letter to Robert Oppenheimer in 1943: "Whatever the
enemy may be planning, American science will be equal to the
challenge."*

*We continue to meet this challenge today as we pursue our principal
objectives -- enabling discoveries at the frontiers of science and
engineering, and encouraging significant improvement in the quality and
scope of math, science and engineering education. Once again, the entire
world acknowledged the fruits of those efforts. Of the 11 winners of
2001 Nobel Prizes in the sciences, eight had been funded by NSF grants.*

*Across the science and engineering frontiers, research progresses in
ways that may culminate in discoveries this year, next year or in the
next decade. This intrinsic uncertainty makes it more difficult for NSF
to assess its overall achievements on a year-to-year basis. Nonetheless,
the GPRA goals constitute a valuable set of benchmarks, and NSF is proud
of its FY 2001 results.*

*The Foundation has set extremely high standards for success. This is
what our tradition of excellence demands, and what taxpayers deserve.
This year, we succeeded in realizing 15 of our 23 goals, and for most of
the goals we did not meet, we demonstrated measurable improvement over
last year's results. You will find a full description of our performance
results in this report, results that draw upon data that are complete
and reliable. Because we are also committed to the highest standards of
management integrity, we will continue to improve our data collection
efforts and management processes.*

*NSF remains committed to the highest standards of management
effectiveness and administrative integrity as we seek to provide science
and engineering outcomes of major significance and widespread benefit to
the nation. We look forward to an even more successful year in FY 2002.*

![](media/image3.png){width="2.2291666666666665in"
height="0.6666666666666666in"}

*Rita R. Colwell,*

*Director*

***EXECUTIVE SUMMARY***

This report, prepared pursuant to the Government Performance and Results
Act (1993), covers activities of the National Science Foundation during
Fiscal Year 2001.

The National Science Foundation (NSF) Act of 1950 charges NSF with
supporting scientific and engineering research and education programs at
all levels. Over time, other responsibilities have been added such as
supporting the development and use of computers and other technologies,
providing Antarctic research facilities and logistics support, and
addressing issues of equal opportunity in science and engineering
research and education.

NSF represents about four percent of the total federal budget for
research and development, but accounts for one-fifth of all federal
support for basic research and 40% of non-life science basic research at
academic institutions. About 95% of our funding supports the work of the
nation's researchers and educators; the agency's administrative overhead
is only five percent.

NSF goals are divided into three broad areas: Strategic Outcome Goals,
Management Goals, and Investment Process Goals.

## **Outcome Goals:** Our outcome goals focus on PEOPLE, IDEAS, and TOOLS and concern the practical, concrete, long-term results of NSF grants and programs. They represent what we seek to accomplish with the investments we make in science and engineering research and education.

**Management Goals:** Our management goals relate to the effectiveness
and efficiency of our activities.

**Investment Process Goals:** Our investment process goals relate to the
procedures we use to make awards, fund and manage capital projects, and
otherwise serve our customers.

**FY 2001 Results:** We met 15 of our 23 goals. Foundation staff
verified and validated all NSF performance data. In addition,
PricewaterhouseCoopers LLP, an independent examiner engaged by NSF,
verified and validated selected performance information and data.

*[Outcome Goals:]{.underline}* We were successful for four of our five
outcome goals (80%) related to:

-   Developing "a diverse, internationally competitive and
    globally-engaged workforce of scientists, engineers, and
    well-prepared citizens" (PEOPLE);

-   Providing intensive professional development programs for at least
    65,000 K-12 teachers (PEOPLE);

-   Enabling "discovery across the frontier of science and engineering,
    connected to learning, innovation and service to society" (IDEAS);
    and,

-   Providing "broadly accessible, state-of-the-art and shared research
    and education tools." (TOOLS)

We were not successful in achieving the goal involving systemic reform
in K-12 schools. While we accomplished two of the three indicators
required for successful attainment of this goal, we did not accomplish
the third.

Examples of accomplishments achieved during this reporting period:

-   New planets were discovered in orbit around far-off stars, in a
    quest to find planetary systems similar to Earth.

-   Computer scientists created what may be the most intrusion-proof
    system invented so far. It has successfully repelled more than
    13,000 hacker attacks from around the world.

-   Researchers designed the first autonomous vehicle to work under the
    Arctic ice.

-   Robotics advanced in numerous ways, with robot scouts used to search
    for victims at the World Trade Center site.

-   The "Deep Green" project revealed major discoveries about the
    history of plant life on Earth.

-   Climate researchers uncovered an apparent self-regulating feedback
    relationship between sea-surface temperature and cloud formation
    that could prompt drastic changes in the way scientists model
    climate.

-   The [Macrogalleria](http://www.psrc.usm.edu/macrog/index.htm), a
    pioneering educational polymer web site, is continuing to garner
    broad recognition. The worldwide popularity is so high it has
    already been translated into Afrikaans, French, and Spanish, and is
    being translated into Italian and Portuguese.

-   *Sounds of the Sea* is an NSF informal science education project
    that serves diverse students and engages diverse communities
    directly in the scientific enterprise*.* It created a national model
    for engaging blind and visually impaired students and adults in
    experiencing hands-on science.

-   The new Terascale Computing System (TCS) has begun operation well
    ahead of schedule and is exceeding performance. The combined peak
    power of the full computer system will be 6 Teraflops, making it the
    most powerful computer available to academic scientists and
    engineers in the U.S.

-   A Lightning Mapping Array (LMA) has been developed that provides
    added lead-time for use in forecasting the onset of lightning
    strokes hitting the ground (one of the deadliest weather phenomena
    occurring today).

Additional examples of accomplishments for each of the outcome goals are
provided within the body of the report. They represent only a small
fraction of the results identified by external experts.

*[Management Goals:]{.underline}* We were successful for four of our
five goals (80%) in this area: We were able to:

-   Ensure that at least 95% of full proposals are submitted
    electronically through the computer-based "FastLane" system. In
    fact, more than 99% were submitted electronically;

-   Increase the total number of science and engineering hires at NSF
    from under-represented groups, as judged against an FY 1997
    baseline. NSF achieved a 138% increase in female hires and a 47%
    increase in minority hires;

-   Increase usage of a broad range of video-conferencing/long distance
    communications technology by 100 percent over the FY 1999 level. The
    increase achieved was 184%; and

-   Meet the goal of having the technological capability to move
    competitive proposals submitted electronically through the entire
    review process without generating paperwork. A pilot project
    involving 10 programs was successful.

We were not successful in meeting a management goal related to
distributing a survey to help establish baselines to enable us to better
assess the quality of the work environment. A survey will be
administered in FY 2002.

*[Investment Process Goals:]{.underline}* We achieved 7 of 13 (54%) of
these goals. These are:

-   Allocating at least 85% of funds to projects reviewed by external
    peer groups and selected through merit-based competition.

-   Ensuring that 95% of program announcements and solicitations are
    available at least three months prior to proposal deadlines. We
    achieved 100%.

-   Increasing our average annualized award size for research projects
    to \$110,000. We exceeded our goal, achieving an average annualized
    award size of \$113,000.

-   Developing the capability and implementing electronic approaches
    that request voluntary demographic data from all reviewers to
    determine participation levels of members of underrepresented groups
    in the NSF reviewer pool.

-   Ensuring that NSF Program Officers address both generic review
    criteria when making award decisions.

-   Keeping annual construction and upgrades expenditures at 90% of our
    facilities within 110% of estimates. Ninety-six percent of the
    projects were within 110%.

-   Keeping total cost of construction and upgrade projects initiated
    after 1996 within 110% of estimates made at the initiation of
    construction. One project was completed and its actual cost was
    equal to the total estimated cost.

We were not successful for six of our investment process goals. These
were:

-   Awarding at least 30% of competitive research grants to new
    investigators. We awarded 28%, and will continue our efforts in
    reaching out to new investigators to promote awareness of funding
    opportunities and to encourage new investigators to submit
    proposals.

-   Ensuring that external merit reviewers take *both* NSF criteria into
    account when evaluating proposals. The two generic criteria are
    intrinsic merit and broader impacts of the proposed activity. We
    have taken several steps, including revising the Grant Proposal
    Guide, to ensure that both the proposer and reviewer communities are
    aware of the importance we attach to both criteria.

-   Processing 70% of our proposals processed within six months of
    receipt. Although we did not achieve our goal, we showed significant
    improvement, rising from 54% in FY 2000 to 62% in FY 2001. This
    improvement took place even though the number of proposals received
    in FY 2001 represented an 8.5% increase over FY 2000 and was the
    largest annual percentage increase in over a decade.

-   Increasing the average duration of awards for research projects to
    at least three years. Sufficient resources were not available to
    achieve both the average annualized award size and the average
    duration goals. We focused on increasing our average annualized
    award size. We will continue to focus on increasing both award size
    and duration.

-   Having 90% of our facilities meet all annual schedule milestones by
    the end of the reporting period. Of 25 construction and upgrade
    projects supported by NSF, 21 (or 84%) met this goal (compared with
    64% achieving the goal in FY 2000).

-   Holding operating time lost due to unscheduled downtime at 90% of
    NSF-funded facilities to less than 10% of total scheduled operating
    time. Of 29 reporting facilities, 25 (86%) met the goal

**Management Challenges:** The NSF Office of the Inspector General
listed 10 major management challenges for FY 2001:

-   FastLane

-   GPRA Data Quality

-   Merit Review

-   Cost Sharing

-   Award Administration

-   Management of Large Infrastructure Projects

-   Management of U.S. Antarctic Program

-   Work Force Planning and Training

-   Fostering a Diverse Scientific Workforce

-   Data Security

Our responses and focused NSF activities in these areas are provided
within the report.

######### I. ABOUT NSF

######### 

**Who we are**

On May 10, 1950, President Harry S. Truman signed Public Law 810-507,
creating NSF and setting forth our mission:

***\"To promote the progress of science; to advance the national health,
prosperity, and welfare; to secure the national defense; and for other
purposes.\"***

Our authorizing legislation directs us to initiate and support basic
scientific and engineering research; to support programs to strengthen
scientific and engineering research potential; to support education
programs at all levels in all fields of science and engineering research
and education, and to establish an information base for science and
engineering appropriate for development of national and international
policy. Since the passage of that legislation 50 years ago, we have
endeavored to maintain American leadership in scientific and engineering
discovery, learning and innovation.

In contrast to other federal agencies that have research objectives such
as energy, biomedicine, or space, we stand alone as the only federal
agency charged with supporting and strengthening all disciplines across
the science and engineering frontier. The Internet, plant genomics,
nanotechnology and biocomplexity are but a handful of examples of
NSF-supported research outcomes that have revolutionized, or have
promise to revolutionize, how we live, work, and play.

America's science and engineering enterprise is unparalleled in scope
and quality and has enabled the United States to become one of the most
productive nations in the world. The return on investments in science
and engineering has been enormous and has directly contributed to the
nation's economic growth and to the health and welfare of its people. It
is estimated that as much as one-half of the nation's economic
productivity can be attributed to technological innovation and the
science and engineering that supports it. Science and technology have
contributed to an increased standard of living in most of the world's
modern industrial societies, and have had enormous impact on health
care, agriculture, environmental protection and national defense.

**What we do**

Our role is to fund the best ideas and most capable people exploring
science, mathematics, and engineering research and education. We award
grants, contracts and cooperative agreements to approximately 2,000
colleges, universities, schools, academic consortia, nonprofit
institutions and small businesses throughout the United States. We also
maintain partnerships with international organizations around the world.
Investments promote the emergence of new disciplines, fields, and
technologies that enable and enhance our nation's capacity for sustained
growth and prosperity.

While our budget accounts for only about four percent of the total
federal expenditure on research, we provide about one-fifth of the
federal support to academic institutions for basic research. Each year
our programs involve nearly 200,000 scientists, engineers,
mathematicians, teachers and students.

**How we do it**

We receive about 30,000 proposals a year from the science and
engineering community. In determining which of these proposals to invest
in, we use external experts to advise us on the merit of the proposed
activities, and how they compare to other proposals. Only about one in
three proposals is selected for award. We consider the merit review
process critical to our efforts to foster the highest standards of
excellence and accountability. Each year, thousands of experts volunteer
their time to evaluate proposals sent to NSF. We ask them to use two
criteria in evaluating proposals---the intellectual merit of the
proposed activity *and* its broader impacts.

**What we fund**

We play a unique role in the federal investment portfolio in that in our
funded activities we integrate research and education activities. We
support individual investigators and small groups engaged in research
and education in traditional fields at about 2,000 colleges and
universities,

K-12 school districts, academic consortia, nonprofit institutions, small
businesses and other research and education institutions throughout the
nation. We provide support for U.S. participation in international

state-of-the art research facilities such as the National Astronomy
Centers, oceanographic research ships, and Antarctic research stations.
Research facilities provide scientists, mathematicians and engineers
access to state-of-the-art capabilities that enable research and
education at the cutting-edge. We support research centers that address
complex scientific and engineering questions through multi-disciplinary,
long-term, coordinated efforts of many researchers and educators.

Our education and training investments support work at all levels, from
pre-kindergarten through career development, across the U.S. These
activities promote public science literacy and help to ensure that our
nation maintains world-class scientists, engineers and mathematicians.
We focus on programs that encourage the participation and achievement of
groups underrepresented in science and engineering. We emphasize K-12
education through the support of partnerships that unite local school
districts, colleges and universities and other stakeholders such as
state and tribal entities. We believe that treating whole systems is the
most effective way to make improvements in science and mathematics
education.

**How we are organized**

The President appoints, with the consent of the U.S. Senate, the NSF
Director to serve a six-year term. Our current director, Dr. Rita R.
Colwell, became NSF's eleventh director in 1998.

The National Science Board (NSB) establishes our policies. The Board
consists of 24 members representing a cross-section of American
leadership in science and engineering research and education.
Presidentially-appointed NSB members are selected solely on the basis of
established records of distinguished accomplishments. They serve
six-year terms, with one-third of the Board's membership appointed and
approved every two years. The NSF Director is a member *ex officio* of
the Board. The NSB also serves the President and the Congress as an
independent advisory body on policies affecting the health of U.S.
science and engineering research and education.

NSF is structured much like an academic institution, with divisions
organized by disciplines and fields of science and engineering, and for
science, mathematics, engineering, and technology education. We have
seven directorates, an Office of Polar Programs and two management
offices.

**Some statistics**

![](media/image4.wmf){width="4.875in" height="3.936111111111111in"}In FY
2001, approximately 93% of our \$4.5 billion budget supported research
and education activities carried out by awardees. These programs and
activities directly engaged nearly 200,000 people[^1], including
researchers, educators, students, and other professionals. Approximately
three percent (\$119 million) of the budget was devoted to major
research equipment and construction. The remaining five percent was
devoted to conducting the administrative work of the agency.

We employ a scientific and engineering staff of approximately 600
permanent and visiting scientists and engineers (approximately 65% of
the agency's scientists and engineers are permanent government
employees), 450 business and operations personnel, and 350 program
support personnel. Our staff manage our investment portfolio: they do
not conduct research nor do they operate laboratories supported by NSF
awards. In FY 2001 staff processed more than 200,000 merit-based reviews

and made funding decisions on nearly 32,000 competitive proposals. About
10,000 new awards were made. During 2001, about 45,000 reviewers were
sent one or more proposals for mail review and about 10,000 reviewers
served as panelists. About 9,000 of these reviewers had never reviewed
an NSF proposal before.

> \*R&RA = Research and Related Activities
>
> EHR = Education and Human Resources
>
> MRE = Major Research Equipment
>
> S&E = Salaries and Expenses
>
> OIG = Office of Inspector General

######### II. SOME NSF ACHIEVEMENTS

**NOBEL PRIZES FOR 2001:** Of the eleven 2001 Nobel Prize winners in the
sciences, eight[^2] have been previously funded by NSF
([http://www.nobel.se](http://www.nobel.se/)).

**The 2001 Nobel Prize in Chemistry** was awarded to William S. Knowles,
Ryoji Noyori and K. Barry Sharpless, for their development of catalytic
techniques for asymmetric chemical synthesis. In nature, many molecules
are found in mirror image, right-handed or left-handed, forms. Often
only one of these asymmetric forms is biologically active. Sharpless
developed highly efficient catalytic synthetic techniques to selectively
produce only one of these mirror image forms. These techniques have
allowed pharmaceutical companies to synthesize only the mirror image
form that they need. The catalytic techniques developed as a result of
this discovery are now used by pharmaceutical companies to produce, for
example, beta-blocker medication to control blood pressure.

**The 2001 Nobel Prize in Economics** was awarded to three NSF-supported
economists, George A. Akerlof, A. Michael Spence, and Joseph E.
Stiglitz, for their fundamental contributions to our understanding of
asymmetric markets -- markets in which one side has more information
than the other. More recent research by economists following up on the
earlier theoretical work has shown that job candidates can signal to
prospective employers their motivation, ability and training through the
wage their current employer is willing to pay or the type of contract
they will accept. \"Signaling theory\" has been widely applied, most

notably to understanding the Internet and e-commerce.

**The 2001 Nobel Prize in Physics** was awarded to three researchers --
Eric A. Cornell, Wolfgang Ketterle, and Carl E. Wieman -- for their
achievement of Bose-Einstein condensation in dilute gases of alkali
atoms and for early fundamental studies of the properties of the
condensates. The Bose-Einstein Condensate is regarded as a new state of
matter in which all the constituents, by virtue of their near-absolute
zero low temperature, are in the same quantum state. For this new state
of matter the corresponding atom waves are coherent. Current speculation
suggests that this new level of "control" of matter is going to bring
revolutionary applications in such fields as precision measurement and
nanotechnology.

Louis Stokes Alliances for Minority Participation: A variety of current
initiatives are directed toward broadening participation of
underrepresented minorities in science, technology, engineering, and
mathematics. A principal one, the Louis Stokes Alliances for Minority
Participation (LSAMP), shows increasingly positive results with growing
numbers of students in this program entering these fields and a higher
percentage earning degrees, including advanced degrees, in these
disciplines. Particularly noteworthy are the increased graduation rates
(from 48% to 62% in science and from 53% to 76% in engineering over a
five-year period) at Puerto Rico LSAMP institutions.

**Aging:** A new model system was developed to study telomeres,
structures that seal the ends of chromosomes in plants and animals much
like the plastic tips on shoelaces, and which wear out, allowing the
\"lace\" to fray. Telomeres break down in most cells in the human body
over time and have been implicated in aging. By exploiting the completed
genome sequence of *Arabidopsis*, it is possible to uncover the
contributions of the DNA damage surveillance machinery in identifying
dysfunctional telomeres
(<http://www.nsf.gov/od/lpa/news/press/01/pr0115.htm>).

**Deep Green:** The \"Deep Green\" project has made radical new
discoveries about the history of plant life on earth. The findings
significantly rearranged the \"family tree\" of green plants since it
was learned that ferns and horsetail are not, as was previously
believed, transitional between mosses and flowering plants. They are, in
fact, the closest living relatives to seed plants. Ramifications of
these findings span practical areas ranging from agriculture to
economics (<http://www.nsf.gov/od/lpa/news/press/01/pr0109.htm>).

**Extra-solar planets:** A major impetus to the observational and
theoretical studies of the formation of stars and their planetary disks
was provided in the last few years by the discovery of extra-solar
planets. The most recent discovery finds a planet three-quarters the
mass of Jupiter in a circular orbit around the solar-like star 47 Ursa
Majoris. Although more than 70 extra-solar planets have been found thus
far, this is the first system with two planets in circular orbits, and
at distances that make the planetary system similar to our own.

**The oldest material:** Reports of the Hadean detrital zircons from
Western Australia include the discovery of one that is approximately 4.4
billion years old. This tiny crystal formed within 160 million years of
the formation of Earth and is the only sample known from the earliest
history of Earth. The zircons also indicate that there was liquid water,
and perhaps even oceans, i.e. a cool early earth, at a time when many
have hypothesized the existence of magma oceans.

**Clouds and climate models:** One of the most uncertain aspects of
climate models is the treatment of clouds. Recent work has led to a
discovery with potentially significant implications for global climate
change scenarios. An analysis of the distribution of upper-level cirrus
clouds and sea surface temperatures (SSTs) shows a strong inverse
relation between cirrus cloud area and the underlying surface
temperatures. The finding suggests that the earth has a natural adaptive
infrared 'iris' that opens and closes the upper dry regions in order to
control the heat radiated by the Earth in response to SST changes, in a
manner similar to how a human iris reacts to changing light levels. If
confirmed, this could significantly affect current global climate
models.

**Intelligent robots used at World Trade Center:** Intelligent
Autonomous Marsupial Robots, prototyped with NSF funds, were used for
search and rescue at the World Trade Center (WTC) during the disaster
recovery efforts following the September 11, 2001 attack. Just as
kangaroos carry their young in a pouch, these "marsupial" robots possess
unique characteristics: the "mother" robot carries smaller ones in its
"pouch" into the site as far as it can maneuver and it then releases and
provides power as the "babies" descend from it to perform their search
-- negotiating smaller crevices and hidden spaces. Equipped to maintain
balance on rough terrain, the robot "mother" and its "children" can
reach, sense, and report on spaces that are too small and/or too
dangerous for human rescue workers to approach or enter. The robots
located five victims and a set of remains, and surveyed three buildings
and two voids in the debris.

As a result of watching the robotic creatures in action, the Federal
Emergency Management Agency (FEMA) Task Force has ordered various small
and semi-autonomous robots for future use
(<http://www.nsf.gov/od/lpa/news/press/01/pr0178.htm>).

**Newly discovered animals in hydrothermal vents:** In the spring of
2001, an interdisciplinary team of scientists and engineers explored a
newly discovered hydrothermal vent field in the Indian Ocean. They
collected biological samples, samples of vent and smoker fluid and
plumes, rocks and sediment samples from the seafloor, and precisely
mapped the area. Newly discovered animals living in the hydrothermal
vent system as well as ancient bacteria found at the site may help
scientists better explain how and whether the fauna living at
hydrothermal vents in the Atlantic and Pacific Oceans are genetically
related. The research expedition was fully integrated with an
educational component entitled \"Dive and Discover,\" co-funded with the
Woods Hole Oceanographic Institution and Ohio\'s Center of Science and
Industry. \"Dive and Discover\" involved live webcasts, interactions
between students and researchers, and companion materials that assisted
teachers in explaining the science and technology behind the expedition
(<http://www.nsf.gov/od/lpa/news/press/01/pr0136.htm>).

**Learning technologies:** Just as human teachers need to listen as well
as speak, some researchers believe that learning technologies should
observe and react to students\' needs, not just present material in an
informative and appealing way. Researchers supported by NSF are opening
up new avenues of education research. Here are several examples:

\- A series of participatory simulations has been developed by which
middle and high school students, using handheld and networked devices,
each become independent agents in simulations of system dynamics. These
simulations may be as varied as chemical molecule collisions or
automotive traffic. They all yield entirely different kinds of
content-rich experiences.

\- Tracking devices that examine students\' eye movements as they
interact with a computerized algebra tutor have already proven effective
as an instructional agent. Eye movements may be indicative of changing
cognitive states on a fine time scale. When the tutor\'s feedback was
contingent on eye movement data, students learned 20% faster
(preliminary unpublished results).

**Wisdom of practice:** Findings from NSF supported research on learning
are directly applicable to the classroom and help to identify the most
effective types of interventions or instructional approaches. A commonly
held \"wisdom of practice\" is that one of the best means to learn a
subject is to teach it. Research has documented this phenomenon and has
used it to design agent-based technology by which mathematics and
science students have to master disciplinary content at a sufficient
level to \"teach\" computer agents. Programs were developed that enable
analysis not only of progress in writing proficiency but also in
evolution of content knowledge. Research of this type supports the
conclusion that children learn better if they are put into the position
of reconstructing or explaining newly learned scientific material to
others. Work in the Detroit Public Schools suggests that youngsters as
young as sixth grade can build and analyze models of complex systems
that without technology would require use of undergraduate mathematics.

**Underwater robots:** The first autonomous underwater vehicle with the
endurance to work under ice in the Arctic was developed. This is part of
a continuing effort to obtain better all-season ocean data.
Strategically related efforts continue to develop robotic samplers for
the atmosphere (aerosondes) and to develop an autonomous, under-ice
ocean bottom seismometer.

**Modeling contamination events:** Newly developed computer models
enable modeling of the flow of water in shallow basins. These models
include factors such as river inflow and standard tidal flow, and allow
for the presence of contaminants being transported by the flow. These
computational methods have been used in the development of a complex
shallow water simulator, called UTBEST (University of Texas Bays and
Estuaries Model). It has also been possible to model a simulated
contamination event in the Houston Ship Channel with the domain modeled
being all of Galveston Bay.

**Cyber-terrorism:** A computer system was built that is capable of
identifying and stopping intrusive behavior on the system it is
protecting. The technology can identify assaults in progress, stop the
offending process, and disable the IP address of the culprit. As a proof
of concept, the team that developed the system placed a highly
vulnerable version of the Linux system (no security patches) on the net
as a web server and invited hackers throughout the world to root the
machine. There were over 13,000 attacks on this box and not one
successful intrusion.

**Lost City:** Hydrothermal vent structures, dubbed the "Lost City,"
were discovered in December 2000 in the mid-Atlantic Ocean. These vent
structures, including a massive 18-story vent, taller by far than any
seen before, are very different from all others discovered across the
world since the 1970s
(<http://www.nsf.gov/od/lpa/news/press/01/pr0156.htm>).

**Antifreeze proteins in fish living in extreme environments:** The
question of how cold-blooded creatures such as fish survive in the
frigid waters off Antarctica is being investigated by studying the role
of antifreeze glycopeptides, antifreeze proteins of Antarctic fishes
involved in freezing avoidance. We anticipate that the results will lead
to major advances in understanding molecular biology and evolution of
antifreeze systems and will be applicable to a wide range of
disciplines.

Researchers on another project, also focused on Antarctic fishes, seek
to determine at the molecular level those adaptations that enhance the
assembly and movement of microtubules and the expression of related
genes. In the broadest sense, this research program should advance the
molecular understanding of the survival of cold-blooded organisms.

Another project is a phylogenetic study of Antarctic microorganisms to
understand the unique adaptations required for survival. The application
of DNA microarray technology to studies of life in extreme environments
offers an outstanding opportunity for identifying new genes for
biotechnological use. Discovering specific adaptations to extreme
environments by detecting genes that are uniquely expressed in the
natural environment is an ultimate goal of this research.

**Sounds of the Sea:** An NSF informal science education project that
serves diverse students and engages diverse communities directly in the
scientific enterprise is *Sounds of the Sea.* This project empowers a
generation of students and teachers to explore scientific concepts and
engage in understanding an array of scientific professions that are not
a traditional part of the K-12 experience. It created a national model
for engaging blind and visually impaired students and adults in
experiencing hands-on science. The program provided curricula to urban
school districts that will reach 15,000 teachers annually through the
Teacher Resource Center. The program will remain in use beyond the grant
period. A legacy of the project is a four-page "advanced organizer," as
well as in-home, in-school, and after-school activity guides distributed
to over 10,000 educators to facilitate students' exploration in marine
science.

**Replacing chlorine-based processes:** Catalysts for the activation of
hydrogen peroxide in water for green oxidation processes have been
developed. They may replace environmentally harmful chlorine-based
processes used in the textile, paper and laundry industries.

**Astroflow:** The activities of faculty members who are active in
bringing their research results to the public are well illustrated by
work taking place in Rochester, NY. A research group is making the
results of their research directly available to the public through a
variety of venues including programs at local grade, middle and high
schools. A program called Astroflow is a suite of software tools
allowing users to interactively control, visualize, and explore
realistic simulations of cosmic events: exploding stars, comets diving
into planetary atmospheres, jets of hypersonic gas driving through
interstellar clouds, and more. Astroflow gives students and
non-scientists the opportunity to learn by experimentation and
exploration. Currently Astroflow is installed in a specially designed
kiosk at the Strasenburgh Planetarium in Rochester, NY. The technology
behind Astroflow has been successfully commercialized.

The [Macrogalleria](http://www.psrc.usm.edu/macrog/index.htm), a
pioneering educational polymer web site, is continuing to garner broad
recognition. NSF Director Rita Colwell described it in these terms in
one of her speeches: "Many of you have seen the wonderful Web Site
called 'Macrogalleria.' It is set up like a shopping mall. The site
bills itself as 'the Internet mall where you net surfers can learn all
kinds of nifty stuff about polymers and polymer science.' The student or
the Internet surfer clicks on the shops and learns that polymers are
everywhere. As he or she ascends to the different levels of the mall,
more complex concepts are conveyed." The Macrogalleria was recently
selected by [***Scientific
American***](http://www.scientificamerican.com/explorations/2001/051401top50/#Scientific American)[^3]
as one of its "50 Top Websites." This follows many other distinctions,
such as the Education Index Top Site and the Top 5% of Chemistry Sites.
The worldwide popularity of the Macrogalleria is so high that it has
already been translated into Afrikaans, French, and Spanish, and is
being translated into Italian and Portuguese.

**VORTEX -- Predicting tornadoes:** The benefits of education and
research within the atmospheric sciences are extremely visible to the
public at large. The importance of the daily weather to individuals is
self-evident. Severe weather prediction is of paramount importance. The
Verification of Origins of Rotation in Tornadoes Experiment (VORTEX)
used airborne and surface mobile Doppler radars and other mobile sensors
to map in detail the structure of tornadoes and their near environment.
The modeling and observations indicated that a localized downdraft is an
important ingredient in the final stages of the formation of a tornado.
This knowledge was directly translated to more accurate tornado
forecasts (tornado watches) being issued by the National Weather
Service.

**Hazard loss:** A major research project on "2^nd^ National Assessment
of Research and Applications on Natural Hazards" involved over 130
national experts in hazards and disasters from all fields of science,
engineering, policy, and practice. The results of this effort greatly
influenced policies for and approach to hazard loss mitigation in the
Federal Emergency Management Agency's (FEMA) Project Impact. The
objectives of Project Impact are to establish a national risk
assessment, meet the need for computer-aided systems to inform local
hazard decision-making \[e.g., FEMA's Natural Hazard Loss Estimation
Methodology (HAZUS)\], and assist local efforts to design safer
communities, such as efforts underway in Berkeley, California and Tulsa,
Oklahoma. A summary brochure of the assessment was distributed to every
member of the U.S. House of Representatives. The document was
subsequently used internationally. It provided the basis for redrafting
New Zealand's environmental and hazards legislation to link sustainable
development with environmental management and hazards mitigation.

**COPLINK:** A collaboration between a university artificial
intelligence laboratory and a local police department has led to
development of an integrated justice information database available over
a secure intranet. COPLINK is still in the deployment phase with 32 law
enforcement professionals currently using this system. Plans are being
developed to deploy COPLINK in other areas of the U.S. COPLINK is an
excellent example of multi-agency partnerships supported under the
Digital Government program. NSF and the Defense Advanced Research
Projects Agency (DARPA) supported the fundamental research that led to
COPLINK under the Digital Library 1 initiative.

**Astro coalitions:** This project links amateur and professional
astronomers, school children, their teachers, and families around a
variety of activities, such as creation of scientific records,
production of student research journals, and collaborative investigation
of astrophysical phenomena. In addition to having formed partnerships
that connect groups who have never worked together before, this project
has created a system that has engendered new coalitions beyond those
targeted in the funding cycle. The project has launched a new idea for
public collaboration in science that is regenerating itself in a variety
of forms across the nation.

**Wavelets and their uses:** The theory of wavelets has had a profound
impact on data compression, signal analysis, scientific calculation,
medical imaging and radar detection. In one application wavelets are
used to identify the key features of an image and allow reconstruction
of the image with only a tiny amount of information about the original
object.

**Environmental changes in Antarctica:** The Cape Roberts Project is a
major international drilling program involving more than 50 people from
seven countries. Researchers have collected more than 1,700 m of
sediment core that is providing a record of environmental changes
extending back more than 30 million years. These cores reveal periods of
high frequency instabilities in the Antarctic climate during the Miocene
(e.g., 24 million years ago) age. A detailed record of environmental
changes extending back this far has not been available previously; the
new discoveries from the cores are expected to make significant
contributions to many areas of Antarctic geosciences.

#########  {#section-4 .unnumbered}

######### III. SUMMARY OF PERFORMANCE RESULTS

Overall, we were successful in achieving 65% -- 15 of 23 -- of our
performance goals.

##### **Results for Outcome Goals:** This is the first year we are reporting on the new outcome goals of People, Ideas, and Tools. We achieved four of our five outcome goals (80%) in FY 2001. We failed to achieve our goal involving schools participating in systemic initiative programs. 

In assessing our results for People, Ideas, and Tools, external experts
noted significant achievement in diversity and broadening participation.
However, participation of underrepresented groups remains lower than
desired. It was also noted that there was limited involvement of agency
programs other than Science Resources Statistics in the development of
information fundamental to national policy debates. A complete
discussion of our outcome goal results is provided in Section IV.

**Results for Management Goals:** We achieved four of our five
management goals (80%) in FY 2001. We are particularly

pleased to have demonstrated the feasibility of implementing a paperless
proposal review environment at NSF. Ten programs participated in a pilot
program in electronic proposal review in FY 2001, and the pilot program
was a success. The one goal we did not achieve involved distribution of
an employee survey to our staff. Our management goals discussion is in
Section V.

**Results for Investment Process Goals:** We achieved seven of our
thirteen investment process goals in FY 2001. Areas identified as
needing improvement include: the use of the generic merit review
criteria by reviewers, decreasing the time to decision, increasing the
percentage of awards to new investigators, and keeping operating time
lost at facilities due to unscheduled downtime to less than 10% of the
total scheduled operating time. Discussion of these goals and how NSF is
addressing issues is provided in Section VI.

The following Table provides a summary of the results for NSF's FY 2001
GPRA reporting.

ANNUAL PERFORMANCE GOALS FOR NSF'S STRATEGIC OUTCOMES

+--------------+-----------------------------------+------------------+
| **Strategic  | **FY 2001 Annual**                | # Results for    |
| Outcome**    |                                   |                  |
|              | **Performance Goal**              | # National Sci   |
|              |                                   | ence Foundation  |
+--------------+-----------------------------------+------------------+
| **People     | ***[Performance Goal              | New goal in FY   |
| Strategic    | III-1a:]{.underline}***           | 2001             |
| Outcome**    |                                   |                  |
|              | **NSF is successful when, in the  | **FY 2001: NSF   |
| **Outcome    | aggregate, results reported in    | is successful    |
| Goal         | the period demonstrate            | for goal         |
| III-1:**     | significant achievement in one or | III-1a.**        |
| Development  | more of the following             |                  |
| of "a        | indicators:**                     | -   Demonstrated |
| diverse,     |                                   |     significant  |
| int          | -   Improved mathematics,         |     achievement  |
| ernationally |     science, and technology       |                  |
| competitive  |     skills for U.S. students at   | ```{=html}       |
| and          |     the K-12 level, and for       | <!-- -->         |
| glob         |     citizens of all ages, so that | ```              |
| ally-engaged |     they can be competitive in a  | -   Demonstrated |
| workforce of |     technological society.        |     significant  |
| scientists,  |                                   |                  |
| engineers,   | -   A science and technology and  |    achievement\* |
| and          |     instructional workforce that  |                  |
| w            |     reflects America's diversity. | ```{=html}       |
| ell-prepared |                                   | <!-- -->         |
| citizens."   | -   Globally engaged science and  | ```              |
|              |     engineering professionals who | -   Demonstrated |
|              |     are among the best in the     |     significant  |
|              |     world.                        |     achievement  |
|              |                                   |                  |
|              | -   A public that is provided     | ```{=html}       |
|              |     access to the benefits of     | <!-- -->         |
|              |     science and engineering       | ```              |
|              |     research and education.       | -   Demonstrated |
|              |                                   |     significant  |
|              | [FY 2001 Result:]{.underline}     |     achievement  |
|              | Reports prepared by external      |                  |
|              | experts during FY 2001 GPRA       |                  |
|              | reporting provide assessments and |                  |
|              | retrospective examples of         |                  |
|              | NSF-supported projects that       |                  |
|              | document significant achievement. |                  |
|              | \*A number of these assessments   |                  |
|              | were emphatic that NSF must       |                  |
|              | continue and increase its efforts |                  |
|              | related to diversity.             |                  |
+--------------+-----------------------------------+------------------+
|              | ***[Performance Goal              | **FY 1999:** NSF |
|              | III-1b:]{.underline}***           | successful       |
|              |                                   |                  |
|              | Over 80 percent of schools        | **FY 2000:** NSF |
|              | participating in systemic         | successful       |
|              | initiative programs will (1)      |                  |
|              | implement a standard-based        | **FY 2001: NSF   |
|              | curriculum in science and         | is not           |
|              | mathematics; (2) further          | successful for   |
|              | professional development of the   | goal             |
|              | instructional workforce; and (3)  | III-1b\*\*.**    |
|              | improve student achievement on a  |                  |
|              | selected battery of tests, after  |                  |
|              | three years of NSF support.       |                  |
|              |                                   |                  |
|              | [FY 2001 Result\*\*:]{.underline} |                  |
|              | The curriculum, instructional     |                  |
|              | workforce, and improved           |                  |
|              | achievement in science components |                  |
|              | of the goal were successful.      |                  |
|              | However, less than 80% of schools |                  |
|              | met the goal of improved student  |                  |
|              | achievement in mathematics.       |                  |
|              |                                   |                  |
|              | In FY 2002, appropriate technical |                  |
|              | assistance will be provided to    |                  |
|              | schools not meeting the goal.     |                  |
+--------------+-----------------------------------+------------------+
|              | ***[Performance Goal              | **FY 1999:** NSF |
|              | III-1c:]{.underline}***           | successful       |
|              |                                   |                  |
|              | Through systemic initiatives and  | **FY 2000:** NSF |
|              | related teacher enhancement       | successful       |
|              | programs, NSF will provide        |                  |
|              | intensive professional            | **FY 2001: NSF   |
|              | development experiences for at    | is successful    |
|              | least 65,000 pre-college          | for goal         |
|              | teachers.                         | III-1c.**        |
|              |                                   |                  |
|              | [FY 2001 Result]{.underline}: In  |                  |
|              | school year 1999-2000, EHR awards |                  |
|              | provided intensive professional   |                  |
|              | development (60 hours or more) to |                  |
|              | a total of 79,000 teachers,       |                  |
|              | exceeding substantially the GPRA  |                  |
|              | goal of 65,000.                   |                  |
+--------------+-----------------------------------+------------------+

**ANNUAL PERFORMANCE GOALS FOR NSF'S STRATEGIC OUTCOMES**

**(continued)**

+---------------+---------------------------------+-------------------+
| **Strategic   | **FY 2001 Annual**              | # Results for     |
| Outcome**     |                                 |                   |
|               | **Performance Goal**            | # National S      |
|               |                                 | cience Foundation |
+---------------+---------------------------------+-------------------+
| **Ideas       | ***[Performance Goal            | New goal in FY    |
| Strategic     | III-2:]{.underline}***          | 2001              |
| Outcome**     |                                 |                   |
|               | **NSF is successful when, in    | **FY 2001: NSF is |
| **Outcome     | the aggregate, results reported | successful for    |
| Goal III-2:** | in the period demonstrate       | goal III-2.**     |
|               | significant achievement in one  |                   |
| Enabling      | or more of the following        | -   Demonstrated  |
| "discovery    | indicators:**                   |     significant   |
| across the    |                                 |                   |
| frontier of   | -   A robust and growing        | achievement       |
| science and   |     fundamental knowledge base  |                   |
| engineering,  |     that enhances progress in   | -   Demonstrated  |
| connected to  |     all science and engineering |     significant   |
| learning,     |     areas including the science |                   |
| innovation    |     of learning.                | achievement       |
| and service   |                                 |                   |
| to society."  | -   Discoveries that advance    | -   Demonstrated  |
|               |     the frontiers of science,   |     significant   |
|               |     engineering and technology. |                   |
|               |                                 | achievement       |
|               | -   Partnerships connecting     |                   |
|               |     discovery to innovation,    | -   Demonstrated  |
|               |     learning, and societal      |     significant   |
|               |     advancement.                |                   |
|               |                                 | achievement       |
|               | -   Research and education      |                   |
|               |     processes that are          |                   |
|               |     synergistic.                |                   |
|               |                                 |                   |
|               | [FY 2001 Result:]{.underline}   |                   |
|               | Reports prepared by external    |                   |
|               | experts during FY 2001 GPRA     |                   |
|               | reporting provide assessments   |                   |
|               | and retrospective examples of   |                   |
|               | NSF-supported projects that     |                   |
|               | document significant            |                   |
|               | achievement.                    |                   |
+---------------+---------------------------------+-------------------+
| **Tools       | ***[Performance Goal            | New goal in FY    |
| Strategic     | III-3:]{.underline}***          | 2001              |
| Outcome**     |                                 |                   |
|               | **NSF is successful when, in    | **FY 2001: NSF is |
| **Outcome     | the aggregate, results reported | successful for    |
| Goal III-3:** | in the period demonstrate       | goal III-3.**     |
|               | significant achievement in one  |                   |
| Providing     | or more of the following        | -   Demonstrated  |
| "broadly      | indicators:**                   |     significant   |
| accessible,   |                                 |                   |
| state-of-the  | -   Shared use platforms,       | achievement       |
| art and       |     facilities, instruments,    |                   |
| shared        |     and databases that enable   | -   Demonstrated  |
| research and  |     discovery and enhance the   |     significant   |
| education     |     productivity and            |                   |
| tools."       |     effectiveness of the        | achievement       |
|               |     science and engineering     |                   |
|               |     workforce.                  | -   Demonstrated  |
|               |                                 |     significant   |
|               | -   Networking and connectivity |                   |
|               |     that take full advantage of | achievement\*     |
|               |     the Internet and make SMET  |                   |
|               |     information available to    |                   |
|               |     all citizens.               |                   |
|               |                                 |                   |
|               | -   Information and policy      |                   |
|               |     analyses that contribute to |                   |
|               |     the effective use of        |                   |
|               |     science and engineering     |                   |
|               |     resources.                  |                   |
|               |                                 |                   |
|               | [FY 2001 Result:]{.underline}   |                   |
|               | Reports prepared by external    |                   |
|               | experts during FY 2001 GPRA     |                   |
|               | reporting provide assessments   |                   |
|               | and retrospective examples of   |                   |
|               | NSF-supported projects that     |                   |
|               | document significant            |                   |
|               | achievement. \*There are very   |                   |
|               | limited contributions and       |                   |
|               | limited involvement of agency   |                   |
|               | programs other than Science     |                   |
|               | Resources Statistics (SRS) in   |                   |
|               | developing information and      |                   |
|               | other materials fundamental to  |                   |
|               | national policy debates.        |                   |
+---------------+---------------------------------+-------------------+

**ANNUAL PERFORMANCE GOALS FOR NSF'S MANAGEMENT**

+-----------+---------------------------------------+------------------+
| **Pe      | **FY 2001 Annual**                    | **Results for**  |
| rformance |                                       |                  |
| Area**    | **Performance Goal**                  | **National       |
|           |                                       | Science          |
|           |                                       | Foundation**     |
+-----------+---------------------------------------+------------------+
| **NSF     |                                       |                  |
| Business  |                                       |                  |
| Pr        |                                       |                  |
| actices** |                                       |                  |
+-----------+---------------------------------------+------------------+
| E         | #####                                 | ##               |
| lectronic |                                       |                  |
| Proposal  | ##### ***[P                           | ## **FY 1999:*   |
| S         | erformance Goal IV-1:]{.underline}*** | * NSF successful |
| ubmission |                                       |                  |
|           | ##### Ninety-five pe                  | ## **FY 2000:*   |
|           | rcent of full proposals will be recei | * NSF successful |
|           | ved electronically through FastLane.  |                  |
|           |                                       | #                |
|           | > FY 1998 Baseline 17%                | # **FY 2001: NSF |
|           |                                       |  is successful f |
|           | #####  FY 1999 Result 44%             | or goal IV-1.**  |
|           |                                       |                  |
|           | > FY 2000 Goal 60%                    |                  |
|           | >                                     |                  |
|           | > FY 2000 Result 81%                  |                  |
|           | >                                     |                  |
|           | > FY 2001 Goal 95%                    |                  |
|           | >                                     |                  |
|           | > [FY 2001 Result]{.underline} 99%    |                  |
+-----------+---------------------------------------+------------------+
| E         | ##### ***[P                           | ## New           |
| lectronic | erformance Goal IV-2:]{.underline}*** |  goal in FY 2001 |
| Proposal  |                                       |                  |
| P         | > In FY 2001, NSF will conduct ten    | **FY 2001: NSF   |
| rocessing | > pilot paperless projects that       | is successful    |
|           | > manage the competitive review       | for goal IV-2.** |
|           | > process in an electronic            |                  |
|           | > environment.                        |                  |
|           | >                                     |                  |
|           | > [FY 2001 Result:]{.underline} Ten   |                  |
|           | > pilot paperless projects were       |                  |
|           | > completed.                          |                  |
+-----------+---------------------------------------+------------------+
| Vid       | #####                                 | ##               |
| eo-Confer |                                       |                  |
| ence/Long | ##### ***[P                           | ## New           |
| -Distance | erformance Goal IV-3:]{.underline}*** |  goal in FY 2001 |
| Commu     |                                       |                  |
| nications | > By the end of FY 2001, NSF will     | **FY 2001: NSF   |
|           | > increase usage of a broad range of  | is successful    |
|           | > video-conferencing / long distance  | for goal IV-3.** |
|           | > communications technology by 100    |                  |
|           | > percent over the FY 1999 level.     |                  |
|           | >                                     |                  |
|           | > [FY 2001 Result:]{.underline} 142   |                  |
|           | > video-conferences were conducted,   |                  |
|           | > an increase of 184 percent over the |                  |
|           | > 1999 level.                         |                  |
+-----------+---------------------------------------+------------------+
| **NSF     |                                       |                  |
| Staff**   |                                       |                  |
+-----------+---------------------------------------+------------------+
| Diversity | ##### ***[P                           | Goal revised in  |
|           | erformance Goal IV-4:]{.underline}*** | FY 2000          |
|           |                                       |                  |
|           | > NSF will show an increase over 1997 | **FY 1999:** NSF |
|           | > in the total number of hires to     | successful for   |
|           | > science and engineering (S&E)       | related goal     |
|           | > positions from underrepresented     |                  |
|           | > groups.                             | **FY 2000:** NSF |
|           | >                                     | successful       |
|           | > FY 1997 Baseline: 16 females and 15 |                  |
|           | > members of underrepresented         | **FY 2001: NSF   |
|           | > minority groups were hired.         | is successful    |
|           | >                                     | for goal IV-4.** |
|           | > FY 2000 Result: 35 females and 19   |                  |
|           | > members of underrepresented         |                  |
|           | > minority groups were hired.         |                  |
|           | >                                     |                  |
|           | > [FY 2001 Result]{.underline}: 38    |                  |
|           | > females and 22 members of           |                  |
|           | > underrepresented minority groups    |                  |
|           | > were hired.                         |                  |
+-----------+---------------------------------------+------------------+

**ANNUAL PERFORMANCE GOALS FOR NSF'S MANAGEMENT**

**(continued)**

+----------+---+---------------------------------------+---+------------------+
| **Per    | * |                                       | * |                  |
| formance | * |                                       | * |                  |
| Area**   | F |                                       | R |                  |
|          | Y |                                       | e |                  |
|          | 2 |                                       | s |                  |
|          | 0 |                                       | u |                  |
|          | 0 |                                       | l |                  |
|          | 1 |                                       | t |                  |
|          | A |                                       | s |                  |
|          | n |                                       | f |                  |
|          | n |                                       | o |                  |
|          | u |                                       | r |                  |
|          | a |                                       | * |                  |
|          | l |                                       | * |                  |
|          | * |                                       |   |                  |
|          | * |                                       | * |                  |
|          |   |                                       | * |                  |
|          | * |                                       | N |                  |
|          | * |                                       | a |                  |
|          | P |                                       | t |                  |
|          | e |                                       | i |                  |
|          | r |                                       | o |                  |
|          | f |                                       | n |                  |
|          | o |                                       | a |                  |
|          | r |                                       | l |                  |
|          | m |                                       | S |                  |
|          | a |                                       | c |                  |
|          | n |                                       | i |                  |
|          | c |                                       | e |                  |
|          | e |                                       | n |                  |
|          | G |                                       | c |                  |
|          | o |                                       | e |                  |
|          | a |                                       | F |                  |
|          | l |                                       | o |                  |
|          | * |                                       | u |                  |
|          | * |                                       | n |                  |
|          |   |                                       | d |                  |
|          |   |                                       | a |                  |
|          |   |                                       | t |                  |
|          |   |                                       | i |                  |
|          |   |                                       | o |                  |
|          |   |                                       | n |                  |
|          |   |                                       | * |                  |
|          |   |                                       | * |                  |
+----------+---+---------------------------------------+---+------------------+
| Work     |   | ##### ***[P                           |   | ## New           |
| Env      |   | erformance Goal IV-5:]{.underline}*** |   |  goal in FY 2001 |
| ironment |   |                                       |   |                  |
|          |   | > NSF will establish various          |   | **FY 2001: NSF   |
|          |   | > baselines that will enable          |   | is not           |
|          |   | > management to better assess the     |   | successful for   |
|          |   | > quality of worklife and work        |   | goal IV-5.**     |
|          |   | > environment within the Foundation.  |   |                  |
|          |   | >                                     |   |                  |
|          |   | > [FY 2001 Result]{.underline}:       |   |                  |
|          |   | > Development of an employee survey   |   |                  |
|          |   | > is underway. This survey will       |   |                  |
|          |   | > provide baseline information on the |   |                  |
|          |   | > quality of worklife and work        |   |                  |
|          |   | > environment at NSF.                 |   |                  |
|          |   | >                                     |   |                  |
|          |   | > In FY 2002, the survey will be made |   |                  |
|          |   | > available to employees.             |   |                  |
+----------+---+---------------------------------------+---+------------------+

**ANNUAL PERFORMANCE GOALS FOR NSF'S INVESTMENT PROCESS**

+-----------+---+---+--------------------------------------+---+---+------------------+
| **Pe      |   |   | **FY 2001 Annual**                   |   |   | **Results for**  |
| rformance |   |   |                                      |   |   |                  |
| Area**    |   |   | **Performance Goal**                 |   |   | **National       |
|           |   |   |                                      |   |   | Science          |
|           |   |   |                                      |   |   | Foundation**     |
+-----------+---+---+--------------------------------------+---+---+------------------+
| *         |   | * |                                      |   |   |                  |
| *Proposal |   | * |                                      |   |   |                  |
| and Award |   |   |                                      |   |   |                  |
| Pr        |   | * |                                      |   |   |                  |
| ocesses** |   | * |                                      |   |   |                  |
+-----------+---+---+--------------------------------------+---+---+------------------+
| Use of    |   | * |                                      |   | G |                  |
| Merit     |   | * |                                      |   | o |                  |
| Review    |   | * |                                      |   | a |                  |
|           |   | [ |                                      |   | l |                  |
|           |   | P |                                      |   | r |                  |
|           |   | e |                                      |   | e |                  |
|           |   | r |                                      |   | v |                  |
|           |   | f |                                      |   | i |                  |
|           |   | o |                                      |   | s |                  |
|           |   | r |                                      |   | e |                  |
|           |   | m |                                      |   | d |                  |
|           |   | a |                                      |   | i |                  |
|           |   | n |                                      |   | n |                  |
|           |   | c |                                      |   | F |                  |
|           |   | e |                                      |   | Y |                  |
|           |   | G |                                      |   | 2 |                  |
|           |   | o |                                      |   | 0 |                  |
|           |   | a |                                      |   | 0 |                  |
|           |   | l |                                      |   | 0 |                  |
|           |   | V |                                      |   |   |                  |
|           |   | - |                                      |   | * |                  |
|           |   | 1 |                                      |   | * |                  |
|           |   | : |                                      |   | F |                  |
|           |   | ] |                                      |   | Y |                  |
|           |   | { |                                      |   | 1 |                  |
|           |   | . |                                      |   | 9 |                  |
|           |   | u |                                      |   | 9 |                  |
|           |   | n |                                      |   | 9 |                  |
|           |   | d |                                      |   | : |                  |
|           |   | e |                                      |   | * |                  |
|           |   | r |                                      |   | * |                  |
|           |   | l |                                      |   | N |                  |
|           |   | i |                                      |   | S |                  |
|           |   | n |                                      |   | F |                  |
|           |   | e |                                      |   | s |                  |
|           |   | } |                                      |   | u |                  |
|           |   | * |                                      |   | c |                  |
|           |   | * |                                      |   | c |                  |
|           |   | * |                                      |   | e |                  |
|           |   |   |                                      |   | s |                  |
|           |   | A |                                      |   | s |                  |
|           |   | t |                                      |   | f |                  |
|           |   | l |                                      |   | u |                  |
|           |   | e |                                      |   | l |                  |
|           |   | a |                                      |   | f |                  |
|           |   | s |                                      |   | o |                  |
|           |   | t |                                      |   | r |                  |
|           |   | 8 |                                      |   | r |                  |
|           |   | 5 |                                      |   | e |                  |
|           |   | p |                                      |   | l |                  |
|           |   | e |                                      |   | a |                  |
|           |   | r |                                      |   | t |                  |
|           |   | c |                                      |   | e |                  |
|           |   | e |                                      |   | d |                  |
|           |   | n |                                      |   | g |                  |
|           |   | t |                                      |   | o |                  |
|           |   | o |                                      |   | a |                  |
|           |   | f |                                      |   | l |                  |
|           |   | b |                                      |   |   |                  |
|           |   | a |                                      |   | * |                  |
|           |   | s |                                      |   | * |                  |
|           |   | i |                                      |   | F |                  |
|           |   | c |                                      |   | Y |                  |
|           |   | a |                                      |   | 2 |                  |
|           |   | n |                                      |   | 0 |                  |
|           |   | d |                                      |   | 0 |                  |
|           |   | a |                                      |   | 0 |                  |
|           |   | p |                                      |   | : |                  |
|           |   | p |                                      |   | * |                  |
|           |   | l |                                      |   | * |                  |
|           |   | i |                                      |   | N |                  |
|           |   | e |                                      |   | S |                  |
|           |   | d |                                      |   | F |                  |
|           |   | r |                                      |   | s |                  |
|           |   | e |                                      |   | u |                  |
|           |   | s |                                      |   | c |                  |
|           |   | e |                                      |   | c |                  |
|           |   | a |                                      |   | e |                  |
|           |   | r |                                      |   | s |                  |
|           |   | c |                                      |   | s |                  |
|           |   | h |                                      |   | f |                  |
|           |   | f |                                      |   | u |                  |
|           |   | u |                                      |   | l |                  |
|           |   | n |                                      |   |   |                  |
|           |   | d |                                      |   | * |                  |
|           |   | s |                                      |   | * |                  |
|           |   | w |                                      |   | F |                  |
|           |   | i |                                      |   | Y |                  |
|           |   | l |                                      |   | 2 |                  |
|           |   | l |                                      |   | 0 |                  |
|           |   | b |                                      |   | 0 |                  |
|           |   | e |                                      |   | 1 |                  |
|           |   | a |                                      |   | : |                  |
|           |   | l |                                      |   | N |                  |
|           |   | l |                                      |   | S |                  |
|           |   | o |                                      |   | F |                  |
|           |   | c |                                      |   | i |                  |
|           |   | a |                                      |   | s |                  |
|           |   | t |                                      |   | s |                  |
|           |   | e |                                      |   | u |                  |
|           |   | d |                                      |   | c |                  |
|           |   | t |                                      |   | c |                  |
|           |   | o |                                      |   | e |                  |
|           |   | p |                                      |   | s |                  |
|           |   | r |                                      |   | s |                  |
|           |   | o |                                      |   | f |                  |
|           |   | j |                                      |   | u |                  |
|           |   | e |                                      |   | l |                  |
|           |   | c |                                      |   | f |                  |
|           |   | t |                                      |   | o |                  |
|           |   | s |                                      |   | r |                  |
|           |   | , |                                      |   | g |                  |
|           |   | w |                                      |   | o |                  |
|           |   | h |                                      |   | a |                  |
|           |   | i |                                      |   | l |                  |
|           |   | c |                                      |   | V |                  |
|           |   | h |                                      |   | - |                  |
|           |   | u |                                      |   | 1 |                  |
|           |   | n |                                      |   | . |                  |
|           |   | d |                                      |   | * |                  |
|           |   | e |                                      |   | * |                  |
|           |   | r |                                      |   |   |                  |
|           |   | g |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | m |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | v |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | w |                                      |   |   |                  |
|           |   | . |                                      |   |   |                  |
|           |   | \ |                                      |   |   |                  |
|           |   | * |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | F |                                      |   |   |                  |
|           |   | Y |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | 2 |                                      |   |   |                  |
|           |   | 0 |                                      |   |   |                  |
|           |   | 0 |                                      |   |   |                  |
|           |   | 0 |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | G |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | 8 |                                      |   |   |                  |
|           |   | 0 |                                      |   |   |                  |
|           |   | % |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | F |                                      |   |   |                  |
|           |   | Y |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | 2 |                                      |   |   |                  |
|           |   | 0 |                                      |   |   |                  |
|           |   | 0 |                                      |   |   |                  |
|           |   | 0 |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | R |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | u |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | 8 |                                      |   |   |                  |
|           |   | 7 |                                      |   |   |                  |
|           |   | % |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | F |                                      |   |   |                  |
|           |   | Y |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | 2 |                                      |   |   |                  |
|           |   | 0 |                                      |   |   |                  |
|           |   | 0 |                                      |   |   |                  |
|           |   | 1 |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | G |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | 8 |                                      |   |   |                  |
|           |   | 5 |                                      |   |   |                  |
|           |   | % |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | [ |                                      |   |   |                  |
|           |   | F |                                      |   |   |                  |
|           |   | Y |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | 2 |                                      |   |   |                  |
|           |   | 0 |                                      |   |   |                  |
|           |   | 0 |                                      |   |   |                  |
|           |   | 1 |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | R |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | u |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | ] |                                      |   |   |                  |
|           |   | { |                                      |   |   |                  |
|           |   | . |                                      |   |   |                  |
|           |   | u |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | d |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | } |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | 8 |                                      |   |   |                  |
|           |   | 8 |                                      |   |   |                  |
|           |   | % |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | * |                                      |   |   |                  |
|           |   | \ |                                      |   |   |                  |
|           |   | * |                                      |   |   |                  |
|           |   | D |                                      |   |   |                  |
|           |   | u |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | g |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | F |                                      |   |   |                  |
|           |   | Y |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | 2 |                                      |   |   |                  |
|           |   | 0 |                                      |   |   |                  |
|           |   | 0 |                                      |   |   |                  |
|           |   | 0 |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | O |                                      |   |   |                  |
|           |   | M |                                      |   |   |                  |
|           |   | B |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | d |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | f |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | d |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | w |                                      |   |   |                  |
|           |   | h |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | c |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | u |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | m |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | - |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | v |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | w |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | d |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | p |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | j |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | c |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | d |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | b |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | h |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | d |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | w |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | g |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | v |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | f |                                      |   |   |                  |
|           |   | > |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | 7 |                                      |   |   |                  |
|           |   | 0 |                                      |   |   |                  |
|           |   | - |                                      |   |   |                  |
|           |   | 9 |                                      |   |   |                  |
|           |   | 0 |                                      |   |   |                  |
|           |   | % |                                      |   |   |                  |
|           |   | . |                                      |   |   |                  |
|           |   | * |                                      |   |   |                  |
+-----------+---+---+--------------------------------------+---+---+------------------+
| Imple     |   | * |                                      |   | G |                  |
| mentation |   | * |                                      |   | o |                  |
| of Merit  |   | * |                                      |   | a |                  |
| Review    |   | [ |                                      |   | l |                  |
| Criteria  |   | P |                                      |   | r |                  |
| --        |   | e |                                      |   | e |                  |
| Reviewers |   | r |                                      |   | v |                  |
|           |   | f |                                      |   | i |                  |
|           |   | o |                                      |   | s |                  |
|           |   | r |                                      |   | e |                  |
|           |   | m |                                      |   | d |                  |
|           |   | a |                                      |   | i |                  |
|           |   | n |                                      |   | n |                  |
|           |   | c |                                      |   | F |                  |
|           |   | e |                                      |   | Y |                  |
|           |   | G |                                      |   | 2 |                  |
|           |   | o |                                      |   | 0 |                  |
|           |   | a |                                      |   | 0 |                  |
|           |   | l |                                      |   | 1 |                  |
|           |   | V |                                      |   |   |                  |
|           |   | - |                                      |   | * |                  |
|           |   | 2 |                                      |   | * |                  |
|           |   | : |                                      |   | F |                  |
|           |   | ] |                                      |   | Y |                  |
|           |   | { |                                      |   | 2 |                  |
|           |   | . |                                      |   | 0 |                  |
|           |   | u |                                      |   | 0 |                  |
|           |   | n |                                      |   | 1 |                  |
|           |   | d |                                      |   | : |                  |
|           |   | e |                                      |   | N |                  |
|           |   | r |                                      |   | S |                  |
|           |   | l |                                      |   | F |                  |
|           |   | i |                                      |   | i |                  |
|           |   | n |                                      |   | s |                  |
|           |   | e |                                      |   | n |                  |
|           |   | } |                                      |   | o |                  |
|           |   | * |                                      |   | t |                  |
|           |   | * |                                      |   | s |                  |
|           |   | * |                                      |   | u |                  |
|           |   |   |                                      |   | c |                  |
|           |   | N |                                      |   | c |                  |
|           |   | S |                                      |   | e |                  |
|           |   | F |                                      |   | s |                  |
|           |   | p |                                      |   | s |                  |
|           |   | e |                                      |   | f |                  |
|           |   | r |                                      |   | u |                  |
|           |   | f |                                      |   | l |                  |
|           |   | o |                                      |   | f |                  |
|           |   | r |                                      |   | o |                  |
|           |   | m |                                      |   | r |                  |
|           |   | a |                                      |   | g |                  |
|           |   | n |                                      |   | o |                  |
|           |   | c |                                      |   | a |                  |
|           |   | e |                                      |   | l |                  |
|           |   | i |                                      |   | V |                  |
|           |   | n |                                      |   | - |                  |
|           |   | i |                                      |   | 2 |                  |
|           |   | m |                                      |   | * |                  |
|           |   | p |                                      |   | * |                  |
|           |   | l |                                      |   | . |                  |
|           |   | e |                                      |   |   |                  |
|           |   | m |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | f |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | h |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | m |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | v |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | w |                                      |   |   |                  |
|           |   | c |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | u |                                      |   |   |                  |
|           |   | c |                                      |   |   |                  |
|           |   | c |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | f |                                      |   |   |                  |
|           |   | u |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | w |                                      |   |   |                  |
|           |   | h |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | v |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | w |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | d |                                      |   |   |                  |
|           |   | d |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | h |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | m |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | f |                                      |   |   |                  |
|           |   | b |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | h |                                      |   |   |                  |
|           |   | g |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | c |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | v |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | w |                                      |   |   |                  |
|           |   | c |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | . |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | [ |                                      |   |   |                  |
|           |   | F |                                      |   |   |                  |
|           |   | Y |                                      |   |   |                  |
|           |   | 2 |                                      |   |   |                  |
|           |   | 0 |                                      |   |   |                  |
|           |   | 0 |                                      |   |   |                  |
|           |   | 1 |                                      |   |   |                  |
|           |   | R |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | u |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | ] |                                      |   |   |                  |
|           |   | { |                                      |   |   |                  |
|           |   | . |                                      |   |   |                  |
|           |   | u |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | d |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | } |                                      |   |   |                  |
|           |   | : |                                      |   |   |                  |
|           |   | P |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | g |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | m |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | p |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | p |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | p |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | d |                                      |   |   |                  |
|           |   | b |                                      |   |   |                  |
|           |   | y |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | x |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | x |                                      |   |   |                  |
|           |   | p |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | d |                                      |   |   |                  |
|           |   | u |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | g |                                      |   |   |                  |
|           |   | F |                                      |   |   |                  |
|           |   | Y |                                      |   |   |                  |
|           |   | 2 |                                      |   |   |                  |
|           |   | 0 |                                      |   |   |                  |
|           |   | 0 |                                      |   |   |                  |
|           |   | 1 |                                      |   |   |                  |
|           |   | G |                                      |   |   |                  |
|           |   | P |                                      |   |   |                  |
|           |   | R |                                      |   |   |                  |
|           |   | A |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | p |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | g |                                      |   |   |                  |
|           |   | p |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | v |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | d |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | m |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | f |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | m |                                      |   |   |                  |
|           |   | p |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | m |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | f |                                      |   |   |                  |
|           |   | m |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | v |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | w |                                      |   |   |                  |
|           |   | c |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | f |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | v |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | w |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | . |                                      |   |   |                  |
|           |   | I |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | F |                                      |   |   |                  |
|           |   | Y |                                      |   |   |                  |
|           |   | 1 |                                      |   |   |                  |
|           |   | 9 |                                      |   |   |                  |
|           |   | 9 |                                      |   |   |                  |
|           |   | 8 |                                      |   |   |                  |
|           |   | - |                                      |   |   |                  |
|           |   | - |                                      |   |   |                  |
|           |   | F |                                      |   |   |                  |
|           |   | Y |                                      |   |   |                  |
|           |   | 2 |                                      |   |   |                  |
|           |   | 0 |                                      |   |   |                  |
|           |   | 0 |                                      |   |   |                  |
|           |   | 0 |                                      |   |   |                  |
|           |   | , |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | v |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | w |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | d |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | d |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | c |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | y |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | d |                                      |   |   |                  |
|           |   | d |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | h |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | b |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | d |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | m |                                      |   |   |                  |
|           |   | p |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | c |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | c |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | . |                                      |   |   |                  |
|           |   | I |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | F |                                      |   |   |                  |
|           |   | Y |                                      |   |   |                  |
|           |   | 2 |                                      |   |   |                  |
|           |   | 0 |                                      |   |   |                  |
|           |   | 0 |                                      |   |   |                  |
|           |   | 1 |                                      |   |   |                  |
|           |   | , |                                      |   |   |                  |
|           |   | N |                                      |   |   |                  |
|           |   | S |                                      |   |   |                  |
|           |   | F |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | d |                                      |   |   |                  |
|           |   | d |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | d |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | p |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | v |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | w |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | c |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | F |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | L |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | b |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | v |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | w |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | d |                                      |   |   |                  |
|           |   | d |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | c |                                      |   |   |                  |
|           |   | h |                                      |   |   |                  |
|           |   | m |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | - |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | v |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | w |                                      |   |   |                  |
|           |   | c |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | p |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | y |                                      |   |   |                  |
|           |   | . |                                      |   |   |                  |
|           |   | N |                                      |   |   |                  |
|           |   | S |                                      |   |   |                  |
|           |   | F |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | b |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | h |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | d |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | k |                                      |   |   |                  |
|           |   | f |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | c |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | x |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | m |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | g |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | m |                                      |   |   |                  |
|           |   | p |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | v |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | b |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | h |                                      |   |   |                  |
|           |   | p |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | p |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | d |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | v |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | w |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | h |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | b |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | d |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | m |                                      |   |   |                  |
|           |   | p |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | c |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | c |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | . |                                      |   |   |                  |
|           |   | A |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | u |                                      |   |   |                  |
|           |   | m |                                      |   |   |                  |
|           |   | b |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | f |                                      |   |   |                  |
|           |   | F |                                      |   |   |                  |
|           |   | Y |                                      |   |   |                  |
|           |   | 2 |                                      |   |   |                  |
|           |   | 0 |                                      |   |   |                  |
|           |   | 0 |                                      |   |   |                  |
|           |   | 1 |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | p |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | h |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | v |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | w |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | m |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | k |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | g |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | g |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | f |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | c |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | p |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | g |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | u |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | z |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | g |                                      |   |   |                  |
|           |   | b |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | h |                                      |   |   |                  |
|           |   | m |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | v |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | w |                                      |   |   |                  |
|           |   | c |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | . |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           |   | I |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | F |                                      |   |   |                  |
|           |   | Y |                                      |   |   |                  |
|           |   | 2 |                                      |   |   |                  |
|           |   | 0 |                                      |   |   |                  |
|           |   | 0 |                                      |   |   |                  |
|           |   | 2 |                                      |   |   |                  |
|           |   | , |                                      |   |   |                  |
|           |   | N |                                      |   |   |                  |
|           |   | S |                                      |   |   |                  |
|           |   | F |                                      |   |   |                  |
|           |   | w |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | c |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | u |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | d |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | v |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | p |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | d |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | p |                                      |   |   |                  |
|           |   | p |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | y |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | c |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | m |                                      |   |   |                  |
|           |   | m |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | d |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | h |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | f |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | c |                                      |   |   |                  |
|           |   | u |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | g |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | w |                                      |   |   |                  |
|           |   | h |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | c |                                      |   |   |                  |
|           |   | h |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | h |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | m |                                      |   |   |                  |
|           |   | p |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | c |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | f |                                      |   |   |                  |
|           |   | u |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | g |                                      |   |   |                  |
|           |   | b |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | h |                                      |   |   |                  |
|           |   | c |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | . |                                      |   |   |                  |
|           |   | I |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | w |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | c |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | c |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | d |                                      |   |   |                  |
|           |   | m |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | k |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | v |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | b |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | x |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | m |                                      |   |   |                  |
|           |   | p |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | f |                                      |   |   |                  |
|           |   | b |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | d |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | r |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | m |                                      |   |   |                  |
|           |   | p |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | c |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | d |                                      |   |   |                  |
|           |   | d |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | v |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | p |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | p |                                      |   |   |                  |
|           |   | l |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | o |                                      |   |   |                  |
|           |   | d |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | s |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | m |                                      |   |   |                  |
|           |   | i |                                      |   |   |                  |
|           |   | n |                                      |   |   |                  |
|           |   | a |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | t |                                      |   |   |                  |
|           |   | h |                                      |   |   |                  |
|           |   | e |                                      |   |   |                  |
|           |   | m |                                      |   |   |                  |
|           |   | . |                                      |   |   |                  |
+-----------+---+---+--------------------------------------+---+---+------------------+
| Imple     | * |   |                                      | G |   |                  |
| mentation | [ |   |                                      | o |   |                  |
| of Merit  | P |   |                                      | a |   |                  |
| Review    | e |   |                                      | l |   |                  |
| Criteria  | r |   |                                      | r |   |                  |
| --        | f |   |                                      | e |   |                  |
| Program   | o |   |                                      | v |   |                  |
| Officers  | r |   |                                      | i |   |                  |
|           | m |   |                                      | s |   |                  |
|           | a |   |                                      | e |   |                  |
|           | n |   |                                      | d |   |                  |
|           | c |   |                                      | i |   |                  |
|           | e |   |                                      | n |   |                  |
|           | G |   |                                      | F |   |                  |
|           | o |   |                                      | Y |   |                  |
|           | a |   |                                      | 2 |   |                  |
|           | l |   |                                      | 0 |   |                  |
|           | V |   |                                      | 0 |   |                  |
|           | - |   |                                      | 1 |   |                  |
|           | 3 |   |                                      |   |   |                  |
|           | : |   |                                      | * |   |                  |
|           | ] |   |                                      | * |   |                  |
|           | { |   |                                      | F |   |                  |
|           | . |   |                                      | Y |   |                  |
|           | u |   |                                      | 2 |   |                  |
|           | n |   |                                      | 0 |   |                  |
|           | d |   |                                      | 0 |   |                  |
|           | e |   |                                      | 1 |   |                  |
|           | r |   |                                      | : |   |                  |
|           | l |   |                                      | N |   |                  |
|           | i |   |                                      | S |   |                  |
|           | n |   |                                      | F |   |                  |
|           | e |   |                                      | i |   |                  |
|           | } |   |                                      | s |   |                  |
|           | * |   |                                      | s |   |                  |
|           |   |   |                                      | u |   |                  |
|           | N |   |                                      | c |   |                  |
|           | S |   |                                      | c |   |                  |
|           | F |   |                                      | e |   |                  |
|           | p |   |                                      | s |   |                  |
|           | e |   |                                      | s |   |                  |
|           | r |   |                                      | f |   |                  |
|           | f |   |                                      | u |   |                  |
|           | o |   |                                      | l |   |                  |
|           | r |   |                                      | f |   |                  |
|           | m |   |                                      | o |   |                  |
|           | a |   |                                      | r |   |                  |
|           | n |   |                                      | g |   |                  |
|           | c |   |                                      | o |   |                  |
|           | e |   |                                      | a |   |                  |
|           | i |   |                                      | l |   |                  |
|           | n |   |                                      | V |   |                  |
|           | i |   |                                      | - |   |                  |
|           | m |   |                                      | 3 |   |                  |
|           | p |   |                                      | . |   |                  |
|           | l |   |                                      | * |   |                  |
|           | e |   |                                      | * |   |                  |
|           | m |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | n |   |                                      |   |   |                  |
|           | t |   |                                      |   |   |                  |
|           | a |   |                                      |   |   |                  |
|           | t |   |                                      |   |   |                  |
|           | i |   |                                      |   |   |                  |
|           | o |   |                                      |   |   |                  |
|           | n |   |                                      |   |   |                  |
|           | o |   |                                      |   |   |                  |
|           | f |   |                                      |   |   |                  |
|           | t |   |                                      |   |   |                  |
|           | h |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | m |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | i |   |                                      |   |   |                  |
|           | t |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | v |   |                                      |   |   |                  |
|           | i |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | w |   |                                      |   |   |                  |
|           | c |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | i |   |                                      |   |   |                  |
|           | t |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | i |   |                                      |   |   |                  |
|           | a |   |                                      |   |   |                  |
|           | i |   |                                      |   |   |                  |
|           | s |   |                                      |   |   |                  |
|           | s |   |                                      |   |   |                  |
|           | u |   |                                      |   |   |                  |
|           | c |   |                                      |   |   |                  |
|           | c |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | s |   |                                      |   |   |                  |
|           | s |   |                                      |   |   |                  |
|           | f |   |                                      |   |   |                  |
|           | u |   |                                      |   |   |                  |
|           | l |   |                                      |   |   |                  |
|           | w |   |                                      |   |   |                  |
|           | h |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | n |   |                                      |   |   |                  |
|           | p |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | o |   |                                      |   |   |                  |
|           | g |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | a |   |                                      |   |   |                  |
|           | m |   |                                      |   |   |                  |
|           | o |   |                                      |   |   |                  |
|           | f |   |                                      |   |   |                  |
|           | f |   |                                      |   |   |                  |
|           | i |   |                                      |   |   |                  |
|           | c |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | s |   |                                      |   |   |                  |
|           | a |   |                                      |   |   |                  |
|           | d |   |                                      |   |   |                  |
|           | d |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | s |   |                                      |   |   |                  |
|           | s |   |                                      |   |   |                  |
|           | t |   |                                      |   |   |                  |
|           | h |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | l |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | m |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | n |   |                                      |   |   |                  |
|           | t |   |                                      |   |   |                  |
|           | s |   |                                      |   |   |                  |
|           | o |   |                                      |   |   |                  |
|           | f |   |                                      |   |   |                  |
|           | b |   |                                      |   |   |                  |
|           | o |   |                                      |   |   |                  |
|           | t |   |                                      |   |   |                  |
|           | h |   |                                      |   |   |                  |
|           | g |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | n |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | i |   |                                      |   |   |                  |
|           | c |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | v |   |                                      |   |   |                  |
|           | i |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | w |   |                                      |   |   |                  |
|           | c |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | i |   |                                      |   |   |                  |
|           | t |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | i |   |                                      |   |   |                  |
|           | a |   |                                      |   |   |                  |
|           | w |   |                                      |   |   |                  |
|           | h |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | n |   |                                      |   |   |                  |
|           | m |   |                                      |   |   |                  |
|           | a |   |                                      |   |   |                  |
|           | k |   |                                      |   |   |                  |
|           | i |   |                                      |   |   |                  |
|           | n |   |                                      |   |   |                  |
|           | g |   |                                      |   |   |                  |
|           | t |   |                                      |   |   |                  |
|           | h |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | i |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | a |   |                                      |   |   |                  |
|           | w |   |                                      |   |   |                  |
|           | a |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | d |   |                                      |   |   |                  |
|           | d |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | c |   |                                      |   |   |                  |
|           | i |   |                                      |   |   |                  |
|           | s |   |                                      |   |   |                  |
|           | i |   |                                      |   |   |                  |
|           | o |   |                                      |   |   |                  |
|           | n |   |                                      |   |   |                  |
|           | s |   |                                      |   |   |                  |
|           | . |   |                                      |   |   |                  |
|           |   |   |                                      |   |   |                  |
|           | [ |   |                                      |   |   |                  |
|           | F |   |                                      |   |   |                  |
|           | Y |   |                                      |   |   |                  |
|           | 2 |   |                                      |   |   |                  |
|           | 0 |   |                                      |   |   |                  |
|           | 0 |   |                                      |   |   |                  |
|           | 1 |   |                                      |   |   |                  |
|           | R |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | s |   |                                      |   |   |                  |
|           | u |   |                                      |   |   |                  |
|           | l |   |                                      |   |   |                  |
|           | t |   |                                      |   |   |                  |
|           | ] |   |                                      |   |   |                  |
|           | { |   |                                      |   |   |                  |
|           | . |   |                                      |   |   |                  |
|           | u |   |                                      |   |   |                  |
|           | n |   |                                      |   |   |                  |
|           | d |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | l |   |                                      |   |   |                  |
|           | i |   |                                      |   |   |                  |
|           | n |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | } |   |                                      |   |   |                  |
|           | : |   |                                      |   |   |                  |
|           | P |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | o |   |                                      |   |   |                  |
|           | g |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | a |   |                                      |   |   |                  |
|           | m |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | p |   |                                      |   |   |                  |
|           | o |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | t |   |                                      |   |   |                  |
|           | s |   |                                      |   |   |                  |
|           | p |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | p |   |                                      |   |   |                  |
|           | a |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | d |   |                                      |   |   |                  |
|           | b |   |                                      |   |   |                  |
|           | y |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | x |   |                                      |   |   |                  |
|           | t |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | n |   |                                      |   |   |                  |
|           | a |   |                                      |   |   |                  |
|           | l |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | x |   |                                      |   |   |                  |
|           | p |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | t |   |                                      |   |   |                  |
|           | s |   |                                      |   |   |                  |
|           | d |   |                                      |   |   |                  |
|           | u |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | i |   |                                      |   |   |                  |
|           | n |   |                                      |   |   |                  |
|           | g |   |                                      |   |   |                  |
|           | F |   |                                      |   |   |                  |
|           | Y |   |                                      |   |   |                  |
|           | 2 |   |                                      |   |   |                  |
|           | 0 |   |                                      |   |   |                  |
|           | 0 |   |                                      |   |   |                  |
|           | 1 |   |                                      |   |   |                  |
|           | G |   |                                      |   |   |                  |
|           | P |   |                                      |   |   |                  |
|           | R |   |                                      |   |   |                  |
|           | A |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | p |   |                                      |   |   |                  |
|           | o |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | t |   |                                      |   |   |                  |
|           | i |   |                                      |   |   |                  |
|           | n |   |                                      |   |   |                  |
|           | g |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | s |   |                                      |   |   |                  |
|           | u |   |                                      |   |   |                  |
|           | l |   |                                      |   |   |                  |
|           | t |   |                                      |   |   |                  |
|           | i |   |                                      |   |   |                  |
|           | n |   |                                      |   |   |                  |
|           | a |   |                                      |   |   |                  |
|           | n |   |                                      |   |   |                  |
|           | a |   |                                      |   |   |                  |
|           | s |   |                                      |   |   |                  |
|           | s |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | s |   |                                      |   |   |                  |
|           | s |   |                                      |   |   |                  |
|           | m |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | n |   |                                      |   |   |                  |
|           | t |   |                                      |   |   |                  |
|           | o |   |                                      |   |   |                  |
|           | f |   |                                      |   |   |                  |
|           | s |   |                                      |   |   |                  |
|           | u |   |                                      |   |   |                  |
|           | c |   |                                      |   |   |                  |
|           | c |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | s |   |                                      |   |   |                  |
|           | s |   |                                      |   |   |                  |
|           | f |   |                                      |   |   |                  |
|           | u |   |                                      |   |   |                  |
|           | l |   |                                      |   |   |                  |
|           | f |   |                                      |   |   |                  |
|           | o |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | t |   |                                      |   |   |                  |
|           | h |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | f |   |                                      |   |   |                  |
|           | o |   |                                      |   |   |                  |
|           | u |   |                                      |   |   |                  |
|           | n |   |                                      |   |   |                  |
|           | d |   |                                      |   |   |                  |
|           | a |   |                                      |   |   |                  |
|           | t |   |                                      |   |   |                  |
|           | i |   |                                      |   |   |                  |
|           | o |   |                                      |   |   |                  |
|           | n |   |                                      |   |   |                  |
|           | i |   |                                      |   |   |                  |
|           | n |   |                                      |   |   |                  |
|           | i |   |                                      |   |   |                  |
|           | m |   |                                      |   |   |                  |
|           | p |   |                                      |   |   |                  |
|           | l |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | m |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | n |   |                                      |   |   |                  |
|           | t |   |                                      |   |   |                  |
|           | a |   |                                      |   |   |                  |
|           | t |   |                                      |   |   |                  |
|           | i |   |                                      |   |   |                  |
|           | o |   |                                      |   |   |                  |
|           | n |   |                                      |   |   |                  |
|           | o |   |                                      |   |   |                  |
|           | f |   |                                      |   |   |                  |
|           | b |   |                                      |   |   |                  |
|           | o |   |                                      |   |   |                  |
|           | t |   |                                      |   |   |                  |
|           | h |   |                                      |   |   |                  |
|           | m |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | i |   |                                      |   |   |                  |
|           | t |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | v |   |                                      |   |   |                  |
|           | i |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | w |   |                                      |   |   |                  |
|           | c |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | i |   |                                      |   |   |                  |
|           | t |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | i |   |                                      |   |   |                  |
|           | a |   |                                      |   |   |                  |
|           | f |   |                                      |   |   |                  |
|           | o |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | p |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | o |   |                                      |   |   |                  |
|           | g |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | a |   |                                      |   |   |                  |
|           | m |   |                                      |   |   |                  |
|           | m |   |                                      |   |   |                  |
|           | a |   |                                      |   |   |                  |
|           | n |   |                                      |   |   |                  |
|           | a |   |                                      |   |   |                  |
|           | g |   |                                      |   |   |                  |
|           | e |   |                                      |   |   |                  |
|           | r |   |                                      |   |   |                  |
|           | s |   |                                      |   |   |                  |
|           | . |   |                                      |   |   |                  |
+-----------+---+---+--------------------------------------+---+---+------------------+

**ANNUAL PERFORMANCE GOALS FOR NSF'S INVESTMENT PROCESS**

**(continued)**

+----------+---+---+---------------------------------------+---+---+------------------+
| **Per    |   |   | **FY 2001 Annual**                    | * |   |                  |
| formance |   |   |                                       | * |   |                  |
| Area**   |   |   | **Performance Goal**                  | R |   |                  |
|          |   |   |                                       | e |   |                  |
|          |   |   |                                       | s |   |                  |
|          |   |   |                                       | u |   |                  |
|          |   |   |                                       | l |   |                  |
|          |   |   |                                       | t |   |                  |
|          |   |   |                                       | s |   |                  |
|          |   |   |                                       | f |   |                  |
|          |   |   |                                       | o |   |                  |
|          |   |   |                                       | r |   |                  |
|          |   |   |                                       | * |   |                  |
|          |   |   |                                       | * |   |                  |
|          |   |   |                                       |   |   |                  |
|          |   |   |                                       | * |   |                  |
|          |   |   |                                       | * |   |                  |
|          |   |   |                                       | N |   |                  |
|          |   |   |                                       | a |   |                  |
|          |   |   |                                       | t |   |                  |
|          |   |   |                                       | i |   |                  |
|          |   |   |                                       | o |   |                  |
|          |   |   |                                       | n |   |                  |
|          |   |   |                                       | a |   |                  |
|          |   |   |                                       | l |   |                  |
|          |   |   |                                       | S |   |                  |
|          |   |   |                                       | c |   |                  |
|          |   |   |                                       | i |   |                  |
|          |   |   |                                       | e |   |                  |
|          |   |   |                                       | n |   |                  |
|          |   |   |                                       | c |   |                  |
|          |   |   |                                       | e |   |                  |
|          |   |   |                                       | F |   |                  |
|          |   |   |                                       | o |   |                  |
|          |   |   |                                       | u |   |                  |
|          |   |   |                                       | n |   |                  |
|          |   |   |                                       | d |   |                  |
|          |   |   |                                       | a |   |                  |
|          |   |   |                                       | t |   |                  |
|          |   |   |                                       | i |   |                  |
|          |   |   |                                       | o |   |                  |
|          |   |   |                                       | n |   |                  |
|          |   |   |                                       | * |   |                  |
|          |   |   |                                       | * |   |                  |
+----------+---+---+---------------------------------------+---+---+------------------+
| Customer |   | * |                                       |   | * |                  |
| Service: |   | * |                                       |   | * |                  |
|          |   | * |                                       |   | F |                  |
| Time to  |   | [ |                                       |   | Y |                  |
| Prepare  |   | P |                                       |   | 1 |                  |
|          |   | e |                                       |   | 9 |                  |
| P        |   | r |                                       |   | 9 |                  |
| roposals |   | f |                                       |   | 9 |                  |
|          |   | o |                                       |   | : |                  |
|          |   | r |                                       |   | * |                  |
|          |   | m |                                       |   | * |                  |
|          |   | a |                                       |   | N |                  |
|          |   | n |                                       |   | S |                  |
|          |   | c |                                       |   | F |                  |
|          |   | e |                                       |   | n |                  |
|          |   | G |                                       |   | o |                  |
|          |   | o |                                       |   | t |                  |
|          |   | a |                                       |   | s |                  |
|          |   | l |                                       |   | u |                  |
|          |   | V |                                       |   | c |                  |
|          |   | - |                                       |   | c |                  |
|          |   | 4 |                                       |   | e |                  |
|          |   | : |                                       |   | s |                  |
|          |   | ] |                                       |   | s |                  |
|          |   | { |                                       |   | f |                  |
|          |   | . |                                       |   | u |                  |
|          |   | u |                                       |   | l |                  |
|          |   | n |                                       |   |   |                  |
|          |   | d |                                       |   | * |                  |
|          |   | e |                                       |   | * |                  |
|          |   | r |                                       |   | F |                  |
|          |   | l |                                       |   | Y |                  |
|          |   | i |                                       |   | 2 |                  |
|          |   | n |                                       |   | 0 |                  |
|          |   | e |                                       |   | 0 |                  |
|          |   | } |                                       |   | 0 |                  |
|          |   | * |                                       |   | : |                  |
|          |   | * |                                       |   | * |                  |
|          |   | * |                                       |   | * |                  |
|          |   |   |                                       |   | N |                  |
|          |   | 9 |                                       |   | S |                  |
|          |   | 5 |                                       |   | F |                  |
|          |   | p |                                       |   | n |                  |
|          |   | e |                                       |   | o |                  |
|          |   | r |                                       |   | t |                  |
|          |   | c |                                       |   | s |                  |
|          |   | e |                                       |   | u |                  |
|          |   | n |                                       |   | c |                  |
|          |   | t |                                       |   | c |                  |
|          |   | o |                                       |   | e |                  |
|          |   | f |                                       |   | s |                  |
|          |   | p |                                       |   | s |                  |
|          |   | r |                                       |   | f |                  |
|          |   | o |                                       |   | u |                  |
|          |   | g |                                       |   | l |                  |
|          |   | r |                                       |   |   |                  |
|          |   | a |                                       |   | * |                  |
|          |   | m |                                       |   | * |                  |
|          |   | a |                                       |   | F |                  |
|          |   | n |                                       |   | Y |                  |
|          |   | n |                                       |   | 2 |                  |
|          |   | o |                                       |   | 0 |                  |
|          |   | u |                                       |   | 0 |                  |
|          |   | n |                                       |   | 1 |                  |
|          |   | c |                                       |   | : |                  |
|          |   | e |                                       |   | N |                  |
|          |   | m |                                       |   | S |                  |
|          |   | e |                                       |   | F |                  |
|          |   | n |                                       |   | i |                  |
|          |   | t |                                       |   | s |                  |
|          |   | s |                                       |   | s |                  |
|          |   | w |                                       |   | u |                  |
|          |   | i |                                       |   | c |                  |
|          |   | l |                                       |   | c |                  |
|          |   | l |                                       |   | e |                  |
|          |   | b |                                       |   | s |                  |
|          |   | e |                                       |   | s |                  |
|          |   | a |                                       |   | f |                  |
|          |   | v |                                       |   | u |                  |
|          |   | a |                                       |   | l |                  |
|          |   | i |                                       |   | f |                  |
|          |   | l |                                       |   | o |                  |
|          |   | a |                                       |   | r |                  |
|          |   | b |                                       |   | g |                  |
|          |   | l |                                       |   | o |                  |
|          |   | e |                                       |   | a |                  |
|          |   | t |                                       |   | l |                  |
|          |   | o |                                       |   | V |                  |
|          |   | r |                                       |   | - |                  |
|          |   | e |                                       |   | 4 |                  |
|          |   | l |                                       |   | . |                  |
|          |   | e |                                       |   | * |                  |
|          |   | v |                                       |   | * |                  |
|          |   | a |                                       |   |   |                  |
|          |   | n |                                       |   |   |                  |
|          |   | t |                                       |   |   |                  |
|          |   | i |                                       |   |   |                  |
|          |   | n |                                       |   |   |                  |
|          |   | d |                                       |   |   |                  |
|          |   | i |                                       |   |   |                  |
|          |   | v |                                       |   |   |                  |
|          |   | i |                                       |   |   |                  |
|          |   | d |                                       |   |   |                  |
|          |   | u |                                       |   |   |                  |
|          |   | a |                                       |   |   |                  |
|          |   | l |                                       |   |   |                  |
|          |   | s |                                       |   |   |                  |
|          |   | a |                                       |   |   |                  |
|          |   | n |                                       |   |   |                  |
|          |   | d |                                       |   |   |                  |
|          |   | o |                                       |   |   |                  |
|          |   | r |                                       |   |   |                  |
|          |   | g |                                       |   |   |                  |
|          |   | a |                                       |   |   |                  |
|          |   | n |                                       |   |   |                  |
|          |   | i |                                       |   |   |                  |
|          |   | z |                                       |   |   |                  |
|          |   | a |                                       |   |   |                  |
|          |   | t |                                       |   |   |                  |
|          |   | i |                                       |   |   |                  |
|          |   | o |                                       |   |   |                  |
|          |   | n |                                       |   |   |                  |
|          |   | s |                                       |   |   |                  |
|          |   | a |                                       |   |   |                  |
|          |   | t |                                       |   |   |                  |
|          |   | l |                                       |   |   |                  |
|          |   | e |                                       |   |   |                  |
|          |   | a |                                       |   |   |                  |
|          |   | s |                                       |   |   |                  |
|          |   | t |                                       |   |   |                  |
|          |   | t |                                       |   |   |                  |
|          |   | h |                                       |   |   |                  |
|          |   | r |                                       |   |   |                  |
|          |   | e |                                       |   |   |                  |
|          |   | e |                                       |   |   |                  |
|          |   | m |                                       |   |   |                  |
|          |   | o |                                       |   |   |                  |
|          |   | n |                                       |   |   |                  |
|          |   | t |                                       |   |   |                  |
|          |   | h |                                       |   |   |                  |
|          |   | s |                                       |   |   |                  |
|          |   | p |                                       |   |   |                  |
|          |   | r |                                       |   |   |                  |
|          |   | i |                                       |   |   |                  |
|          |   | o |                                       |   |   |                  |
|          |   | r |                                       |   |   |                  |
|          |   | t |                                       |   |   |                  |
|          |   | o |                                       |   |   |                  |
|          |   | t |                                       |   |   |                  |
|          |   | h |                                       |   |   |                  |
|          |   | e |                                       |   |   |                  |
|          |   | p |                                       |   |   |                  |
|          |   | r |                                       |   |   |                  |
|          |   | o |                                       |   |   |                  |
|          |   | p |                                       |   |   |                  |
|          |   | o |                                       |   |   |                  |
|          |   | s |                                       |   |   |                  |
|          |   | a |                                       |   |   |                  |
|          |   | l |                                       |   |   |                  |
|          |   | d |                                       |   |   |                  |
|          |   | e |                                       |   |   |                  |
|          |   | a |                                       |   |   |                  |
|          |   | d |                                       |   |   |                  |
|          |   | l |                                       |   |   |                  |
|          |   | i |                                       |   |   |                  |
|          |   | n |                                       |   |   |                  |
|          |   | e |                                       |   |   |                  |
|          |   | o |                                       |   |   |                  |
|          |   | r |                                       |   |   |                  |
|          |   | t |                                       |   |   |                  |
|          |   | a |                                       |   |   |                  |
|          |   | r |                                       |   |   |                  |
|          |   | g |                                       |   |   |                  |
|          |   | e |                                       |   |   |                  |
|          |   | t |                                       |   |   |                  |
|          |   | d |                                       |   |   |                  |
|          |   | a |                                       |   |   |                  |
|          |   | t |                                       |   |   |                  |
|          |   | e |                                       |   |   |                  |
|          |   | . |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          |   | > |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          |   | F |                                       |   |   |                  |
|          |   | Y |                                       |   |   |                  |
|          |   | > |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          |   | 1 |                                       |   |   |                  |
|          |   | 9 |                                       |   |   |                  |
|          |   | 9 |                                       |   |   |                  |
|          |   | 8 |                                       |   |   |                  |
|          |   | > |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          |   | B |                                       |   |   |                  |
|          |   | a |                                       |   |   |                  |
|          |   | s |                                       |   |   |                  |
|          |   | e |                                       |   |   |                  |
|          |   | l |                                       |   |   |                  |
|          |   | i |                                       |   |   |                  |
|          |   | n |                                       |   |   |                  |
|          |   | e |                                       |   |   |                  |
|          |   | > |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          |   | 6 |                                       |   |   |                  |
|          |   | 6 |                                       |   |   |                  |
|          |   | % |                                       |   |   |                  |
|          |   | > |                                       |   |   |                  |
|          |   | > |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          |   | F |                                       |   |   |                  |
|          |   | Y |                                       |   |   |                  |
|          |   | > |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          |   | 1 |                                       |   |   |                  |
|          |   | 9 |                                       |   |   |                  |
|          |   | 9 |                                       |   |   |                  |
|          |   | 9 |                                       |   |   |                  |
|          |   | > |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          |   | R |                                       |   |   |                  |
|          |   | e |                                       |   |   |                  |
|          |   | s |                                       |   |   |                  |
|          |   | u |                                       |   |   |                  |
|          |   | l |                                       |   |   |                  |
|          |   | t |                                       |   |   |                  |
|          |   | > |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          |   | 7 |                                       |   |   |                  |
|          |   | 5 |                                       |   |   |                  |
|          |   | % |                                       |   |   |                  |
|          |   | > |                                       |   |   |                  |
|          |   | > |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          |   | F |                                       |   |   |                  |
|          |   | Y |                                       |   |   |                  |
|          |   | > |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          |   | 2 |                                       |   |   |                  |
|          |   | 0 |                                       |   |   |                  |
|          |   | 0 |                                       |   |   |                  |
|          |   | 0 |                                       |   |   |                  |
|          |   | > |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          |   | G |                                       |   |   |                  |
|          |   | o |                                       |   |   |                  |
|          |   | a |                                       |   |   |                  |
|          |   | l |                                       |   |   |                  |
|          |   | > |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          |   | 9 |                                       |   |   |                  |
|          |   | 5 |                                       |   |   |                  |
|          |   | % |                                       |   |   |                  |
|          |   | > |                                       |   |   |                  |
|          |   | > |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          |   | F |                                       |   |   |                  |
|          |   | Y |                                       |   |   |                  |
|          |   | > |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          |   | 2 |                                       |   |   |                  |
|          |   | 0 |                                       |   |   |                  |
|          |   | 0 |                                       |   |   |                  |
|          |   | 0 |                                       |   |   |                  |
|          |   | > |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          |   | R |                                       |   |   |                  |
|          |   | e |                                       |   |   |                  |
|          |   | s |                                       |   |   |                  |
|          |   | u |                                       |   |   |                  |
|          |   | l |                                       |   |   |                  |
|          |   | t |                                       |   |   |                  |
|          |   | > |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          |   | 8 |                                       |   |   |                  |
|          |   | 9 |                                       |   |   |                  |
|          |   | % |                                       |   |   |                  |
|          |   | > |                                       |   |   |                  |
|          |   | > |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          |   | F |                                       |   |   |                  |
|          |   | Y |                                       |   |   |                  |
|          |   | > |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          |   | 2 |                                       |   |   |                  |
|          |   | 0 |                                       |   |   |                  |
|          |   | 0 |                                       |   |   |                  |
|          |   | 1 |                                       |   |   |                  |
|          |   | > |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          |   | G |                                       |   |   |                  |
|          |   | o |                                       |   |   |                  |
|          |   | a |                                       |   |   |                  |
|          |   | l |                                       |   |   |                  |
|          |   | > |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          |   | 9 |                                       |   |   |                  |
|          |   | 5 |                                       |   |   |                  |
|          |   | % |                                       |   |   |                  |
|          |   | > |                                       |   |   |                  |
|          |   | > |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          |   | [ |                                       |   |   |                  |
|          |   | F |                                       |   |   |                  |
|          |   | Y |                                       |   |   |                  |
|          |   | > |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          |   | 2 |                                       |   |   |                  |
|          |   | 0 |                                       |   |   |                  |
|          |   | 0 |                                       |   |   |                  |
|          |   | 1 |                                       |   |   |                  |
|          |   | > |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          |   | R |                                       |   |   |                  |
|          |   | e |                                       |   |   |                  |
|          |   | s |                                       |   |   |                  |
|          |   | u |                                       |   |   |                  |
|          |   | l |                                       |   |   |                  |
|          |   | t |                                       |   |   |                  |
|          |   | ] |                                       |   |   |                  |
|          |   | { |                                       |   |   |                  |
|          |   | . |                                       |   |   |                  |
|          |   | u |                                       |   |   |                  |
|          |   | n |                                       |   |   |                  |
|          |   | d |                                       |   |   |                  |
|          |   | e |                                       |   |   |                  |
|          |   | r |                                       |   |   |                  |
|          |   | l |                                       |   |   |                  |
|          |   | i |                                       |   |   |                  |
|          |   | n |                                       |   |   |                  |
|          |   | e |                                       |   |   |                  |
|          |   | } |                                       |   |   |                  |
|          |   | > |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          |   | 1 |                                       |   |   |                  |
|          |   | 0 |                                       |   |   |                  |
|          |   | 0 |                                       |   |   |                  |
|          |   | % |                                       |   |   |                  |
+----------+---+---+---------------------------------------+---+---+------------------+
| Customer | * |   |                                       |   |   | FY 1999: NSF not |
| Service: | * |   |                                       |   |   | successful       |
| Time to  | * |   |                                       |   |   |                  |
| Decision | [ |   |                                       |   |   | **FY 2000:** NSF |
|          | P |   |                                       |   |   | not successful   |
|          | e |   |                                       |   |   |                  |
|          | r |   |                                       |   |   | **FY 2001: NSF   |
|          | f |   |                                       |   |   | is not           |
|          | o |   |                                       |   |   | successful for   |
|          | r |   |                                       |   |   | goal V-5.**      |
|          | m |   |                                       |   |   |                  |
|          | a |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | c |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | G |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | a |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | V |   |                                       |   |   |                  |
|          | - |   |                                       |   |   |                  |
|          | 5 |   |                                       |   |   |                  |
|          | : |   |                                       |   |   |                  |
|          | ] |   |                                       |   |   |                  |
|          | { |   |                                       |   |   |                  |
|          | . |   |                                       |   |   |                  |
|          | u |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | d |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | r |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | } |   |                                       |   |   |                  |
|          | * |   |                                       |   |   |                  |
|          | * |   |                                       |   |   |                  |
|          | * |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | F |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | r |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | 7 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | p |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | r |   |                                       |   |   |                  |
|          | c |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | t |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | f |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | p |   |                                       |   |   |                  |
|          | r |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | p |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | a |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | , |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | b |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | a |   |                                       |   |   |                  |
|          | b |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | t |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | t |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | a |   |                                       |   |   |                  |
|          | p |   |                                       |   |   |                  |
|          | p |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | c |   |                                       |   |   |                  |
|          | a |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | t |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | w |   |                                       |   |   |                  |
|          | h |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | t |   |                                       |   |   |                  |
|          | h |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | r |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | t |   |                                       |   |   |                  |
|          | h |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | r |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | p |   |                                       |   |   |                  |
|          | r |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | p |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | a |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | h |   |                                       |   |   |                  |
|          | a |   |                                       |   |   |                  |
|          | v |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | b |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | d |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | c |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | d |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | r |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | r |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | c |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | m |   |                                       |   |   |                  |
|          | m |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | d |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | d |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | f |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | r |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | f |   |                                       |   |   |                  |
|          | u |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | d |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | g |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | w |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | t |   |                                       |   |   |                  |
|          | h |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | x |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | m |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | t |   |                                       |   |   |                  |
|          | h |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | f |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | r |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | c |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | p |   |                                       |   |   |                  |
|          | t |   |                                       |   |   |                  |
|          | . |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | F |   |                                       |   |   |                  |
|          | Y |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | 1 |   |                                       |   |   |                  |
|          | 9 |   |                                       |   |   |                  |
|          | 9 |   |                                       |   |   |                  |
|          | 8 |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | B |   |                                       |   |   |                  |
|          | a |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | 5 |   |                                       |   |   |                  |
|          | 9 |   |                                       |   |   |                  |
|          | % |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | F |   |                                       |   |   |                  |
|          | Y |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | 1 |   |                                       |   |   |                  |
|          | 9 |   |                                       |   |   |                  |
|          | 9 |   |                                       |   |   |                  |
|          | 9 |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | R |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | u |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | t |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | 5 |   |                                       |   |   |                  |
|          | 8 |   |                                       |   |   |                  |
|          | % |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | F |   |                                       |   |   |                  |
|          | Y |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | 2 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | G |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | a |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | 7 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | % |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | F |   |                                       |   |   |                  |
|          | Y |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | 2 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | R |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | u |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | t |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | 5 |   |                                       |   |   |                  |
|          | 4 |   |                                       |   |   |                  |
|          | % |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | F |   |                                       |   |   |                  |
|          | Y |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | 2 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | 1 |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | G |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | a |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | 7 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | % |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | [ |   |                                       |   |   |                  |
|          | F |   |                                       |   |   |                  |
|          | Y |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | 2 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | 1 |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | R |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | u |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | t |   |                                       |   |   |                  |
|          | ] |   |                                       |   |   |                  |
|          | { |   |                                       |   |   |                  |
|          | . |   |                                       |   |   |                  |
|          | u |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | d |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | r |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | } |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | 6 |   |                                       |   |   |                  |
|          | 2 |   |                                       |   |   |                  |
|          | % |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | [ |   |                                       |   |   |                  |
|          | F |   |                                       |   |   |                  |
|          | Y |   |                                       |   |   |                  |
|          | 2 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | 1 |   |                                       |   |   |                  |
|          | R |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | u |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | t |   |                                       |   |   |                  |
|          | ] |   |                                       |   |   |                  |
|          | { |   |                                       |   |   |                  |
|          | . |   |                                       |   |   |                  |
|          | u |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | d |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | r |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | } |   |                                       |   |   |                  |
|          | : |   |                                       |   |   |                  |
|          | I |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | F |   |                                       |   |   |                  |
|          | Y |   |                                       |   |   |                  |
|          | 2 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | 1 |   |                                       |   |   |                  |
|          | , |   |                                       |   |   |                  |
|          | 6 |   |                                       |   |   |                  |
|          | 2 |   |                                       |   |   |                  |
|          | % |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | f |   |                                       |   |   |                  |
|          | p |   |                                       |   |   |                  |
|          | r |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | p |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | a |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | w |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | r |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | p |   |                                       |   |   |                  |
|          | r |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | c |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | d |   |                                       |   |   |                  |
|          | w |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | t |   |                                       |   |   |                  |
|          | h |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | 6 |   |                                       |   |   |                  |
|          | m |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | t |   |                                       |   |   |                  |
|          | h |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | f |   |                                       |   |   |                  |
|          | r |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | c |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | p |   |                                       |   |   |                  |
|          | t |   |                                       |   |   |                  |
|          | . |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | I |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | F |   |                                       |   |   |                  |
|          | Y |   |                                       |   |   |                  |
|          | 2 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | 2 |   |                                       |   |   |                  |
|          | , |   |                                       |   |   |                  |
|          | N |   |                                       |   |   |                  |
|          | S |   |                                       |   |   |                  |
|          | F |   |                                       |   |   |                  |
|          | w |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | c |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | t |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | u |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | t |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | f |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | c |   |                                       |   |   |                  |
|          | u |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | m |   |                                       |   |   |                  |
|          | p |   |                                       |   |   |                  |
|          | r |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | v |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | g |   |                                       |   |   |                  |
|          | t |   |                                       |   |   |                  |
|          | h |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | f |   |                                       |   |   |                  |
|          | f |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | c |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | c |   |                                       |   |   |                  |
|          | y |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | f |   |                                       |   |   |                  |
|          | p |   |                                       |   |   |                  |
|          | r |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | p |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | a |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | p |   |                                       |   |   |                  |
|          | r |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | c |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | g |   |                                       |   |   |                  |
|          | , |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | c |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | u |   |                                       |   |   |                  |
|          | d |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | g |   |                                       |   |   |                  |
|          | t |   |                                       |   |   |                  |
|          | h |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | d |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | m |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | a |   |                                       |   |   |                  |
|          | t |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | f |   |                                       |   |   |                  |
|          | b |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | t |   |                                       |   |   |                  |
|          | p |   |                                       |   |   |                  |
|          | r |   |                                       |   |   |                  |
|          | a |   |                                       |   |   |                  |
|          | c |   |                                       |   |   |                  |
|          | t |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | c |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | t |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | p |   |                                       |   |   |                  |
|          | r |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | g |   |                                       |   |   |                  |
|          | r |   |                                       |   |   |                  |
|          | a |   |                                       |   |   |                  |
|          | m |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | t |   |                                       |   |   |                  |
|          | a |   |                                       |   |   |                  |
|          | f |   |                                       |   |   |                  |
|          | f |   |                                       |   |   |                  |
|          | . |   |                                       |   |   |                  |
+----------+---+---+---------------------------------------+---+---+------------------+
| Award    | * |   |                                       |   |   | New goal in FY   |
| Size     | * |   |                                       |   |   | 2001             |
|          | * |   |                                       |   |   |                  |
|          | [ |   |                                       |   |   | **FY 2001: NSF   |
|          | P |   |                                       |   |   | is successful    |
|          | e |   |                                       |   |   | for goal V-6a**. |
|          | r |   |                                       |   |   |                  |
|          | f |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | r |   |                                       |   |   |                  |
|          | m |   |                                       |   |   |                  |
|          | a |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | c |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | G |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | a |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | V |   |                                       |   |   |                  |
|          | - |   |                                       |   |   |                  |
|          | 6 |   |                                       |   |   |                  |
|          | a |   |                                       |   |   |                  |
|          | : |   |                                       |   |   |                  |
|          | ] |   |                                       |   |   |                  |
|          | { |   |                                       |   |   |                  |
|          | . |   |                                       |   |   |                  |
|          | u |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | d |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | r |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | } |   |                                       |   |   |                  |
|          | * |   |                                       |   |   |                  |
|          | * |   |                                       |   |   |                  |
|          | * |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | N |   |                                       |   |   |                  |
|          | S |   |                                       |   |   |                  |
|          | F |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | w |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | c |   |                                       |   |   |                  |
|          | r |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | a |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | t |   |                                       |   |   |                  |
|          | h |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | a |   |                                       |   |   |                  |
|          | v |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | r |   |                                       |   |   |                  |
|          | a |   |                                       |   |   |                  |
|          | g |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | a |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | u |   |                                       |   |   |                  |
|          | a |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | z |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | d |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | a |   |                                       |   |   |                  |
|          | w |   |                                       |   |   |                  |
|          | a |   |                                       |   |   |                  |
|          | r |   |                                       |   |   |                  |
|          | d |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | z |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | f |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | r |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | r |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | a |   |                                       |   |   |                  |
|          | r |   |                                       |   |   |                  |
|          | c |   |                                       |   |   |                  |
|          | h |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | p |   |                                       |   |   |                  |
|          | r |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | j |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | c |   |                                       |   |   |                  |
|          | t |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | t |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | \ |   |                                       |   |   |                  |
|          | $ |   |                                       |   |   |                  |
|          | 1 |   |                                       |   |   |                  |
|          | 1 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | , |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | . |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | F |   |                                       |   |   |                  |
|          | Y |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | 1 |   |                                       |   |   |                  |
|          | 9 |   |                                       |   |   |                  |
|          | 9 |   |                                       |   |   |                  |
|          | 8 |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | B |   |                                       |   |   |                  |
|          | a |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | \ |   |                                       |   |   |                  |
|          | $ |   |                                       |   |   |                  |
|          | 9 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | , |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | F |   |                                       |   |   |                  |
|          | Y |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | 1 |   |                                       |   |   |                  |
|          | 9 |   |                                       |   |   |                  |
|          | 9 |   |                                       |   |   |                  |
|          | 9 |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | R |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | u |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | t |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | \ |   |                                       |   |   |                  |
|          | $ |   |                                       |   |   |                  |
|          | 9 |   |                                       |   |   |                  |
|          | 4 |   |                                       |   |   |                  |
|          | , |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | F |   |                                       |   |   |                  |
|          | Y |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | 2 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | R |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | u |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | t |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | \ |   |                                       |   |   |                  |
|          | $ |   |                                       |   |   |                  |
|          | 1 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | 5 |   |                                       |   |   |                  |
|          | , |   |                                       |   |   |                  |
|          | 8 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | F |   |                                       |   |   |                  |
|          | Y |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | 2 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | 1 |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | G |   |                                       |   |   |                  |
|          | o |   |                                       |   |   |                  |
|          | a |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | \ |   |                                       |   |   |                  |
|          | $ |   |                                       |   |   |                  |
|          | 1 |   |                                       |   |   |                  |
|          | 1 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | , |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | [ |   |                                       |   |   |                  |
|          | F |   |                                       |   |   |                  |
|          | Y |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | 2 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | 1 |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | R |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | s |   |                                       |   |   |                  |
|          | u |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | t |   |                                       |   |   |                  |
|          | ] |   |                                       |   |   |                  |
|          | { |   |                                       |   |   |                  |
|          | . |   |                                       |   |   |                  |
|          | u |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | d |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | r |   |                                       |   |   |                  |
|          | l |   |                                       |   |   |                  |
|          | i |   |                                       |   |   |                  |
|          | n |   |                                       |   |   |                  |
|          | e |   |                                       |   |   |                  |
|          | } |   |                                       |   |   |                  |
|          | > |   |                                       |   |   |                  |
|          |   |   |                                       |   |   |                  |
|          | \ |   |                                       |   |   |                  |
|          | $ |   |                                       |   |   |                  |
|          | 1 |   |                                       |   |   |                  |
|          | 1 |   |                                       |   |   |                  |
|          | 3 |   |                                       |   |   |                  |
|          | , |   |                                       |   |   |                  |
|          | 6 |   |                                       |   |   |                  |
|          | 0 |   |                                       |   |   |                  |
|          | 1 |   |                                       |   |   |                  |
+----------+---+---+---------------------------------------+---+---+------------------+

**ANNUAL PERFORMANCE GOALS FOR NSF\'S INVESTMENT PROCESS**

**(continued)**

+----------+---------------------------------------+------------------+
| **Per    | **FY 2001 Annual Performance Goal**   | Results for      |
| formance |                                       |                  |
| Area**   |                                       | National Science |
|          |                                       | Foundation       |
+----------+---------------------------------------+------------------+
| Award    | ***[Performance Goal                  | **FY 1999:** NSF |
| Duration | V-6b:]{.underline}***                 | successful       |
|          |                                       |                  |
|          | > NSF will increase the average       | **FY 2000:** Not |
|          | > duration of awards for research     | applicable       |
|          | > projects to at least three years.   |                  |
|          | >                                     | **FY 2001: NSF   |
|          | > FY 1998 Baseline 2.7 years          | is not           |
|          | >                                     | successful for   |
|          | > FY 1999 Goal 2.8 years              | goal V-6b**.     |
|          | >                                     |                  |
|          | > FY 1999 Result 2.8 years            |                  |
|          | >                                     |                  |
|          | > FY 2000 Result 2.8 years            |                  |
|          | >                                     |                  |
|          | > FY 2001 Goal 3.0 years              |                  |
|          | >                                     |                  |
|          | > [FY 2001 Result]{.underline} 2.9    |                  |
|          | > years                               |                  |
|          |                                       |                  |
|          | [FY 2001 Result:]{.underline}         |                  |
|          | Resource limitations impacted NSF's   |                  |
|          | ability to achieve both the award     |                  |
|          | size and award duration goals. NSF    |                  |
|          | focused its efforts on increasing     |                  |
|          | average annualized award size.        |                  |
|          |                                       |                  |
|          | In FY 2002, NSF will continue to      |                  |
|          | focus on increasing award size and    |                  |
|          | duration in order to improve the      |                  |
|          | efficiency of the research process.   |                  |
+----------+---------------------------------------+------------------+
| Mai      | ***[Performance Goal                  | FY 1999: NSF not |
| ntaining | V-7:]{.underline}***                  | successful       |
| Openness |                                       |                  |
| in the   | NSF will award 30 percent of its      | **FY 2000:** NSF |
| System   | research grants to new investigators. | not successful   |
|          |                                       |                  |
|          | > FY 1997 Baseline 27%                | **FY 2001: NSF   |
|          | >                                     | is not           |
|          | > FY 1998 27%                         | successful for   |
|          | >                                     | goal V-7.**      |
|          | > FY 1999 Goal 30%                    |                  |
|          | >                                     |                  |
|          | > FY 1999 Result 27%                  |                  |
|          | >                                     |                  |
|          | > FY 2000 Goal 30%                    |                  |
|          | >                                     |                  |
|          | > FY 2000 Result 28%                  |                  |
|          | >                                     |                  |
|          | > FY 2001 Goal 30%                    |                  |
|          | >                                     |                  |
|          | > [FY 2001 Result]{.underline} 28%    |                  |
|          | >                                     |                  |
|          | > In FY 2002, NSF will continue its   |                  |
|          | > outreach to new investigators to    |                  |
|          | > promote awareness of research       |                  |
|          | > funding and to encourage new        |                  |
|          | > investigators to submit proposals.  |                  |
+----------+---------------------------------------+------------------+

**ANNUAL PERFORMANCE GOALS FOR NSF\'S INVESTMENT PROCESS**

**(continued)**

+-----------+---+---------------------------------------+---+------------------+
| **Perf    |   | **FY 2001 Annual Performance Goal**   |   | **Results for**  |
| ormance** |   |                                       |   |                  |
|           |   |                                       |   | **National       |
| **Area**  |   |                                       |   | Science          |
|           |   |                                       |   | Foundation**     |
+-----------+---+---------------------------------------+---+------------------+
| **B       |   |                                       |   |                  |
| roadening |   |                                       |   |                  |
| Partic    |   |                                       |   |                  |
| ipation** |   |                                       |   |                  |
+-----------+---+---------------------------------------+---+------------------+
| Reviewer  |   | ***[Performance Goal                  |   | New goal in FY   |
| Pool      |   | V-8:]{.underline}***                  |   | 2001.            |
|           |   |                                       |   |                  |
|           |   | NSF will begin to request voluntary   |   | FY 2001: NSF is  |
|           |   | demographic data electronically from  |   | successful for   |
|           |   | all reviewers to determine            |   | goal V- 8.       |
|           |   | participation levels of members of    |   |                  |
|           |   | underrepresented groups in the NSF    |   |                  |
|           |   | reviewer pool.                        |   |                  |
|           |   |                                       |   |                  |
|           |   | [FY 2001 Result:]{.underline} The     |   |                  |
|           |   | reviewer system in FastLane was       |   |                  |
|           |   | revised to gather voluntary           |   |                  |
|           |   | demographic data.                     |   |                  |
+-----------+---+---------------------------------------+---+------------------+
| **F       |   |                                       |   |                  |
| acilities |   |                                       |   |                  |
| Ov        |   |                                       |   |                  |
| ersight** |   |                                       |   |                  |
+-----------+---+---------------------------------------+---+------------------+
| Con       | # |                                       | G |                  |
| struction | # |                                       | o |                  |
| and       | # |                                       | a |                  |
| Upgrade   | # |                                       | l |                  |
|           | # |                                       | r |                  |
|           |   |                                       | e |                  |
|           | * |                                       | v |                  |
|           | * |                                       | i |                  |
|           | * |                                       | s |                  |
|           | [ |                                       | e |                  |
|           | P |                                       | d |                  |
|           | e |                                       | i |                  |
|           | r |                                       | n |                  |
|           | f |                                       | F |                  |
|           | o |                                       | Y |                  |
|           | r |                                       | 2 |                  |
|           | m |                                       | 0 |                  |
|           | a |                                       | 0 |                  |
|           | n |                                       | 1 |                  |
|           | c |                                       |   |                  |
|           | e |                                       | * |                  |
|           |   |                                       | * |                  |
|           | G |                                       | F |                  |
|           | o |                                       | Y |                  |
|           | a |                                       | 1 |                  |
|           | l |                                       | 9 |                  |
|           |   |                                       | 9 |                  |
|           | V |                                       | 9 |                  |
|           | - |                                       | : |                  |
|           | 9 |                                       | * |                  |
|           | a |                                       | * |                  |
|           | ] |                                       | N |                  |
|           | { |                                       | S |                  |
|           | . |                                       | F |                  |
|           | u |                                       | s |                  |
|           | n |                                       | u |                  |
|           | d |                                       | c |                  |
|           | e |                                       | c |                  |
|           | r |                                       | e |                  |
|           | l |                                       | s |                  |
|           | i |                                       | s |                  |
|           | n |                                       | f |                  |
|           | e |                                       | u |                  |
|           | } |                                       | l |                  |
|           | : |                                       | f |                  |
|           | * |                                       | o |                  |
|           | * |                                       | r |                  |
|           | * |                                       | r |                  |
|           |   |                                       | e |                  |
|           | F |                                       | l |                  |
|           | o |                                       | a |                  |
|           | r |                                       | t |                  |
|           | 9 |                                       | e |                  |
|           | 0 |                                       | d |                  |
|           | p |                                       | g |                  |
|           | e |                                       | o |                  |
|           | r |                                       | a |                  |
|           | c |                                       | l |                  |
|           | e |                                       |   |                  |
|           | n |                                       | * |                  |
|           | t |                                       | * |                  |
|           | o |                                       | F |                  |
|           | f |                                       | Y |                  |
|           | f |                                       | 2 |                  |
|           | a |                                       | 0 |                  |
|           | c |                                       | 0 |                  |
|           | i |                                       | 0 |                  |
|           | l |                                       | : |                  |
|           | i |                                       | * |                  |
|           | t |                                       | * |                  |
|           | i |                                       | N |                  |
|           | e |                                       | S |                  |
|           | s |                                       | F |                  |
|           | , |                                       | s |                  |
|           | k |                                       | u |                  |
|           | e |                                       | c |                  |
|           | e |                                       | c |                  |
|           | p |                                       | e |                  |
|           | c |                                       | s |                  |
|           | o |                                       | s |                  |
|           | n |                                       | f |                  |
|           | s |                                       | u |                  |
|           | t |                                       | l |                  |
|           | r |                                       |   |                  |
|           | u |                                       | * |                  |
|           | c |                                       | * |                  |
|           | t |                                       | F |                  |
|           | i |                                       | Y |                  |
|           | o |                                       | 2 |                  |
|           | n |                                       | 0 |                  |
|           | a |                                       | 0 |                  |
|           | n |                                       | 1 |                  |
|           | d |                                       | : |                  |
|           | u |                                       | N |                  |
|           | p |                                       | S |                  |
|           | g |                                       | F |                  |
|           | r |                                       | i |                  |
|           | a |                                       | s |                  |
|           | d |                                       | s |                  |
|           | e |                                       | u |                  |
|           | s |                                       | c |                  |
|           | w |                                       | c |                  |
|           | i |                                       | e |                  |
|           | t |                                       | s |                  |
|           | h |                                       | s |                  |
|           | i |                                       | f |                  |
|           | n |                                       | u |                  |
|           | a |                                       | l |                  |
|           | n |                                       | f |                  |
|           | n |                                       | o |                  |
|           | u |                                       | r |                  |
|           | a |                                       | g |                  |
|           | l |                                       | o |                  |
|           | e |                                       | a |                  |
|           | x |                                       | l |                  |
|           | p |                                       | V |                  |
|           | e |                                       | - |                  |
|           | n |                                       | 9 |                  |
|           | d |                                       | a |                  |
|           | i |                                       | . |                  |
|           | t |                                       | * |                  |
|           | u |                                       | * |                  |
|           | r |                                       |   |                  |
|           | e |                                       |   |                  |
|           | p |                                       |   |                  |
|           | l |                                       |   |                  |
|           | a |                                       |   |                  |
|           | n |                                       |   |                  |
|           | , |                                       |   |                  |
|           | n |                                       |   |                  |
|           | o |                                       |   |                  |
|           | t |                                       |   |                  |
|           | t |                                       |   |                  |
|           | o |                                       |   |                  |
|           | e |                                       |   |                  |
|           | x |                                       |   |                  |
|           | c |                                       |   |                  |
|           | e |                                       |   |                  |
|           | e |                                       |   |                  |
|           | d |                                       |   |                  |
|           | 1 |                                       |   |                  |
|           | 1 |                                       |   |                  |
|           | 0 |                                       |   |                  |
|           | p |                                       |   |                  |
|           | e |                                       |   |                  |
|           | r |                                       |   |                  |
|           | c |                                       |   |                  |
|           | e |                                       |   |                  |
|           | n |                                       |   |                  |
|           | t |                                       |   |                  |
|           | o |                                       |   |                  |
|           | f |                                       |   |                  |
|           | e |                                       |   |                  |
|           | s |                                       |   |                  |
|           | t |                                       |   |                  |
|           | i |                                       |   |                  |
|           | m |                                       |   |                  |
|           | a |                                       |   |                  |
|           | t |                                       |   |                  |
|           | e |                                       |   |                  |
|           | s |                                       |   |                  |
|           | . |                                       |   |                  |
|           |   |                                       |   |                  |
|           | F |                                       |   |                  |
|           | Y |                                       |   |                  |
|           | 1 |                                       |   |                  |
|           | 9 |                                       |   |                  |
|           | 9 |                                       |   |                  |
|           | 9 |                                       |   |                  |
|           | R |                                       |   |                  |
|           | e |                                       |   |                  |
|           | s |                                       |   |                  |
|           | u |                                       |   |                  |
|           | l |                                       |   |                  |
|           | t |                                       |   |                  |
|           | : |                                       |   |                  |
|           | M |                                       |   |                  |
|           | a |                                       |   |                  |
|           | j |                                       |   |                  |
|           | o |                                       |   |                  |
|           | r |                                       |   |                  |
|           | i |                                       |   |                  |
|           | t |                                       |   |                  |
|           | y |                                       |   |                  |
|           | o |                                       |   |                  |
|           | f |                                       |   |                  |
|           | f |                                       |   |                  |
|           | a |                                       |   |                  |
|           | c |                                       |   |                  |
|           | i |                                       |   |                  |
|           | l |                                       |   |                  |
|           | i |                                       |   |                  |
|           | t |                                       |   |                  |
|           | i |                                       |   |                  |
|           | e |                                       |   |                  |
|           | s |                                       |   |                  |
|           | w |                                       |   |                  |
|           | i |                                       |   |                  |
|           | t |                                       |   |                  |
|           | h |                                       |   |                  |
|           | i |                                       |   |                  |
|           | n |                                       |   |                  |
|           | 1 |                                       |   |                  |
|           | 1 |                                       |   |                  |
|           | 0 |                                       |   |                  |
|           | p |                                       |   |                  |
|           | e |                                       |   |                  |
|           | r |                                       |   |                  |
|           | c |                                       |   |                  |
|           | e |                                       |   |                  |
|           | n |                                       |   |                  |
|           | t |                                       |   |                  |
|           | o |                                       |   |                  |
|           | f |                                       |   |                  |
|           | a |                                       |   |                  |
|           | n |                                       |   |                  |
|           | n |                                       |   |                  |
|           | u |                                       |   |                  |
|           | a |                                       |   |                  |
|           | l |                                       |   |                  |
|           | s |                                       |   |                  |
|           | p |                                       |   |                  |
|           | e |                                       |   |                  |
|           | n |                                       |   |                  |
|           | d |                                       |   |                  |
|           | i |                                       |   |                  |
|           | n |                                       |   |                  |
|           | g |                                       |   |                  |
|           | e |                                       |   |                  |
|           | s |                                       |   |                  |
|           | t |                                       |   |                  |
|           | i |                                       |   |                  |
|           | m |                                       |   |                  |
|           | a |                                       |   |                  |
|           | t |                                       |   |                  |
|           | e |                                       |   |                  |
|           | s |                                       |   |                  |
|           | . |                                       |   |                  |
|           |   |                                       |   |                  |
|           | F |                                       |   |                  |
|           | Y |                                       |   |                  |
|           | 2 |                                       |   |                  |
|           | 0 |                                       |   |                  |
|           | 0 |                                       |   |                  |
|           | 0 |                                       |   |                  |
|           | R |                                       |   |                  |
|           | e |                                       |   |                  |
|           | s |                                       |   |                  |
|           | u |                                       |   |                  |
|           | l |                                       |   |                  |
|           | t |                                       |   |                  |
|           | : |                                       |   |                  |
|           | O |                                       |   |                  |
|           | f |                                       |   |                  |
|           | t |                                       |   |                  |
|           | h |                                       |   |                  |
|           | e |                                       |   |                  |
|           | 1 |                                       |   |                  |
|           | 1 |                                       |   |                  |
|           | c |                                       |   |                  |
|           | o |                                       |   |                  |
|           | n |                                       |   |                  |
|           | s |                                       |   |                  |
|           | t |                                       |   |                  |
|           | r |                                       |   |                  |
|           | u |                                       |   |                  |
|           | c |                                       |   |                  |
|           | t |                                       |   |                  |
|           | i |                                       |   |                  |
|           | o |                                       |   |                  |
|           | n |                                       |   |                  |
|           | a |                                       |   |                  |
|           | n |                                       |   |                  |
|           | d |                                       |   |                  |
|           | u |                                       |   |                  |
|           | p |                                       |   |                  |
|           | g |                                       |   |                  |
|           | r |                                       |   |                  |
|           | a |                                       |   |                  |
|           | d |                                       |   |                  |
|           | e |                                       |   |                  |
|           | p |                                       |   |                  |
|           | r |                                       |   |                  |
|           | o |                                       |   |                  |
|           | j |                                       |   |                  |
|           | e |                                       |   |                  |
|           | c |                                       |   |                  |
|           | t |                                       |   |                  |
|           | s |                                       |   |                  |
|           | , |                                       |   |                  |
|           | a |                                       |   |                  |
|           | l |                                       |   |                  |
|           | l |                                       |   |                  |
|           | w |                                       |   |                  |
|           | e |                                       |   |                  |
|           | r |                                       |   |                  |
|           | e |                                       |   |                  |
|           | w |                                       |   |                  |
|           | i |                                       |   |                  |
|           | t |                                       |   |                  |
|           | h |                                       |   |                  |
|           | i |                                       |   |                  |
|           | n |                                       |   |                  |
|           | a |                                       |   |                  |
|           | n |                                       |   |                  |
|           | n |                                       |   |                  |
|           | u |                                       |   |                  |
|           | a |                                       |   |                  |
|           | l |                                       |   |                  |
|           | e |                                       |   |                  |
|           | x |                                       |   |                  |
|           | p |                                       |   |                  |
|           | e |                                       |   |                  |
|           | n |                                       |   |                  |
|           | d |                                       |   |                  |
|           | i |                                       |   |                  |
|           | t |                                       |   |                  |
|           | u |                                       |   |                  |
|           | r |                                       |   |                  |
|           | e |                                       |   |                  |
|           | p |                                       |   |                  |
|           | l |                                       |   |                  |
|           | a |                                       |   |                  |
|           | n |                                       |   |                  |
|           | s |                                       |   |                  |
|           | ; |                                       |   |                  |
|           | m |                                       |   |                  |
|           | o |                                       |   |                  |
|           | s |                                       |   |                  |
|           | t |                                       |   |                  |
|           | w |                                       |   |                  |
|           | e |                                       |   |                  |
|           | r |                                       |   |                  |
|           | e |                                       |   |                  |
|           | u |                                       |   |                  |
|           | n |                                       |   |                  |
|           | d |                                       |   |                  |
|           | e |                                       |   |                  |
|           | r |                                       |   |                  |
|           | b |                                       |   |                  |
|           | u |                                       |   |                  |
|           | d |                                       |   |                  |
|           | g |                                       |   |                  |
|           | e |                                       |   |                  |
|           | t |                                       |   |                  |
|           | . |                                       |   |                  |
|           |   |                                       |   |                  |
|           | [ |                                       |   |                  |
|           | F |                                       |   |                  |
|           | Y |                                       |   |                  |
|           | 2 |                                       |   |                  |
|           | 0 |                                       |   |                  |
|           | 0 |                                       |   |                  |
|           | 1 |                                       |   |                  |
|           | R |                                       |   |                  |
|           | e |                                       |   |                  |
|           | s |                                       |   |                  |
|           | u |                                       |   |                  |
|           | l |                                       |   |                  |
|           | t |                                       |   |                  |
|           | ] |                                       |   |                  |
|           | { |                                       |   |                  |
|           | . |                                       |   |                  |
|           | u |                                       |   |                  |
|           | n |                                       |   |                  |
|           | d |                                       |   |                  |
|           | e |                                       |   |                  |
|           | r |                                       |   |                  |
|           | l |                                       |   |                  |
|           | i |                                       |   |                  |
|           | n |                                       |   |                  |
|           | e |                                       |   |                  |
|           | } |                                       |   |                  |
|           | : |                                       |   |                  |
|           | O |                                       |   |                  |
|           | f |                                       |   |                  |
|           | 2 |                                       |   |                  |
|           | 5 |                                       |   |                  |
|           | c |                                       |   |                  |
|           | o |                                       |   |                  |
|           | n |                                       |   |                  |
|           | s |                                       |   |                  |
|           | t |                                       |   |                  |
|           | r |                                       |   |                  |
|           | u |                                       |   |                  |
|           | c |                                       |   |                  |
|           | t |                                       |   |                  |
|           | i |                                       |   |                  |
|           | o |                                       |   |                  |
|           | n |                                       |   |                  |
|           | a |                                       |   |                  |
|           | n |                                       |   |                  |
|           | d |                                       |   |                  |
|           | u |                                       |   |                  |
|           | p |                                       |   |                  |
|           | g |                                       |   |                  |
|           | r |                                       |   |                  |
|           | a |                                       |   |                  |
|           | d |                                       |   |                  |
|           | e |                                       |   |                  |
|           | p |                                       |   |                  |
|           | r |                                       |   |                  |
|           | o |                                       |   |                  |
|           | j |                                       |   |                  |
|           | e |                                       |   |                  |
|           | c |                                       |   |                  |
|           | t |                                       |   |                  |
|           | s |                                       |   |                  |
|           | , |                                       |   |                  |
|           | 2 |                                       |   |                  |
|           | 4 |                                       |   |                  |
|           | ( |                                       |   |                  |
|           | 9 |                                       |   |                  |
|           | 6 |                                       |   |                  |
|           | p |                                       |   |                  |
|           | e |                                       |   |                  |
|           | r |                                       |   |                  |
|           | c |                                       |   |                  |
|           | e |                                       |   |                  |
|           | n |                                       |   |                  |
|           | t |                                       |   |                  |
|           | ) |                                       |   |                  |
|           | w |                                       |   |                  |
|           | e |                                       |   |                  |
|           | r |                                       |   |                  |
|           | e |                                       |   |                  |
|           | w |                                       |   |                  |
|           | i |                                       |   |                  |
|           | t |                                       |   |                  |
|           | h |                                       |   |                  |
|           | i |                                       |   |                  |
|           | n |                                       |   |                  |
|           | 1 |                                       |   |                  |
|           | 1 |                                       |   |                  |
|           | 0 |                                       |   |                  |
|           | p |                                       |   |                  |
|           | e |                                       |   |                  |
|           | r |                                       |   |                  |
|           | c |                                       |   |                  |
|           | e |                                       |   |                  |
|           | n |                                       |   |                  |
|           | t |                                       |   |                  |
|           | o |                                       |   |                  |
|           | f |                                       |   |                  |
|           | a |                                       |   |                  |
|           | n |                                       |   |                  |
|           | n |                                       |   |                  |
|           | u |                                       |   |                  |
|           | a |                                       |   |                  |
|           | l |                                       |   |                  |
|           | e |                                       |   |                  |
|           | x |                                       |   |                  |
|           | p |                                       |   |                  |
|           | e |                                       |   |                  |
|           | n |                                       |   |                  |
|           | d |                                       |   |                  |
|           | i |                                       |   |                  |
|           | t |                                       |   |                  |
|           | u |                                       |   |                  |
|           | r |                                       |   |                  |
|           | e |                                       |   |                  |
|           | p |                                       |   |                  |
|           | l |                                       |   |                  |
|           | a |                                       |   |                  |
|           | n |                                       |   |                  |
|           | s |                                       |   |                  |
|           | . |                                       |   |                  |
+-----------+---+---------------------------------------+---+------------------+

**ANNUAL PERFORMANCE GOALS FOR NSF\'S INVESTMENT PROCESS**

**(continued)**

+-----------+---------------------------------------+------------------+
| **Perf    | **FY 2001 Annual Performance Goal**   | **Results for**  |
| ormance** |                                       |                  |
|           |                                       | **National       |
| **Area**  |                                       | Science          |
|           |                                       | Foundation**     |
+-----------+---------------------------------------+------------------+
| Con       | ***[Performance Goal                  | ## Goal re       |
| struction | V-9b:]{.underline}***                 | vised in FY 2001 |
| and       |                                       |                  |
| Upgrade   | Ninety percent of facilities will     | ##               |
|           | meet all annual schedule milestones   |                  |
|           | by the end of the reporting period.   | ## **FY 1999:**  |
|           |                                       |  NSF successful  |
|           | FY 1999 Result: Majority of           | for related goal |
|           | facilities on schedule.               |                  |
|           |                                       | **FY 2000:** NSF |
|           | FY 2000 Result: Majority (7 of 11) of | not successful   |
|           | construction/upgrade projects within  | for related goal |
|           | the annual schedule goal.             |                  |
|           |                                       | **FY 2001: NSF   |
|           | [FY 2001 Result:]{.underline} Of the  | is not           |
|           | 25 construction and upgrade projects, | successful for   |
|           | 21 (84 percent) met all annual        | goal V-9b.**     |
|           | schedule milestones by the end of the |                  |
|           | reporting period.                     |                  |
|           |                                       |                  |
|           | Project delays were caused in part by |                  |
|           | circumstances beyond the control of   |                  |
|           | the facility, technical problems and  |                  |
|           | personnel issues.                     |                  |
|           |                                       |                  |
|           | In FY 2002, NSF will work with        |                  |
|           | awardees to identify obstacles to     |                  |
|           | successful performance and implement  |                  |
|           | plans to avoid or mitigate their      |                  |
|           | consequences in the future.           |                  |
+-----------+---------------------------------------+------------------+
| Con       | ***[Performance Goal                  | **FY 1999 and FY |
| struction | V-9c:]{.underline}***                 | 2000:** There    |
| and       |                                       | were no projects |
| Upgrade   | For all construction and upgrade      | completed,       |
|           | projects initiated after 1996, keep   | therefore this   |
|           | total cost within 110 percent of      | goal did not     |
|           | estimates made at the initiation of   | apply.           |
|           | construction.                         |                  |
|           |                                       | **FY 2001:**     |
|           | [FY 2001 Result:]{.underline} One     | **NSF is         |
|           | project was completed. The actual     | successful for   |
|           | total cost was equal to the estimated | goal V-9c.**     |
|           | total cost.                           |                  |
+-----------+---------------------------------------+------------------+
| O         | #####                                 | Goal revised in  |
| perations |                                       | FY 2001          |
| and       | ##### ***[P                           |                  |
| M         | erformance Goal V-10:]{.underline}*** | ## **FY 1999:    |
| anagement |                                       | ** Inconclusive  |
| of        | For 90 percent of facilities, keep    | for related goal |
| F         | operating time lost due to            |                  |
| acilities | unscheduled downtime to less than 10  | **FY 2000:** NSF |
|           | percent of the total scheduled        | not successful   |
|           | operating time.                       | for related goal |
|           |                                       |                  |
|           | FY 1999 Result: Reporting database    | **FY 2001: NSF   |
|           | under development.                    | is not           |
|           |                                       | successful for   |
|           | FY 2000 Result: Of the 26 reporting   | goal V-10.**     |
|           | facilities, 22 (85%) met the goal of  |                  |
|           | keeping unscheduled downtime to below |                  |
|           | 10% of the total scheduled operating  |                  |
|           | time.                                 |                  |
|           |                                       |                  |
|           | [FY 2001 Result:]{.underline} Of the  |                  |
|           | 29 reporting facilities, 25 (86       |                  |
|           | percent) met the goal of keeping      |                  |
|           | unscheduled downtime to below 10      |                  |
|           | percent of the total scheduled        |                  |
|           | operating time.                       |                  |
|           |                                       |                  |
|           | Some causes of failure were outside   |                  |
|           | the control of the facility or were   |                  |
|           | related to technical problems.        |                  |
|           |                                       |                  |
|           | In FY 2002, NSF will continue to work |                  |
|           | with awardees to identify obstacles   |                  |
|           | to successful performance and develop |                  |
|           | plans to avoid or mitigate their      |                  |
|           | consequences in the future.           |                  |
|           |                                       |                  |
|           | #####                                 |                  |
+-----------+---------------------------------------+------------------+

# 

**SUPPORTING INFORMATION**

![](media/image7.png)

**FOR FY 2001 GPRA REPORTING**

###### 

# *Performance Reporting Requirements and Where to Find Them in Our Report*

*The Government Performance and Results Act of 1993* requires each
Federal agency to report, no later than 180 days following the close of
each fiscal year, to the President and the Congress on its performance
for the previous fiscal year.

According to the OMB Circular No. A-11 Part 2, Section 230, dated July,
2001, pp. 539-541, each report must include the following elements[^4]:

-   *"A comparison of your actual performance with the projected levels
    of performance as set out in the performance goals in your annual
    performance plan (see section 232.4);*

-   *An explanation, where a performance goal was not achieved, for why
    the goal was not met (see section 232.6);*

-   *A description of the plans and schedules to meet an unmet goal in
    the future, or alternatively, your recommended action regarding an
    unmet goal where you have concluded it is impractical or infeasible
    to achieve the goal (see section 232.7);*

-   *An evaluation of your performance plan for the current fiscal year,
    taking into account the actual performance achieved in the fiscal
    year covered by your report (see section 232.8);*

-   *Eventually, actual performance information for at least four fiscal
    years (see section 232.9); and*

-   *An assessment of the reliability and completeness of the
    performance data included in the report (see section 232.10)."*

Other features as they apply to the agency[^5]:

-   Program evaluations;

-   Information on use of non-Federal parties;

-   Classified appendices not available to the public;

-   Description of the quality of the reported performance information;

-   Budget information;

-   Analysis of tax expenditures; and

-   Waivers of administrative requirements.

**OUTCOME GOALS**

![](media/image8.png)

**IV. NSF OUTCOME GOALS**

# 

![](media/image9.wmf){width="0.5in" height="0.5in"}**A. PEOPLE**

**PERFORMANCE OUTCOME GOAL III-1a: Development of "a diverse,
internationally competitive and globally engaged workforce of scientists
and engineers, and well-prepared citizens."**

**** **Goal Achieved**

To achieve this outcome, we invest in the best and brightest students,
researchers and educators to ensure a well-prepared workforce and
citizenry. We provide support for formal and informal science,
mathematics, engineering and technology education at all levels -- pre
K-12, undergraduate, graduate -- in addition to professional development
and public science literacy projects that engage people of all ages in
life-long learning. Our efforts must serve as a catalyst and a test bed
for a gradual change in the process and philosophy of educating the
workforce. This is particularly true of science education at K-12 level,
given the small fraction of total resources in K-12 education
represented by NSF funding.

Goal III-1a is a new performance goal for us. Our performance for this
goal is successful if *assessments from external evaluators* find that
results we reported for the period FY 2001 demonstrate significant
achievement for one or more of the following indicators:

-   Improved mathematics, science, and technology skills for U.S.
    students at the K-12 level, and for citizens of all ages, so that
    they can be competitive in a technological society;

```{=html}
<!-- -->
```
-   A science and technology and instructional workforce that reflects
    America's diversity;

-   Globally engaged science and engineering professionals who are among
    the best in the world; and

-   A public that is provided access to the benefits of science and
    engineering research and education.

**RESULT FOR PERFORMANCE GOAL III-1a:** Reports prepared by external
experts provided examples of significant achievement in reports they
developed during FY 2001 reporting. A number of reports indicate that
further improvement is needed in activities related to diversity.

**Implications for the FY 2002 Performance Plan:** This goal will be
continued in FY 2002. The set of performance indicators related to the
People goal has been expanded and modified to appropriately reflect the
breadth of NSF activities (See Section X. for details).

**SYSTEMIC ACTIVITIES**

**Systemic Reform:** This combination of programs \[Statewide Systemic
Initiatives (SSI), Rural Systemic Initiatives (RSI), Comprehensive
Partnerships for Mathematics and Science Achievement (CPMSA)\] address
state, region, and school district science, mathematics, engineering,
and technology (SMET)[^6] education. While it is difficult, if not
impossible, to directly attribute the changes in U.S. SMET education
solely to the Educational System Reform (ESR) programs, the ESR programs
clearly contributed to these changes. First, the notion of systemic
reform now permeates SMET education, and education in general. Prior to
the SSI, RSI, and the CPMSA programs, improving SMET education was not
characterized as a systemic problem. These programs were the first to
recognize that the typical strategies used by school districts or state
departments of education to increase SMET achievement were not
sufficient to produce high-quality SMET education for all students.
Second, these programs made improvement of mathematics and science
education a priority issue for K-12 education. This combination of
programs focused the attention of school districts, states, and other
institutions on improvement of mathematics and science education and
prompted these institutions to allocate resources to this issue. Third,
these programs made serving all students a national mandate. The
importance of SMET education for all students, including those from
groups traditionally underrepresented in SMET jobs and careers, is now
widely recognized. Though bridging the achievement gap between students
from traditionally underrepresented groups and other students remains a
significant challenge, the importance of addressing this problem is
firmly established.

In the following paragraphs we highlight statements and quotations taken
directly from reports of the FY 2001 COVs (Committees of Visitors) on
NSF systemic programs.

 

**Student Achievement:**

The vast majority of the USI/CPMSA programs has shown significant
increases in the number of students participating and succeeding in
gatekeeper courses. In mathematics over the period 1993-1998 the average
annual increase for students enrolling in these courses is about 9% and
the annual increase of successful mathematics completions during this
time was about 7%. In science during the same period enrollment
increased about 7% annually and completions increased by 8% annually,
which is excellent.

> Except for Cohort I, the RSI have not been in effect long enough for
> program-wide evidence of substantial student achievement gains to
> materialize. The RSI program has made significant progress in building
> the capacity and infrastructure of rural districts to understand,
> initiate and sustain mathematics and science reform. This action
> should result in increased student achievement. Given the initial
> state of RSI districts---poor rural districts with few resources to
> initiate and support reform efforts or to provide challenging
> educational opportunities to students---the focus on building
> infrastructure and capacity is the appropriate starting point for
> reform efforts.

Of particular note are the achievement gains made in the Puerto Rico
SSI, the Louisiana SSI, and the Massachusetts SSI...For the
Massachusetts SSI, we find that over the last three years, the trend has
been of increased enrollment in all AP classes among all populations.
About 79% of the more than 12,000 students taking AP courses also took
the Advanced Placement (AP) exam, of which over 71% achieved a score of
3 or better on the exam. The Calculus enrollment (both AB and BC) has
received the more robust raises with more than a 50% increase in the
African American population and a third in the Hispanic population. The
passing rates among students taking Calculus are also superior (ca. 73%
for the first course and 90% for the second course).

**Curriculum Implementation**:

Five RSI projects have been in effect for three of more years (Alaska
RSI, Appalachian RSI, Delta RSI, Tribal Colleges RSI, Navajo Nation
RSI). Some evidence of improvement in mathematics and science
achievement is beginning to emerge from these RSI projects. For example,
the SAT9 test scores of Arkansas students in schools in which the Delta
RSI has worked intensely have increased substantially compared to those
in other Delta districts and the state overall (DRSI 2000 Annual
Report). Similarly, 94% of the Appalachia RSI (ARSI) "Catalyst Schools"
are showing improvements in state assessment results relative to
comparison schools in non-participating districts (ARSI 2000 Annual
Report). And the Alaska RSI (AKRSI) reported increases in college
enrollment (from 36 students in 1994 to 70 students in 1998) by Alaskan
Native students from AKRSI districts

"The RSI have been particularly effective in the area of standards and
curricula. All RSI have increased awareness of national mathematics and
science standards; most have played key roles in state and district
standards development. For example, the Alaska RSI (AKRSI) was
responsible for the development of state science as well as mathematics
standards as well as providing professional development about these
standards. (Initially, the state department intended to create only
mathematics standards.) ... Most notably, RSI working with indigenous
populations (AKRSI, Navajo Nation RSI) have developed culturally
sensitive instructional materials that integrate the knowledge base of
indigenous peoples with western science."

The RSI have focused their limited resources to develop the knowledge
base and leadership capacity of teacher leaders and district
administrators, and have worked intensively with a subset of \"catalyst
schools,\" schools that, for a variety of factors, were most poised to
implement standards-based mathematics and science programs. The most
recent annual reports of the five RSI indicate the increased capacity of
teacher leaders and teachers in "catalysts schools" to implement
standards-based mathematics and science instruction. For example, ARSI
reported substantial increases in the percentage of "catalyst schools"
with standards-based mathematics and science curricula (from 29%
pre-ARSI to 79% in 1999), with instructional materials aligned to
curricula (from 31% pre-ARSI to 86% in 1999) and extent of implementing
inquiry-based teaching (10% pre-ARSI to 75% in 1999).

A key element to the success of the CPMSA investment is geared toward
the partnerships that have been established. These relationships not
only provide support for students and teachers but also increase
community awareness of the efforts of the CPMSA and its influence on
science and mathematics teaching and learning. Eligible activities under
this initiative include research-based professional development for
teachers; summer institutes for teachers; research-based internships
and/or mentorships for students; and tutorial programs involving
graduate/undergraduate students. In one site, the Partners in
Education/Adopt A School Project has attracted 250 volunteers to assist
with mentoring students. CPMSA sites are all involved in developing
standards-based curricula. Progress in this area varies from the
developmental phase to clearly defined time schedules. There is
indication that these efforts incorporate National Council of Teachers
of Mathematics (NCTM) standards; National Research Council (NRC)
National Science Education Standards; and state Mathematics and Science
Frameworks. The sites appear to be demonstrating more progress toward
implementation at the lower grades than at the higher grades.

**NON-SYSTEMIC ACTIVITIES**

The **Show-Me** Center, the Implementation and Dissemination Site for
middle-school mathematics curriculum: (a) has had approximately 35,000
hits on its Website
([http://showmecenter.missouri.edu](http://showmecenter.missouri.edu/))
that is estimated to reach approximately 250 new users each week; (b)
conducted a Leadership Conference attended by nearly 400 state and
national leaders in mathematics education seeking information on the
curricula; and (c) gives presentations at local, state, and national
teacher and professional society conferences.

The Leadership and Assistance for Science Education Reform Center
(**LASER**), the dissemination site for K-8 science curriculum, exceeded
its five-year goal within its first 3.5 years of operation. It has
disseminated and helped to implement NSF-supported curriculum materials
in over 300 school districts, serving some 1,000,000 K-8 students
nationwide.

**The Teachers Experiencing Antarctica and the Arctic (TEA):** K-12
teachers who join a research team in Antarctica or the Arctic post daily
electronic journals for their students and develop teaching materials
for classroom use. Fifteen teachers participated in the program in the
year 2001. NSF also supported a program for K-12 Teacher Training in
Arctic Science. A partnership between the University of Texas at Austin
Marine Science Institute and the Port Arkansas Independent School
District provided training for K-12 teachers on Arctic science,
including topics such as climate, sea ice, ozone depletion, and human
adaptations.

**IRIS Education:** The Education & Outreach (E&O) Program of the IRIS
Consortium runs one-day workshops that train both pre-service and
in-service K-12 teachers. IRIS uses a two-pronged approach to reach the
K-12 teacher population: directly through workshops held at national
professional scientific and education meetings, and by training
seismologists in the research community to run teacher workshops and
then providing these seismologists with the resources to run a workshop
locally. The IRIS E&O Program has placed high priority on development of
educational materials that can be used at all levels. One-page handouts
and posters have been distributed to a wide audience worldwide that
includes research scientists, college and K-12 educators and their
students, and the public. Over the past two years, more than 15,000
handouts and posters have been distributed nationally and
internationally upon request, with a similar number distributed at
national and regional scientific and educational meetings. Both English
and Spanish versions of the one-pagers have been developed
(<http://www.iris.washington.edu/EandO/onepager.html>).

**Distributed Mentor Project:** The primary goal of this project is to
increase the number of women entering graduate school in Computer
Science and Engineering (CS&E) by involving them in research at a
university with a female mentor. Each year, approximately twenty
undergraduate women have participated in the research and mentoring
activities of the Distributed Mentor Project (DMP). The students are
involved in research, learn how a research university operates, meet
graduate students and professors, and get a chance to observe a
successful female researcher first hand. A longitudinal evaluation study
of the project is being conducted by the Learning through Evaluation,
Adaptation, and Dissemination (LEAD) Center of the University of
Wisconsin[^7].

The longitudinal evaluation shows the DMP project to be spectacularly
successful at meeting its goal of increasing the number of women
entering graduate school in CS&E. Using a Baccalaureate & Beyond study
conducted in 1994 as a comparison, the best male CS&E graduates were 10
time more likely to enter graduate or professional school within one
year of graduation than the best female CS&E graduates; the figure for
men being 29.19% of graduates, for women being 2.53% of graduates[^8].
Of the DMP participants, over 50% were enrolled in graduate or
professional school the year following their graduation. In both cases
the surveys considered only graduates with Grade Point Averages (GPA)
greater than or equal to 3.5.

One successful program that provides a "pipeline\" to bring ethnically
diverse undergraduates into careers in the atmospheric and related
sciences **Significant Opportunities in Atmospheric Research and Science
(SOARS):** SOARS resides at the National Center for Atmospheric Research
(NCAR) and identifies, recruits, and provides students attending
colleges/universities in the U.S. and Puerto Rico with multi-summer
research experiences, year-round guidance, counseling, and mentoring,
and up to two years of graduate school support. SOARS boasts an 82%
retention rate. Since its 1996 inception, 62 students/protégés have
participated. Forty-two are current participants, nine are alumni and 11
have left the program. To date 38 have completed bachelor's degrees in a
science field, 17 are enrolled in atmospheric or related science
graduate programs, three are American Mathematical Society (AMS)
graduate fellows, nine have completed master's degree programs, and
three are Ph.D. candidates. No protégé has left college/university
without completing an undergraduate degree in science, mathematics or
engineering. Seven protégés have co-authored nine papers published in
refereed journals. Many protégés have presented papers and posters of
their summer research results at numerous regional, national, and
international professional conferences; two papers received best
presentation awards, one paper received an honorable mention; one poster
was awarded best poster in atmospheric science. Six alumni are in the
scientific workforce; three are enrolled in Ph.D. programs.

**Louis Stokes Alliances for Minority Participation (LSAMP):** Increased
efforts are underway to keep more of the Science, Technology,
Engineering and Mathematics (STEM) graduates from LSAMP institutions in
the graduate school pipeline. There is a Bachelor of Science/Master of
Science (BS/MS) program involving several institutions in the Atlanta
area: Emory University, Georgia Institute of Technology, Georgia State
University, Morehouse School of Medicine, and the colleges of the
Atlanta University Center. Other initiatives focus on providing better
linkage of the NSF-funded undergraduate programs with NSF-funded
graduate programs. Attention to making the early STEM curriculum more
student-friendly is underway in programs at several universities
including the one at North Carolina A & T State University in
Greensboro, NC, which focuses on tackling the three introductory
\"killer courses\" in chemistry, mathematics, and physics. Other
initiatives are taking place in the Historically Black Colleges and
Universities Undergraduate Program and include one at Jackson State
University to revise the teaching of calculus to include technology and
laboratory experiences. At Morgan State University the initiative
emphasizes research experiences for the STEM students.

The Advanced Technological Education (ATE) program contributes to
broadened participation by under-represented minorities, as well as
general improvement of technological education, through its focus on
community colleges. Community colleges are where a large fraction of
minority students begin their higher education. For example, the
American Association of Community Colleges is conducting a series of
activities designed to advance community colleges' leadership in STEM
education. These activities include organizing an annual conference for
ATE Principal Investigators; organizing a "Summit on Technological
Education" that will bring community college and business
representatives together to develop strategies to strengthen the
nation's technical workforce and faculty pipeline; and continuing a
successful mentoring program that helps community colleges establish new
programs in STEM by pairing them with colleges that have already
developed exemplary programs.

"Team-Mentoring: Catapulting Women through the **Glass Ceiling**" refers
to an innovative and very successful mentoring conference for 40 junior
women economists supported by NSF. The conference accomplished its goal
of improving the research, grant writing, networking, and life balancing
skills of the 40 junior women economists. It also produced conference
plans, supporting materials and a video that so far have been used to
hold four more mentoring conferences for women in economics. The
American Economic Association Summer Training Program is an intensive
eight and one-half week program designed to encourage and prepare
talented undergraduates for success in economics doctoral programs. This
is combined with a Scholarship Program that recruits, selects and funds
under-represented minority (Black, Hispanic and Native American)
participants who are U.S. citizens or permanent residents. Recent
statistics show that the Summer program has been very successful in
raising the numbers of minority students in strong graduate programs.

The **Gateway Coalition** alters engineering education from a focus on
course content to the development of human resources and the broader
experience in which individual curriculum parts are connected and
integrated. The scope of the program includes four major parts:
curriculum structure, human potential development, instructional
technology and methodology, and evaluation measures. Gateway efforts aim
at increasing the numbers of women, underrepresented minorities and the
disabled participating in engineering; and drawing engineering faculty
to a dedicated investment in the teaching of undergraduate students.
During the period in which the Gateway Coalition was established, the
retention of African American engineering students from the freshman to
the sophomore year in the Gateway Coalition universities increased from
67% in 1992 to 87% in the year 2000. For women students, this retention
increased from 75% to 90%, and for all engineering students retention
increased from 79% to 86%.

**Costa Rican Biodiversity Laboratory:** A collaborative project
involving the ALAS (Arthropods of La Selva) research team, the
Organization for Tropical Studies (OTS), and the Instituto Nacional de
Biodiversidad (INBio, Costa Rica) has produced an inventory of tropical
rainforest insects at the La Selva Biological Station, Costa Rica, (a
biodiversity laboratory) and trained a staff of parataxonomists drawn
from local citizens, who conduct research and give tours. ALAS is highly
visible to the public. The ALAS lab is a magnet for La Selva visitors,
including local school children, U.S. senators, and Latin American
decision-makers. ALAS was the subject of a New York Times (\'Science
Times\') article (Yoon 1995), a live telecast from La Selva by Turner
Educational Services, and a Nova-like segment on the Japanese Television
Workshop. ALAS has been featured in Costa Rican television programs and
newspapers.

**International Geotechnical Activities:** In addition to promoting the
internationally-linked engineering research activities, NSF has been
proactive in fostering international initiatives and collaborations-at-a
distance. An example is the geotechnical reconnaissance report on
Hyogoken-Nanbu earthquake. Eighteen U.S., Japanese, and Canadian
researchers were rapidly deployed in the area affected by the
Hyogoken-Nanbu earthquake to prepare a document that was of substantial
value to U.S. practitioners and researchers. The report was available on
the Web shortly after the earthquake. The work represented exceptional
international collaboration and provided invaluable reconnaissance
training for young researchers. They have become leaders of the
reconnaissance efforts that took place following major earthquakes in
Turkey, Taiwan, and Washington State.

David Dobson of Texas A & M University received the first **Felix Klein
prize** awarded by the European Mathematical Union. The award, made in
June 2000, recognized Dobson for his work to design light-diffracting
structures for use in devices such as laser instrumentation and optical
communications equipment. The early part of this work was supported by
an Industrial Postdoctoral Fellowship and by a Mathematical Sciences
Postdoctoral Research Fellowship, both funded by NSF.

NSF supports very high quality, field programs in **global
meteorology**. The Dynamics and Chemistry of Marine Stratocumulus
(DYCOMS) program was a study of cloud microphysics, boundary-layer
entrainment, and evaluation of the applicability of large-eddy
simulation modeling of the coastal marine boundary layer. In addition to
the US participants, DYCOMS included researchers from France and Poland.
The Physical Meteorology (PMET) also contributed to support of the
Cooperative Atmosphere-Surface Exchange Study-1999 (CASES-99)
observational program, led by the Army Research Office and conducted in
Kansas. CASES-99 sought to provide a testbed for obtaining high quality,
comprehensive boundary layer and hydrological measurements in a small
watershed in order to probe the structure of the nocturnal boundary
layer and to seek to close the water balance in a small watershed. In
addition to numerous US investigators, CASES-99 also involved European
scientists from Scandinavia, Spain and the Netherlands.

**International Plant Genome Collaboration:** One investigator was
involved in establishing workshops at the International Triticeae
Mapping Initiative, involving a variety of genome and Expressed Sequence
Tags (EST) projects related to the cereal crops, especially projects in
Europe and Australia. Two others have collaborations with the
International Rice Research Institute in the Philippines, and one of
these also has formal collaborations with groups in Japan and Brazil.
Another investigator heads up a project that has become the hub for
research on the model legume system Medicago truncatula. This activity
includes collaborations at the Centre National de Recherche
Scientifique/Institut National de la Recherche Agronomique (CNRS/INRA)
in France, the Agricultural University of Wageningen, the Netherlands.
Another investigator has active collaboration with scientists at the
Friedrich Miescher Institute in Basel, Switzerland.

Frederick Brooks of the University of North Carolina
(<http://www.cs.unc.edu/~brooks/>) received the **Turing award**, the
highest honor in computer science, in February 2000 for "landmark
contributions to computer architecture, operating systems, and software
engineering." Dr. Brooks is now working primarily in the areas of
computer graphics and supported in part by an NSF grant. This enables
him to work on "Real-Time Walkthroughs of Serious Synthetic
Environments" (<http://www.cs.unc.edu/~walk/>). The goal of this project
is to create interactive computer graphics systems that enable a viewer
to experience an architectural model by simulating a walk through of the
model. While Dr. Brooks supplies much expertise for the integration of
the system, other team members have made fundamental advances in
computer graphics. This includes collision detection -- for which Dinesh
Manocha of the University of North Carolina
(<http://www.cs.unc.edu/~dm/>) won the Best Paper award at Eurographics
in 1999 -- simplification of models for visualization, and image-based
rendering.

**Explore the Universe:** NSF is supporting the creation and
installation of a major new permanent exhibit, *Explore the Universe*,
in the Smithsonian Institution's National Air and Space Museum. This
exhibit, opened in September 2001, provide the 9 million annual visitors
to the Air and Space Museum with a perspective on how our understanding
of the Universe has changed over time as the tools we use to study it
have evolved. The exhibition makes use of a selection of artifacts,
working models, images, interactive videos and computer programs,
hands-on exhibits, and live demonstrations to explore scientists' view
of the Universe as well as how they use ground- and space-based
technology to study it.

**Nanoworld 'Picture Books':** To bring the nanoworld to everyone, the
University of Wisconsin Materials Research Science and Engineering
Center has developed a series of short web-based 'picture books'
accompanied by hands-on demonstrations that illustrate nanoscale
materials and devices. This work is the basis for a lead article in the
Smithsonian\'s *Muse* magazine Spring 2001 issue dedicated to
nanotechnology.

The American Association for Microbiology's Microbial Literacy
Collaborative produced a four-part TV series that was viewed by 1.6
million households each week. As of June 2000, twenty thousand copies of
this project's publication of hands-on activities, "**Meet the
Microbe**," had been distributed nationwide (over half to individuals
and organization upon specific request), and an additional 17,000
activities were downloaded from the
[www.microbeworld.org](../www.microbeworld.org) web site. This Website
received 689,620 hits and recorded visits by nearly 29,800 individuals
during its first month, and received over 3 million hits in its first
year.

For the last fifteen years, the NSF and the United States Geological
Survey (USGS), along with private foundations and universities, have
made significant investments in development of the Global Seismographic
Network (GSN) and its associated data collection facilities. **The
Incorporated Research Institutions for Seismology (IRIS)** Consortium,
in collaboration with the USGS, has begun to exploit this scientific
resource for educational purposes, by making data from the GSN
accessible to the general public through museum displays. By bringing
live research-quality seismic data over the Internet and broadcasting it
in museums, IRIS provides visitors with evidence that Earth's surface is
in motion. Accompanying handouts and classroom exercises provide the
visitor with follow-up educational materials. Currently, the IRIS
Education and Outreach Program has museum exhibits at the American
Museum of Natural History (NY), the Carnegie Museum of Natural History
(PA), the New Mexico Museum of Natural History (NM), and an exhibit on
tour with the Franklin Institute as part of their "Powers of Nature"
exhibit. Together, these exhibits reach approximately 9 million visitors
each year.

The **Thomas Edison Paper Project**, which NSF supported in the 1980s
and early 1990s, has pursued the development of extensive materials
accessible through the web. The Edison site now has more than a million
pages on-line (<http://edison.rutgers.edu/>). NSF also continues to
support the editing, annotating, and publishing of the ***Collected
Papers of Albert Einstein***, of which Volume 8 was published in January
2002. The other large-scale, long-running infrastructure project
supported by NSF is the publication of the **letters of Charles
Darwin**. In 2001 the project released Volume 12 of the letters,
covering 1864.

NSF **Informal Science Education (ISE)** programs are directed at
improving science, mathematics, and technical skills for citizens of all
ages by addressing how families learn in informal settings and by
involving parents as participants in children's science and math
education. Projects targeting early childhood learners also demonstrate
achievement of this indicator. ZOOM, a WGBH Educational Foundation media
project and BUSYTOWN, an Oregon Museum of Science and Industry museum
project, are examples of early childhood projects in science and
mathematics programs supported by NSF.

**National Virtual Observatory:** FY 2001 saw the beginning of
coordinated efforts to realize the National Virtual Observatory (NVO).
This project, which received a strong recommendation from the recent
National Academy of Sciences Decadal Survey, will federate astronomical
data sets and establish them as a common resource for both researchers
and the public. The project will focus not only on the archives, but
also on establishing the protocols, standards, and tools that will
permit the large astronomical datasets of the future to be fully
utilized. Coordinated efforts are also underway at collaborating
institutions to develop archives, visualization tools, and related
resources. The first concept was developed with the help of an NSF Small
Grant for Exploratory Research (SGER) award. NASA and NSF will be
cooperating in this activity.

The report ***Academic Excellence for Urban Students: Their
Accomplishments in Science and Mathematics*** (Kim, 2001)*,* an
evaluative study of 22 of the Urban Systemic Initiative districts funded
between 1994-1999, indicated that the urban program has been a catalyst
for large-scale systemic change directed towards improving the science
and mathematics achievement of all students. Further, the report
presented evidence that the greatest gains were in districts that had
participated in the Urban Systemic Initiative (USI) program for the
longest period of time. Assessment results showed that USI students made
gains in science and mathematics achievement, while reducing achievement
gaps among racial/ethnic groups. Moreover, students in these districts
substantially improved their enrollment rates in advanced science and
mathematics courses. Additionally underrepresented minority students
made even greater gains than their peers during the same period,
resulting in reduced enrollment disparities in advanced courses. The
study provides credible evidence that the implementation of a
standards-based curriculum and instruction, aligned assessment
practices, and appropriate professional development are key to an
increase in student achievement. In addition, the results show that the
convergence of resources, a strong leadership structure, and effective
partners were also critical to the improvement in student performance.
The study also concludes that it takes 7-10 years to bring about
substantial improvement in systemic reform that may lead to the gains
cited in the report.

The **Statewide Systemic Initiatives (SSI) Program** created
"extraordinary activity in terms of very large statewide professional
development efforts" and statewide curricular standards focused on more
ambitious teaching and learning for children. In particular, there is
evidence that infrastructures were created in state-level organizations
to support this teaching and learning as well. The presence of such an
infrastructure increases the likelihood that gains in student
achievement will continue.

The **Centers for Learning and Teaching (CLT) Program** contributes in a
significant way to the NSF's "enhancing instructional workforce"
emphasis area. These centers offer a new approach to teacher education
that responds to needs for increasing the ability of practicing teachers
to deliver standards-based instruction; rebuilding and diversifying the
national infrastructure for science, mathematics, and technology
education; facilitating workforce induction/retention during initial
years of service; and strengthening linkages between pre-/in-service
teacher education. These large-scale projects are closely linked to K-12
school districts and required to build on shared expertise of local
education agencies, institutions of higher education, and the informal
science community. They link K-12, higher education, and/or informal
science education performers to provide a systemic, K-graduate school
approach for educating the instructional workforce in an environment of
research and practice.

Two prototype centers were funded in FY 2000 and five additional full
centers were funded in FY 2001. In addition, six Developmental grants
were funded to build related institutional infrastructures.

The **NSF Graduate Teaching Fellows in K-12 Education** **Program**
places graduate and advanced undergraduate students in K-12 schools to
serve as science and mathematics content resources for teachers. The
program supports the fellowships and appropriate training necessary for
these fellows to enter the schools, but it is *not* a teacher
preparation program. The primary objective of the program is to provide
fellowships to highly qualified graduate and advanced undergraduate
students in science, mathematics, and engineering and technology
disciplines to achieve advanced degrees, *and* to contribute toward the
improvement of the nation's schools, while introducing K-12 students and
teachers to active researchers. This is part of a comprehensive approach
to workforce development that reaches from grade school through graduate
school. The expected outcomes of this initiative are: 1) improved
communication and instructional skills for the Fellows; 2) professional
development opportunities and content gain for K-12 teachers; 3)
enriched learning by K-12 students; and 4) strengthened partnerships
between institutions of higher education and local school districts. The
GK-12 program seeks, in the words of one observer, to create \"citizen
scientists\" for the 21^st^ Century.

The AMANDA/IceCube project supported by the Graduate Teaching Fellows in
K-12 Education Program (GK-12) is an exciting example of bringing
scientific inquiry into the classroom. AMANDA (Antarctic Muon and
Neutrino Detection Array) observatory located at the South Pole helps
researchers track sub-atomic particles (neutrinos) as they pass by this
array located 1500 meters below the polar ice cap. One of the teams
supported by the GK-12 project integrates middle and high school
education with the astrophysical research community involved with
AMANDA. Graduate students, researchers, and teachers team up to develop
a curriculum utilizing information generated by AMANDA. Through the
"Astronomy in the Ice" course, teachers receive the astrophysical
background they need to teach their students about AMANDA and related
physical sciences. Throughout the year, graduate students serve as
resource persons to schools while they establish communication networks
with teachers and researchers. The GK-12 project couples education with
the excitement of discovery and the understanding of science generated
by the AMANDA/IceCube project.

The NSF initiative for *Tribal Colleges and Universities Program*[^9]
encourages Native Americans to pursue information technology and other
science and technology fields of study, as well as increase the capacity
of tribal colleges to offer relevant science and technology courses and
enhance K-12 education in feeder school systems.

The project "Sustained Economic Growth of the **Oglala Lakota Nation**
through Development of the Technological Infrastructure" is a
multifaceted approach to developing the technological infrastructure of
the institution and the reservation as a whole. Collaborations with
several federal and state certified labs will provide the capacity to
conduct research while strengthening curricula and developing new
laboratory classes in several areas.

Programs are in place to foster gender-equity in the STEM disciplines
such as the one at Arizona State University which involves guidance
counselors who work with \"**talented at-risk girls**\" to improve self
confidence and encourage them to enter the STEM fields. This program
reached over 400 students at forty-eight schools including more than 100
Native Americans and 130 Mexican Americans.

The Rural Systemic Initiatives (RSI) Program has explicitly addressed
the needs of **Native Americans** through the funding of several
initiatives. An excellent example is the reconfiguration of the
collaborative High Plains RSI into several individually funded RSI
Tribal Colleges. By reallocating resources amongst the tribal colleges,
the RSI has increased support to the rural systemic infrastructure
serving Native American populations in the high plains. Another example
would be the spin-off of the Navajo Nations RSI from the
Utah-Colorado-Arizona-New Mexico (UCAN) RSI.

Two workshops brought together **Native experts** on sea-ice conditions
that have affected whale hunting in the Arctic and scientists with their
instrumental data, anthropological records, and sea-ice models. The
symposia revealed a wealth of knowledge from both the scientific
approach and Traditional Ecological Knowledge (TEK) that, when utilized
together, enhance understanding of the impacts of environmental change
on subsistence and cultural hunting practices.

NSF continuous its efforts to broaden community and public access to the
benefits of scientific research. Several activities supported in 2001
were aimed directly at documenting **traditional indigenous knowledge**,
cultural heritage, craft and subsistence practices for its sharing with
and modern use by polar residents. Scientific knowledge collected by
researchers and shared with communities increases awareness of modern
economic, occupational, and social challenges, advances the ability to
support public and educational initiatives, and improves potential
responses in case of natural catastrophes and/or social losses.

NSF statistics show that large fractions of the students involved in the
Engineering Research Centers (ERC) for FY 2000 were female and
**underrepresented minorities** (URM). The 600 pre-college students
included 48% female and 36% URM students; the 4000 undergraduate
students included 29% female and 12% URM students; the 2200 graduate
students included 22% female and 5% URM students; and the 344 Research
Experiences for Undergraduate (REU) students included 39% female and 35%
URM students.

With support from two NSF awards, researchers are addressing retention
and participation of **traditionally underrepresented groups** in
computing. Developing a framework involving undergraduate and graduate
students in research, they have created laboratories to support research
in neuro-fuzzy systems, parallel and distributed systems, signal
processing and communication systems, software engineering, and
theoretical applications. Students involved in this study include 73
graduate students (12 Ph.D. students); 102 undergraduate students; 136
students from underrepresented groups (38 female). To date 61 students
graduated with BS; 38 students graduated with MS; two students graduated
with a Ph.D. and 31 undergraduate students continued to graduate school.
The distribution of publications, talks, and awards over the five years
is as follows: over 150 research publications; over 100 research
publications (journal and conferences) with students as co-authors; 23
publications and talks on the Affinity model; 66 student presentations
at student conferences; 25 student awards and recognition.

The **Partnerships for Innovation (PFI) Program** focuses on connections
between new knowledge created in the discovery process to learning and
innovation. The goals of the program are: (1) to stimulate the
transformation of knowledge created by the national research and
education enterprise into innovations that create new wealth, build
strong local, regional and national economies and improve the national
well‑being; (2) to broaden the participation of all types of academic
institutions and all citizens in NSF activities to more fully meet the
broad workforce needs of the national innovation enterprise; and (3) to
create the enabling infrastructure necessary to foster and sustain
innovation in the long term.

The \$14 million in grants awarded in FY 2000 have been supplemented
with over \$7 million for 12 new grants[^10] in FY 2001 to cover
projects in 11 states involving more than 150 partner organizations. The
lead institutions are selected to act as catalysts in helping their
surrounding communities transform research-based knowledge into
innovations that create opportunities for new wealth and a broader
economic base that benefit communities and the nation at large. Examples
of innovation that the grants are meant to foster might involve
development of advanced new technologies to boost local economic growth.
A university may serve as the research base, then incorporate its
results into the corporate arena through knowledge and technology
transfer. Corporate partners may develop the industrial processes for
the innovation or product. Local governments or other non-profit
activities may become a third leg in the partnership process by creating
the climate for new businesses or funding the marketing of the product
or innovation. Concurrently, a science and engineering workforce focused
on innovation is created.

Advanced Technological Education (ATE) Centers of Excellence focus on
systemic approaches to technician education, usually within a specific
discipline. ATE Articulation Partnerships bring two-year colleges and
four-year colleges and universities together to enhance two-year college
programs for prospective K-12 teachers and for students who continue
their education in four-year degree programs in science, mathematics,
engineering, or technology. Fifteen Centers were active during FY 2001.
In FY 2001 the ATE program supported 65 new projects and over 140
projects that were begun in previous years remained active. Cooperative
efforts among ATE participants assure that the programs have a national
impact. For example, the extensive collaborations of ATE projects with
private industry have resulted in "skill standards" for technicians in
particular fields and have enabled projects to develop real-world,
workplace relevant educational materials and programs. In FY 2001 the
program introduced two new emphases: (1) regional centers for
information technology or manufacturing education to strengthen the
workforce in the areas critical to the US economy and (2) "articulation
partnerships" to facilitate the transition from two-year to four-year
colleges for students preparing for technology careers or to become K-12
teachers.

The **Northwest Center for Emerging Technologies**, an ATE Center of
Excellence at Bellevue Community College (Bellevue, Washington) is
developing and refining industry-validated skill standards and
associated curricula for information technology. The center's strategic
partners include Microsoft, Boeing, and CompTIA (the Computing
Technology Industry Association). In 1999--2000, those skill standards
influenced or informed the education of approximately 78,000 students
nationwide. In the same period, over 500 students took classes using
courseware that the center developed in partnership with Dryden Online,
and the center sold or licensed over 1,000 copies of its internally
developed curricular products, which were used in the instruction of
over 50,000 students. Microsoft recently produced a video profiling how
the center stays on the cutting edge of information technology (IT)
education.

The goal of "**Digital Arts and Sciences**" at the University of Florida
is to train students to acquire a hybrid-knowledge of computer
engineering and the arts, enabling them to understand the formalism of
visualization and the practicality of human communications that deal
with aesthetic interpretation. This enables students to work effectively
in production-oriented teams focused on education, interactive games,
scientific and engineering visualization, software engineering, and
video production. Research is integrated in an Aesthetic Computing
course and a series of Digital World Production Studio courses to the
curriculum. Fine arts as well as computer science and engineering
students will take these courses. "Aesthetic Computing" uses genres and
styles in fine art as metaphors for formal and diagrammatically rendered
model structures commonly found in computing, including automata, data
flow graphs, data models, and the comprehensive Unified Modeling
Language (UML). This work involves areas generally regarded outside the
sphere of computer science, including semiotics, linguistics, analogy,
metaphor, and the arts.

**PERFORMANCE GOAL III-1b: Over 80 percent of schools participating in
systemic initiative programs will (1) implement a standards-based
curriculum in science and mathematics; (2) further professional
development of the instructional workforce; and (3) improve student
achievement on a selected battery of tests, after three years of NSF
support.**

 **Goal Not Achieved**

NSF manages a portfolio of programs that encourages and facilitates
coordinated approaches to systemic, standards-based reform of science,
mathematics, and technology (SMET) education. Systemic reform relies on
partnerships to identify needs, articulate visions, and develop goals,
strategies, and activities for improvement of targeted areas. Although
each systemic initiative is unique in its approach, all must begin as a
collaborative effort among individuals and organizations that are
committed to requiring high expectations for all students through
challenging educational opportunities. Systemic initiatives cultivate
coordination within cities, states, rural areas, school systems, and
other organizations involved with education.

**RESULTS:** We did not achieve this goal in FY 2001. The curriculum,
instructional workforce, and improved achievement in science components
of the goal were successful. However, less than 80% of schools met the
goal of improved student achievement in mathematics. Forty-seven
Systemic Initiative projects implemented mathematics and science
standards-based curriculum in 89% of the participating schools and
provided professional development for more than 226,900 teachers. The
Systemic Initiative projects reported improved student achievement in
math in 74% of the 6,255 schools and improved student performance in
science in 80% of the 4,082 schools using the same assessments for the
last 3 years.

[FY 1999 Result:]{.underline} In 1999, 46 NSF-sponsored projects
implemented mathematics and science standards-based curricula in over
81% of participating schools, and provided professional development for
more than 156,000 teachers. All participating educational systems
demonstrated some level of improvement in student achievement in
mathematics and science.

[FY 2000 Result:]{.underline} In 2000, 47 Systemic Initiative projects
implemented mathematics and science standards in over 80% of the
participating schools and provided professional development for more
than 214,792 teachers. The Systemic Initiative projects reported
improved student achievement in math in 81% of the 4,187 schools and
improved student performance in science in 86% of the 2,474 schools
using the same assessments for the last 3 years.

**Why we did not achieve this Goal:** No single factor has been
identified that explains the drop in performance on student achievement
relative to FY 2000 and FY 1999. Fluctuations in performance from year
to year are expected, since there are differences in which schools are
included within projects, and projects are able to use their own
criteria for what constitutes an increase in student achievement. Review
of performance data by NSF indicated that the goal for student
achievement was not met for mathematics at some sites because, during
the year, they adopted a more rigorous definition of increased student
achievement.

**STEPS WE WILL TAKE IN FY 2002 TO ACHIEVE THIS GOAL:** In FY 2002,
appropriate technical assistance will be provided to schools not meeting
the goal.

**Implications for the FY 2002 Performance Plan:** This goal will be
maintained in FY 2002.

**Performance Goal III-1c: Through systemic initiatives and related
teacher enhancement programs, NSF will provide intensive professional
development experiences for at least 65,000 pre-college teachers.**

**** **Goal Achieved**

Teacher professional development is a core strategy used by
EHR-supported projects to promote reform. For example, in the Saint
Louis Systemic Initiative schools, more than 75% (1,239 of 1,600) of the
science and mathematics teachers received intensive professional
development during the 1999-2000 school year.

[]{.mark}

**RESULTS:** In school year 1999-2000, EHR awards provided intensive
professional development (60 hours or more) to a total of 79,000
teachers, exceeding substantially the GPRA goal of 65,000.

  ------------------------- ----------- ----------- ------------------ ----------
  **Teacher professional                                               
  development**                                                        

                            **FY 1999** **FY 2000** **FY 2001**        **FY
                                                                       2002**

  **Goal**                  \>65,000    \>65,000    \>65,000           N/A

  **Actual Number of        82,400      89,700      **79,000**[^11]   
  Teachers**                                                           
  ------------------------- ----------- ----------- ------------------ ----------

**Implications for the FY 2002 Performance Plan:** This performance goal
is not part of our FY 2002 Performance Plan. For FY 2002 NSF has
reapportioned a substantial amount of the funds for the Systemic
Initiatives to support the new Presidential Math and Science Partnership
(MSP) activity. No new competitions or awards are anticipated under the
Systemic programs. A goal related to the MSP has been included in the FY
2003 GPRA Performance Plan.

**NSF OUTCOME GOALS**

# 

**B. IDEAS**

![](media/image9.wmf){width="0.7in" height="0.5098425196850394in"}

**PERFORMANCE OUTCOME GOAL III-2: Enabling "discovery across the
frontier of science and engineering, connected to learning, innovation,
and service to society."**

**** **Goal Achieved**

NSF invests in ideas to provide a deep and broad fundamental science and
engineering knowledge base. Investments in ideas support cutting-edge
research that yields new and important discoveries and promote the
development of new knowledge and techniques within and across
traditional boundaries. The results of NSF-funded research and education
projects provide a rich foundation for broad and useful applications of
knowledge and the development of new technologies. Support in this area
also promotes the education and training of the next generation of
scientists and engineers by providing them with an opportunity to
participate in discovery-oriented projects.

This is a new goal in FY 2001. Our performance is successful when, *in
the aggregate*, results reported in the period demonstrate significant
achievement for one or more of the following indicators:

-   A robust and growing fundamental knowledge base that enhances
    progress in all science and engineering areas including the science
    of learning;

-   Discoveries that advance the frontiers of science, engineering, and
    technology;

-   Partnerships connecting discovery to innovation, learning, and
    societal advancement;

-   Research and education processes that are synergistic.

**RESULT:** External experts provided examples of significant
achievement in reports they developed during FY 2001 reporting. A sample
of these is provided for each of the performance indicators and areas of
emphasis for this goal.

**Implications for the FY 2002 Performance Plan:** This goal will be
continued in FY 2002. The performance indicators related to the Ideas
goal have been expanded and modified to appropriately reflect the
breadth of NSF activities (see Section X. for details).

One study that has captured significant scientific attention, as well as
public interest, is the confirmation of viable microorganisms throughout
the 4-km long ice column overlying **Lake Vostok**, **Antarctica**. This
has implications for the presence of life in the lake itself. NSF
supports a diversity of ecological investigations including Arctic bird
adaptations, arctic tree growth and needle retention, establishing the
metabolic rates of cold-adapted organisms, feedbacks between vegetation
and climate change, and the phylogenetic composition of polar bacteria.

The 1987 Whittier Narrows earthquake revealed the need to improve
understanding of the **impact of major natural disasters on the business
sector**. Most of the existing research had focused on the economic
impacts of disasters on households and families. The NSF-supported
principal investigators were able to carry out a large-scale survey of
businesses in Memphis/Shelby County, TN on business owners' perception
of earthquake hazard, on their judgment of vulnerability to and effects
of lifeline disruptions, and on their preparedness measures. Similar
data was collected from the Northridge earthquake. Another NSF grant
extended such studies to Santa Cruz County, CA and Dade County, FL.
Overall, these studies have produced data on nearly 5,000 businesses of
all sizes and types, comprising the largest database on business
disaster readiness and effects in the world.

Advanced techniques have been developed for the communications,
coordination, and vision of autonomous **multi-robot systems**. This
allows robots to coordinate with each other in varying environments.
When obstacles are presented in their path, the multi-robot systems can
rearrange themselves to navigate around the object and then return to
their original formation. For example, three small robots can coordinate
themselves to sense, coordinate, and move a large object that could not
be moved by a single robot.

What are the **predictors of children's learning interests** and how
does parental involvement influence learning? Findings indicate specific
parental traits that influence children's learning: Most importantly,
children who develop focused encyclopedic play interests tend to have
parents that value family discussions. Why does the value of family
discussions predict children's interests? Children cannot easily learn
about some domains without adult help; they need parents to talk with
them about the domain, to read them relevant books, to participate in
their play, and to provide them with relevant resources (toys, models,
etc). Preschoolers cannot acquire this kind of knowledge on their own.
The impact of **adult investment in child learning** is not
insignificant: Children who develop focused encyclopedic play interests
are more likely to be boys (by a factor of 5) and to demonstrate higher
cognitive skills (particularly verbal).

An NSF-supported project has developed a process for the conversion of
silicon carbide (SiC) to crystalline diamond at ambient pressure. This
has direct application to the surface science of microelectromechanical
systems (**MEMS**). The entire technological field of MEMS represents an
area that is gaining momentum with the commercialization of these
devices in applications in communications, transportation and aerospace.
An article describing this work was published in *Nature* and received
substantial attention from the news media.

A suite of investigations has examined the stability of the **West
Antarctic Ice Sheet (WAIS)**, a potential contributor to rising sea
levels. Results from glaciological modeling and glacial geologic
research have highlighted a rapid retreat of the WAIS grounding line in
the Ross Sea since the end the last glacial stage. Glacial geologists
have developed a history of grounding line retreat for the WAIS. Results
concerning the retreat of the WAIS have been of widespread interest,
particularly for those who are studying past and present changes in sea
level, ocean circulation, and atmospheric circulation.

Predicting the properties of materials from first principles (i.e. from
the behavior of their atoms and molecules) is an important and
challenging field. The challenge comes from the range of scales
required - from electrons to clusters large enough to be seen. To span
this range, researchers supported by NSF have developed numerical
algorithms and a software infrastructure to implement hierarchical and
adaptive methods to concentrate the solution on areas of greatest
interest. The algorithms include techniques that improved previous
techniques by 100-fold in terms of computer time and memory. The
software infrastructure included the well-known Kernel Lattice
Parallelism (KeLP) system, which manages communications among the memory
hierarchies of parallel computers. KeLP is a framework for implementing
portable scientific applications on distributed memory parallel
computers. It is intended for applications with special needs, in
particular, that adapt to data-dependent or hardware dependent
conditions at run time. KeLP is currently used in full-scale
applications including subsurface modeling, turbulence studies, and
first principles simulation of real materials. In addition to materials
design, KeLP has been applied to a wide range of partial differential
equation solutions.

Neutron stars are born in supernova explosions when approximately one
solar mass of burnt nuclear matter in the core collapses from about the
size of the Earth to the size of an average city. The collapse of matter
into this dense core can leave the **newborn neutron star** spinning
extremely rapidly, some 1,000 revolutions per second. NSF-supported
researchers have carried out the most detailed computational modeling
ever done of the stability of these stars. In addition to surprising new
insights into neutron stars, these simulations are yielding a clear
"signature" to search for as the new Laser Interferometer
Gravitational-Wave Observatory (LIGO) tries to detect gravity waves
directly for the first time.

Researchers have, through the use of mathematical modeling, devised a
simple and inexpensive method for the prevention of **Chagas disease**.
A model predicted how four populations-bugs, chickens, dogs and
people-interact with the protozoan parasite *Trypanosoma cruzi* each
season in an individual household. The model allowed the researchers to
determine that the practice of keeping chickens and dogs inside the
house increased the population of blood-feeding bugs called triatomines,
also known as the "kissing bug," which in turn increase the parasite
population that causes Chagas disease. By excluding infected animals
from sleeping areas and spraying insecticides on a predetermined
schedule, the people could virtually eliminate transmission of the
parasite.

**Cloning clock genes:** In circadian biology, largely supported and
fostered by NSF, "clock" genes have been cloned from several species,
and the mechanisms by which circadian rhythms are generated are now
being defined. In 1998 a Science magazine cited this research area as
one in which major advances were made, and this research has now risen
to the public consciousness. These areas, largely initiated by and
nurtured by NSF, have become incorporated into the common knowledge base
of undergraduate biology and ultimately many of the important advances
are incorporated into the public consciousness.

**The Arctic Oscillation and global change:** The Arctic Oscillation, an
annular mode of atmospheric circulation, has been linked in recent years
to a variety of issues in the detection and diagnosis of global change.
NSF-supported investigators have played leading roles in the study of
this annular mode, beginning with Wallace and Thompson\'s introduction
of the term \"Arctic Oscillation\" (or \"AO\") in the late 1990s. The AO
is strongest in the Atlantic sector of the Northern Hemisphere and has
direct effects on the climate of Eurasia, eastern North America, and the
Arctic. A large portion of decadal climate variability, including
large-scale temperature and precipitation anomalies in the Northern
Hemisphere during the 1980s and 1990s, is associated with the AO. There
is evidence of a similar mode in the Southern Hemisphere, where this
mode appears to have played a role in recent trends in the atmospheric
and sea ice conditions. It has been shown that the coherent vertical
structure and phasing represents the strongest evidence to data of a
dynamical coupling between the upper atmosphere and near-surface
climate. There are tantalizing indications that stratospheric variations
can precede surface variations, offering potential predictive
possibilities for short-term climate variations, although the robustness
and physical basis of this linkage require further investigation. In
addition, there are suggestions that there may be a linkage between
stratospheric chemistry (possibly including decreases of stratospheric
ozone) and the troposphere via the stratospheric circulation.

Research groups supported by NSF pioneered the use of high precision
measurements of the oxygen/nitrogen (O2/N2) ratio in air to study the
biogeochemical cycles of carbon and oxygen. The ability to monitor these
fluxes in real-time is clearly a critical asset in terms of developing
the capability for managing a national or global carbon emission
strategy. Using two independent techniques, mass spectrometry and
interferometry, they have documented regional and global variations in
O2/N2 and explored the use of this data to place constraints on the
fluxes of carbon dioxide (CO2) and oxygen between the atmosphere, ocean,
and terrestrial biosphere. The combination of atmospheric O2/N2 and CO2
provides **a powerful tool to study the fate of fossil fuel.** This is
true of CO2 because it allows discrimination between uptake via
inorganic dissolution in the oceans (which does not release oxygen) and
photosynthesis (which does release oxygen). The record to date has
revealed significant interannual variability in the magnitude of the
terrestrial biospheric sink for CO2. The ability to detect interannual
variability in these fluxes is invaluable in terms of understanding the
factors controlling them. For example, 1998 appears to be an anomalous
year during which apparently all of the anthropogenically (man-made)
emitted CO2 remained in the atmosphere and O2 fell abruptly. This
suggests that more CO2 than usual was released by the land biosphere.
This line of research has a significant impact on several fields of
research, including climate, ecosystem studies, and oceanography.

**Environmental factors in human health:** The Center for Integrated
Studies of the Human Dimensions of Global Change (CIS‑HDGC) at Carnegie
Mellon University has been exploring the ramifications of a broad range
of interactions among human behavior, engineered systems, and natural
processes through refinement, adaptation, and expansion of a highly
sophisticated, integrated assessment model. In one application designed
to better understand the role of environmental factors on human health,
CIS‑HDGC investigators developed and tested a model for assessing risks
associated with *Cryptosporidium*, the water-borne parasite that
sickened hundreds of thousands in the Milwaukee area during the early
1990s. The model integrates processes associated with engineering (water
treatment technologies), ecology (land-use and land-cover patterns),
biology, (proliferation of and testing for parasites), public health
(surveillance and notification), and psychology (public perceptions and
responses). One analysis using this model demonstrated that under
typical conditions in the U.S., traditional responses to
*Cryptosporidium* outbreaks calling for the boiling of drinking water
are ineffective because most vulnerable people have been exposed before
the problem is detected. Analyses designed to assess whether climate
changes might increase threats to human populations demonstrated that
changes in physical conditions were relatively unimportant as long as
effective public health systems and procedures remained in place. As
with related analyses of the risks associated with air pollution, the
*Cryptosporidium* studies showed the importance of effectively
incorporating social institutions into predictive models.

NSF has supported two major surveys of approximately 25,000 Internet
users in late 1997 and late 1998 to examine **the sensitivity of
Internet commerce to tax rates**. Tax sensitivity is high and is not
falling despite the rapidly increasing number of new users because as
new users gain experience, particularly young Internet users, their tax
sensitivity appears to rise substantially. Internet commerce as a whole
continues to be highly sensitive to tax rates and would fall
significantly if existing sales taxes were enforced online. This
research suggests that applying existing sales taxes to Internet
commerce might reduce the number of online buyers by up to 24%. But the
data also suggest that the potential sales tax revenue losses from
continued exemption of Internet commerce would be quite modest over the
next several years. So, taxing Internet commerce would not yield much
government revenue and would substantially setback and slow the growth
of e-commerce ([http://gsbadg.uchicago.edu/]{.underline}).

An innovative proposal supported by NSF deals with the design of **palm
size airplanes**, among the world's smallest, using flexible wings made
of composite materials. The flight characteristics of these airplanes
are similar to biological counterparts. The aircraft can be used in
unmanned surveillance and reconnaissance missions.

An NSF award has supported groundbreaking research into the signal
transduction pathway controlling responses to the hormone ethylene in
plants. This research has used mutational and positional cloning
approaches to identify genes that are involved in **ethylene
signaling**. Ethylene signaling is now better understood than any other
plant hormone signal transduction pathway. Responses to hormones such as
ethylene affect plant growth and development, and their ability to
adjust to adverse environmental conditions, and by extension to crop
quality.

[]{.mark}

**Classroom 2000:** How does one substantially reduce the human input
for creating and accessing large collections of multimedia, particularly
multimedia created by capturing what occurs in an environment? The
existing software system used as the starting point for this
investigation is **Classroom 2000**, which is designed to capture what
happens in classrooms, meetings, and offices. Classroom 2000 integrates
and synchronizes multiple streams of captured text, images, handwritten
annotations, audio, and video. In a sense, it automates note taking for
a lecture or meeting. The research challenge is to make sense of this
flood of captured data. The project explores how the output of Classroom
2000 can be automatically structured, segmented, indexed, and linked.
Machine learning and statistical approaches to language are employed to
understand the captured data. Techniques from computational perception
are used to find structure in the captured data. An important component
of this research is an experimental analysis of the software system
being built. It is expected that this research will have a dramatic
impact on how humans work and learn, as the developed technology will
aid humans by capturing and making accessible what occurs in an
environment.

Most physicists believe that general relativity ceases to provide a good
description of the physical reality in the vicinity of the Big Bang. On
general physical grounds it is clear that such a theory must incorporate
effects not only of general relativity but also of quantum physics.
NSF-supported researchers, attempting to unify general relativity and
quantum physics, have found that at the tiniest scale conceivable today,
called the Planck length (10^-33^ cm), the continuum picture of space
breaks down and has to be replaced by a precise 'polymer-like geometry.'
While at the laboratory scales (10^-18^ cm and above) this true geometry
can be approximated by a continuum, the approximation fails miserably
near the Planck scale. As the universe expands, its volume does not
change continuously but only in discrete steps. Near the Big Bang one
must abandon the use of differential equations on which most of physics
is based and replace them by more fundamental difference equations
describing the 'true' time-evolution. Once this is done, infinities
disappear and regular physics is restored without any ad-hoc
assumptions. While general relativity is an excellent approximation for
today's universe, **space-time 'dissolves' near the Big Bang**.
Einstein's deterministic, geometric universe has to be replaced by a
specific, probabilistic universe built from polymer geometry.

NSF's **Children's Research Initiative** (CRI) is providing five years
of support for research centers that are developing new thrusts in the
field of integrative developmental science. Individually, the centers
represent leading edge research about children and media, developmental
science, and the integration and dissemination of developmental science
to inform both research and policy.

The *Research Center on Children and Media* is a collaboration of five
research entities across four universities: Georgetown University,
Northwestern University, University of Texas -- Austin, and University
of California-Los Angeles. The research activities of this center
emphasize the types and impacts of emergent digital media on children;
interactive media experiences on children's social and academic
adjustment; how digital media impact learning; and influences of media
on the developing brain.

The *North Carolina Child Development Research Collaborative (CDRC)*
builds upon multidisciplinary activities across departments at the
University of North Carolina-Chapel Hill, Duke University, University of
North Carolina-Greensboro, and North Carolina State University.
Investigators from diverse areas of developmental inquiry -- health,
nursing, social work, education, and developmental psychology -- form
the CDRC and have a common objective of developing an integrative model
of developmental science. Their interdisciplinary approach to human
development encompasses the main levels of analysis from the cultural
and societal levels studied by anthropologists and sociologists to the
neural and genetic levels studied by biologists and neuroscientists.

Development and use of a mixed-lubrication laboratory for
university-industry collaboration and education has provided work
experience for graduate students through partnerships with a major oil
company, an automobile firm, a manufacturer, and a federal agency. In
addition to enhancing the professional development of the Principal
Investigator (PI), one post-doc, three graduate students and four
undergraduates were involved in this project. There was interaction with
NASA Marshall Space Flight Center, Mobil Technology Company, Ford Motor
Company, Ilmor Engineering, Ltd and The Timken Company. These industrial
partners provided the matching funds, enabling the PI and his students
to leverage the NSF-supported research. Experiments were conducted in
the university laboratory and at some of the industrial sites. A senior
undergraduate/graduate course in tribology (the study of the effects of
friction on moving machine parts) was developed and taught several
times. Four industry experts were invited to give guest lectures to the
class. The NSF funds were used as seed monies to stimulate the
university/industry collaborations. Several students designed and built
apparatus and computer programs that are now used at some of the
industrial laboratories. The research collaborations led to links
between theory and practice on how to best balance calculation accuracy
and application practicality. Eleven conference and journal papers
produced reflect this concept. The benefit of the award goes beyond the
project period, since the PI still interacts with some of the industrial
partners and is creating relations with new partners. Some of these new
relations reflect ideas that were "seeded" during the NSF-funded project
but came to fruition after it was completed.

In Antarctica, a recently recovered ice core from Siple Dome reveals a
record of **rapid climate change** that appears to be correlated
strongly with northern hemisphere changes. These data strongly suggest
global links in the climate system. These data are also crucial for
assessments of the state of the climate system, such as the recent
Intergovernmental Panel on Climate Change (IPCC) 2001 Report, that
provide information for policy-makers.

**Icing models for aircraft:** An NSF-supported Engineering Research
Center at Mississippi State University working in collaboration with the
Centers for Research Excellence in Science and Technology (CREST)
project at Tennessee State University is engaged in the development,
application, and evaluation of one-dimensional icing models with
important implications for the safe operation of commercial aircraft.

NSF supports many atmospheric research programs that involve large
numbers of partnerships -- both among multi-disciplinary groups and
among international researchers and institutions. The **U.S. Weather
Research Program** (USWRP) is one such program. NSF, the National
Oceanic and Atmospheric Administration (NOAA), the Federal Aviation
Administration (FAA), and the Naval Research Laboratory (NRL) jointly
support the USWRP. Its overarching objective is to conduct fundamental
and applied research that will lead to improvements in various aspects
of weather forecasting, such as hurricane landfall and quantitative
precipitation prediction. The USWRP supports research in two
categories\--physical scientific research and identification of the
societal needs for forecast information. The latter seeks to identify
societal impacts of weather and weather information, determine the types
of weather products that users require, and better understand how these
products are or might be used. This, in turn, provides input and
direction to the scope and focus of certain of the physical research
studies.

A strong emphasis in the Experimental Program to Stimulate Competitive
Research (EPSCoR) is the development of partnerships connecting
discovery to innovation, learning, and societal advancement. A major
concern of EPSCoR states is economic development and many EPSCoR
projects produce results that are quickly transferred to industrial
partners or result in new small businesses. One example involves the
University of Alabama - Huntsville, where a team is developing an
integrated research environment for intermeshed optoelectronics, thus
allowing the study of systems in which optical and electronic elements
are intermeshed in close physical proximity. Through the acquisition of
a field emission scanning electron microscope the microfabrication
facility is being expanded into a nanofabrication facility. The research
and facilities supported by this project have resulted in over \$1
million in fabrication contracts and \$2 million in private cash
donations for expansion of the cleanroom facility and purchase of
additional equipment to expand nano- and microfabrication capabilities.

NSF supports a number of **Collaboratives to Integrate Research and
Education (CIRE)**. These Collaboratives are designed to establish
long-term research and education relationships between minority-serving
institutions and NSF-supported facilities and centers by using both the
human and practical resources of the facilities and centers to establish
joint research programs and sponsor summer exchange programs. For
example, the University of Puerto Rico at Humacao, and the Materials
Research Science and Engineering Center (MRSEC) at the University of
Pennsylvania are such a collaborative. A new Masters program in
materials physics, the first graduate program at UPR-Humacao, has been
developed through the collaboration. The annual University of
Pennsylvania-UPR CIRE meeting was held in Puerto Rico in October
2000[^12]. The meeting featured Alan MacDiarmid who gave an
inspirational talk to over 400 undergraduate and high school students in
Humacao. This was the first meeting MacDiarmid attended after he was
recognized with the 2000 Nobel Prize in Chemistry earlier that month.

In 2001 NSF offered its third **Biology Training Course in Antarctica**.
This past year an international graduate-level training course entitled
\"Integrative Biology and Adaptation of Antarctic Marine Organisms" was
offered. Taught in Antarctica for one month during the austral summer,
the participants included 22 individuals from six countries (18 graduate
students and 4 postdoctoral researchers). The goals for the course were
to introduce students to the diversity of organisms in Antarctica, to
study the unique aspects of biology that permit life in such an extreme
environment, and to give students firsthand experience in dealing with
the unique problems inherent to Antarctic field sampling. The research
emphasis of the course was on experimental Antarctic biology, allowing a
number of aspects of evolution, physiology and ecology to be considered.
These included investigations on bacteria, algae, invertebrates and fish
that addressed molecular phylogeny, ultraviolet radiation effects,
energy metabolism and biochemical adaptations to cold temperature. The
course attracted an extremely competitive group of young scientists,
introduced new researchers to Antarctica, and provided participants the
opportunity to use the most modern research methods to study mechanisms
that are unique to polar biology. The course also fostered
collaborations between participants that will further influence their
future research activities.

**Research Experiences for Undergraduates (REU) Sites:** This program
introduces undergraduate students to meaningful science and engineering
research, and in many cases, motivates them to continue studies for an
advanced degree in the science and engineering disciplines. A hallmark
of the REU program is diversity, not only in terms of the participating
students, but also in the geographic locations, participating
institutions, and available research areas. REU sites are located across
the country. At all REU sites participating students choose research
projects from a spectrum of both traditional and interdisciplinary
topics. Some activities allow students to participate in international
research collaborations. For example:

-   The Division of Chemistry (CHE) has one of the largest investments
    in REU sites in the National Science Foundation. Support is provided
    for 64 sites in 32 states where more than 600 students conduct
    research each summer. Half the participants are female, and 23% are
    from underrepresented ethnic or racial groups.

-   The REU sites supported in the Division of Astronomical Sciences
    play a crucial role in providing a science and technology workforce
    that reflects America's diversity. For example, of the 125
    undergraduates supported in FY 2001, fully 50% of them were women
    and 15% were from members of underrepresented minority groups.

-   The Division of Mathematical Sciences supported 243 undergraduates
    on research awards and an additional 296 undergraduates at 29
    Research Experiences for Undergraduates (REU) sites.

-   The Division of Physics supported 55 REU sites in 2001, providing
    undergraduate research opportunities for approximately 700 students.
    Eight of the physics REU sites also include Research Experiences for
    Teachers (RET) programs to bring high school teachers into the
    research experience.

-   The Division of Materials Research supported the research efforts of
    well over 1,000 undergraduates in FY 2001, including 760 at REU
    Sites, Materials Research Science and Engineering Centers (MRSECs)
    and other national facilities, and several hundred more through
    individual-investigator awards. About 350 students participated in
    summer research experiences at 36 locations established through the
    annual NSF-wide REU site competition. In addition, 23 MRSECs
    supported by DMR in FY 2001 incorporated REU Sites as an integral
    part of the MRSEC efforts; about 380 students participated. DMR user
    facilities including the National High Magnetic Field Laboratory and
    the Synchrotron Radiation Center also supported REU Sites involving
    30 students[^13].

-   The Cross-Directorate Activities Program (CDA) in the Directorate
    for Social and Behavioral Sciences supported 24 REU sites in
    FY 2001. Forty percent of the Principal Investigators for these
    sites are female and 13% are underrepresented minorities. Almost 50%
    of the nearly 200 students participating in these sites are
    underrepresented minorities and two-thirds are female.

-   One additional example comes from the **Oklahoma Weather Center
    (OWC)** REU activity. It is bringing 30 undergraduate students, over
    a three-year period, to Norman, Oklahoma to participate in the OWC
    program. Each student: 1) was matched with an atmospheric scientist
    based upon his/her interest and ability to conduct research; 2)
    attended atmospheric science lectures; 3) participated in workshops
    on topics such as technical writing, numerical modeling,
    meteorological tools, graduate school preparation; 4) participated
    in field trips to regional sites; 5) tried various research
    methods, 6) collected and analyzed data; and 7) presented his or her
    research results in an formal presentation and a written paper. The
    NSF REU program was leveraged by incorporating students
    participating in similar programs sponsored by NSF's CIRE
    (Collaboratives for Integration of Research and Education) and the
    Department of Energy's ORISE (Oak Ridge Institute for Science and
    Education) programs.

Maintaining a diverse, balanced portfolio is an essential aspect of any
investment strategy, and this holds true for investments we make in
science and engineering research and education. We recognize that there
is a significant probability of failure associated with high-risk
research, that there is often a lack of experimental data or
methodologies, little consensus on theory, information and/or approach.
If successful, however, such high-risk research can result in a
significant advance in a scientific or technological field. In addition
to our regular grants, our Small Grants for Exploratory Research (SGER)
are meant to encourage Program Officers to invest in new, innovative
concepts and ideas and to support small-scale, high-risk exploratory
work.

Our external reviewers assessed our investment portfolio for FY 1998, FY
1999 and FY 2000 with respect to this area. There were numerous comments
on this area of emphasis, and the vast majority of these comments
indicated that investments made by the Directorates contained an
appropriate balance of high-risk, multidisciplinary or innovative
activities. Some of our COVs felt, however, that NSF needed to support
more high-risk activities. Some comments from our Advisory Committees
and COVs:

-   "In FY 2001, the Directorate funded \$28.14 million in awards for
    the Network for Earthquake Engineering Simulation (NEES)---a high
    risk program to shift the emphasis of earthquake engineering
    research from physical testing to integrated experimentation,
    computation, theory, databases, and model-based simulation."

-   "The Engineering Directorate invests heavily in high-risk research
    and education activities in Nanoscale Science and Engineering."

-   "In addition to the single investigator grants, the directorate
    places a high priority on multidisciplinary work and on
    partnerships. Within MPS, the Office of Multidisciplinary Activities
    serves as a catalyst in emerging areas of research and education at
    disciplinary boundaries."

-   "The balance of high-risk, multi-disciplinary or innovative research
    was cited as a particular success; the examples give ample evidence
    of success; the awards in the ITR program position CISE for
    continued success in this area of emphasis."

-   "The OAC noted that the mix of research activities OPP sponsors cuts
    across the spectrum from high risk to high innovation. In supporting
    research that is considered on the "cutting-edge," OPP makes
    investments that require taking risks."

-   "The COV generally felt there was a good balance to the portfolio
    but more high-risk projects should be considered ...."

-   "All projects, because of the nature of the program, systemic focus
    in extremely complex organizations that serve urban, diverse,
    low-socio-economic districts, are high-risk. In this area, balance
    of the portfolio is especially important."

-   "Using data compiled by the Division for its various component
    programs, it is clear that funded proposals generally display a high
    degree of novelty and disciplinary impact with a medium (theory) to
    medium high (experimental) level of risk. There was no evidence of
    bias against proposals, which were of high risk, multidisciplinary,
    and innovative. There was appropriate evidence of bias against
    proposals that were not innovative. In two cases, one could
    interpret the unsuccessful outcome of the proposals as being due to
    an insufficient level of risk."

-   "The division seems to have an appropriate balance of high risk and
    multidisciplinary projects."

Examples of high-risk, innovative, or multidisciplinary awards include
support to:

-   Determine whether electrical stimulation can control neural cell
    behavior as well as chemicals;

-   Explore the use of nanotechnology techniques for DNA sequencing,
    based on a pioneering technology that combines the fields of
    nanotechnology, new fractionation science, and single-molecule
    molecular biology. This technology could become the basis for the
    development of a "biology laboratory on a chip."

-   Develop novel experimental techniques that will make it easier to
    study chemical reactions at the atomic scale. The results will pave
    the way for the design of catalyst nanoparticles to enhance the
    performance of methanol fuel cells.

-   Develop a multidisciplinary bioprocessing curriculum that integrates
    expertise from biochemistry, microbiology, chemistry, botany, and
    chemical engineering. This effort has led to a new multidisciplinary
    graduate training program in "technologies for a bio-based economy."

-   Investigate the feasibility of creating sophisticated folded
    structures from metal and polymer sheets for applications requiring
    light but stiff structures;

-   Work with automotive engine designers to cut down vibration and
    shake in combustion engines. Researchers are exploring an approach
    similar to that used in skyscrapers.

-   Develop computational and statistical methods to evaluate
    comparative genetic map data;

-   Test a novel expression vector system developed to directly rescue
    open reading frames (ORFs) from genomic DNA;

-   Collect large amounts of expressed sequence information for floral
    structures across the angiosperms and gymnosperms, and use the
    information to determine which genes are required for floral organ
    development. This represents the first of a new kind of Virtual
    Center funded in FY 2001. Its focus is "evolutionary genomics".

-   Determine the presence or absence of biological signatures in
    Antarctic meteorites;

-   Use high-resolution satellite imagery to census Emperor penguin
    colonies;

-   Develop new sampling and measurement methods and tools to explore
    under the Arctic Ocean ice cap in the Gakkel Ridge region;

-   Develop autonomous underwater vehicles (AUVs) designed to explore
    and collect samples from underneath Arctic sea ice or Antarctic ice
    shelves;

-   Study biology on surfaces. This is an example of research at the
    interface between the physical sciences of surfaces and the life
    sciences of biological organisms.

-   Have a highly multidisciplinary team of scientists look at
    environmental catalysis and environmental interfacial
    characterization.

A number of multidisciplinary areas of research and education are
identified as being of particular importance for their potential
connections to use in service to society. These fit within the
Foundation's broad emphasis areas of ITR, BE, and Nanoscale Science and
Engineering. The amounts invested in FY 2001 are given in the following
Table:

+---------------------------------------------+------------------------+
| **FY 2001 Initiatives**                     |                        |
|                                             |                        |
| **(Millions of Dollars)**                   |                        |
+---------------------------------------------+------------------------+
| Information Technology Research             | \$216.27               |
+---------------------------------------------+------------------------+
| **Nanoscale Science and Engineering**       | \$149.68               |
+---------------------------------------------+------------------------+
| **Biocomplexity in the Environment**        | \$54.88                |
+---------------------------------------------+------------------------+

***Information Technology Research* (ITR):** We have been designated as
lead agency for a multi-agency *Information Technology Research*
initiative begun in FY 2000. In supporting research and education in the
ITR area, we work in partnership with other agencies.

Advanced information technology has expanded the scope of science and
engineering -- from the subatomic level to the cosmos -- by adding the
computer as a third partner to the time-tested methods of theory and
experimentation. This new, virtual mode of inquiry is where much of
today\'s most important fundamental research is happening. Virtually
every field of science and engineering now benefits from -- and in many
cases relies heavily on -- the use of information technology (IT)

IT research and education has also fundamentally changed almost every
sector of our **society and economy**. Economists, including Federal
Reserve Bank chairman Alan Greenspan, agree that advances in IT have
dramatically boosted productivity of the U.S. workforce. To quote Mr.
Greenspan:

> \"When historians look back at the latter half of the 1990s a decade
> or two hence, I suspect that they will conclude we are now living
> through a pivotal period in American economic history....It is the
> growing use of information technology throughout the economy that
> makes the current period unique[^14].\"

While development of commercial products is not the point of basic IT
research, the field has recently yielded spectacular results. In the
early 1990s, for example, students working under the direction of senior
researchers at an NSF supercomputing center helped create Mosaic, the
**web browser** that helped trigger a new era of electronic commerce.
Other prior successes of NSF-funded IT research and education activities
include the vast expansion and privatization of Internet infrastructure
in the mid-1990s, and projects that led to many of today\'s leading
commercial web search engines such as Lycos, Google and Inktomi.

-   **Intelligent radar sensors:** NSF supported an ITR award for the
    development and deployment of intelligent radar sensors for
    measuring key glaciological parameters. Radar instrumentation will
    consist of a synthetic aperture radar (SAR) that can operate in
    bistatic or monostatic mode. A tracked vehicle and an automated
    snowmobile will be used to test and demonstrate the utility of an
    intelligent radar in glaciological investigations. This project
    involves innovative research in intelligent systems, sounding radars
    and ice-sheet modeling. In addition it has a very strong public
    outreach and education program that includes near-real-time image
    broadcasts via the World Wide Web.

```{=html}
<!-- -->
```
-   NSF is supporting a project to make computers more proactive in
    their interactions with people. In **human interaction**, someone
    who waits for each command before making any communication attempt
    would be regarded as uncooperative and unhelpful. In order for a
    computer to bear its part of the burden in initiating interactions,
    it must first have much more real-time information about its user,
    and second, algorithms that select actions based on this
    information, in addition to user commands. The computer needs
    information about the user\'s current and past emotional,
    motivational and cognitive state, as well as the state of the task
    at hand. Research and education activities will include development
    of methods to sense user postures, movements, expressions and
    speech; fusion of this information to track user states; creating
    means for effective communication, human-centered action decisions,
    and a corpus of emotion- and action-labeled videotapes for use with
    computer learning; and finally, evaluation of computer pro-action on
    human behavior and response.

***Nanoscale Science and Engineering* (NSE)** represents a new focused
investment opportunity in FY 2001. Support for nanoscale research and
education is motivated by the impressive potential for economic return
and social benefit. The initiative will lead to potential breakthroughs
in areas such as materials and manufacturing, nanoelectronics,
healthcare, environment and energy, chemical and pharmaceutical
industries, biotechnology and agriculture, computation and information
technology, and national security.

Areas of emphasis include biosystems at the nanoscale, nanoscale
structures and novel phenomena, device and system architecture,
nanoscale processes in the environment, multi-scale and multi-phenomena
modeling and simulation. NSF supports a wide range of research and
education activities in this priority area, including approximately 15
nanotechnology research and education centers, which focus on
electronics, biology, optoelectronics, advanced materials and
engineering. Examples of collaborations with other agencies and private
sector include: quantum computing with DARPA; Materials Research Science
and Engineering Centers (MRSEC) with the Department of Defense (DOD);
Semiconductor Industry Association and Engineering Research Centers
(ERC); Grant Opportunities for Academic Liaisons with Industry (GOALI)
awards (collaboration with private sector); and cofunding two new
Nanoscale Science and Engineering Centers (NSEC) with the DOD.

-   **Societal implications of nanotechnology:** On September 28-29,
    2000, a workshop was held at NSF on "The Societal Implications of
    Nanoscience and Nanotechnology." The aim was to: (1) survey current
    studies on the societal implications of nanotechnology (educational,
    technological, economic, medical, environmental, ethical, legal,
    etc.); (2) identify investigative and assessment methods for future
    studies of societal implications; and (3) propose a vision for
    accomplishing nanotechnology's promise while minimizing undesirable
    consequences. The extensive report of this workshop has been
    published both on the web and in book form and has also been
    distributed to all agencies and to the President's National Science
    and Technology Council. A second workshop was held December 3-4,
    2001, to address how cognitive science can be integrated with
    nanotechnology, biotechnology and information technology to enhance
    human performance.

-   **Manipulating Matter at the Nanoscale:** Manipulating matter on the
    nanometer scale is important for many electronic, chemical and
    biological advances. Available solid-state fabrication methods do
    not reproducibly achieve nanometer-scale dimensional control.
    However, an ion beam can be used to poke holes in thin films to
    produce structures that are used to manipulate nanoscale matter. The
    method can fabricate a molecular scale hole, or nanopore, in a thin
    insulating solid-state membrane. Nanopores localize molecular scale
    junctions and switches and act as masks to create other small
    structures. Nanopores also function as membrane channels in all
    living systems, where they are extremely sensitive electromechanical
    devices that regulate electric potential, ionic flow and molecular
    transport across cellular membranes. "Ion beam sculpting" has been
    used to fabricate a robust electronic detector capable of
    registering single DNA molecules in aqueous solution. Such detectors
    may find utility in extremely rapid sequencing of DNA for medical
    diagnostics of genetic diseases and rapid drug design for large
    populations.

***Biocomplexity in the Environment* (BE)** became an area of focus in
FY 1999, beginning with a special competition on the "Interrelationships
between Microorganisms and Biological, Chemical, Geological, Physical,
and Social Environments." In FY 2000, NSF sponsored a \$50 million
initiative -- *Integrated Research to Understand and Model Complexity
Among Biological, Physical, and Social Systems*. In FY 2001 and FY 2002
the Biocomplexity in the Environment competitions focused on Integrated
Research and Education in Environmental Systems.

Understanding the dynamics of biological complexity and its role in
environmental systems is critical to knowledge of living organisms and
of the vital natural resources biological systems provide (e.g., food,
fiber) and upon which humans depend. Advancing our understanding of the
nature and role of biological complexity demands increased attention and
new collaborations of scientists and engineers from a broad spectrum of
fields -- biology, physics, chemistry, geology, hydrology, statistics,
engineering, computation, social sciences. Such collaborations can
capitalize on powerful new emerging technologies -- including genome
sequencing, new computational algorithms and mathematical methods,
sensors and monitoring devices, and remote sensing -- that have greatly
enhanced our ability to understand ecosystem complexity and dynamics.

-   *Organohalide Pollutants***.** Organohalides pose serious threats to
    air and water quality. They are among the most widely used
    industrial chemicals, with applications in such familiar processes
    as dry cleaning and air conditioning. Of the top 25 organic
    contaminants found in ground water in U.S. urban cities, 17 are
    organohalides. Investigators and their collaborators from national
    laboratories and industry are studying ways to break down
    organohalides into less toxic substances. They hope to develop new
    techniques for detecting environmental organohalides, predicting and
    monitoring their rates of natural attenuation, and, wherever
    practical, decontaminating sites where organohalides are found.

**Mathematical Sciences Research:**

-   **The Simplex algorithm:** Two NSF-supported investigators have
    solved a long-standing open question in mathematical programming,
    optimization, and theoretical computer science, proving that the
    Simplex Method for Linear Programming usually takes a polynomial
    number of steps. They developed a new algorithm-analysis framework,
    called smoothed analysis, which can help explain the success of many
    algorithms and heuristics that traditional algorithm-analysis
    frameworks, such as worse-case and average-case analysis, cannot.
    The simplex algorithm is a classic example of an algorithm known to
    perform well in practice yet it consumes exponential time in the
    worst case. It has been an active subject for mathematical and
    experimental studies for more than 50 years.

**Functional Genomics:**

-   The research project, \"The Role of Gamma Aminobutyric Acid (GABA)
    in Plant Growth and Productivity," awarded to Sun Dance Genetics,
    Durham, NC has added to understanding of the role of GABA in plant
    growth. Commercial application of formulations containing GABA is
    found throughout the world. Agriculture, horticulture, floriculture,
    and **turf grass** industries are now benefiting from a product that
    increases plant growth and productivity while reducing the amount of
    fertilizer needed for optimal plant productivity. The results of
    this NSF-supported research have established the efficacy of the
    company\'s formulations for increasing plant growth and productivity
    and reducing a plant's fertilizer requirements, and demonstrate
    their commercial utility. The company estimates the total U.S.
    market opportunity for such products to be approximately \$3.0
    billion per year.

-   ***Arabidopsis thaliana*:** DNA sequence data is an essential tool
    but is not enough to tell us everything about how an organism
    develops and functions. Building on the large and growing store of
    information amassed in the international sequence databases,
    biologists are now able to tackle the next frontier in biology,
    functional genomics, which uses genome sequence information in
    combination with data from other biological research to study what
    genes do -- that is, how patterns of sequence are related to
    patterns of function. Functional genomics offers unprecedented
    opportunity to understand living systems through use of large-scale,
    genome-derived information. NSF\'s first major program in functional
    genomics, the \"2010 Project,\" began in FY 2001, and will continue
    through the year 2010. Its goal is to determine the functions of the
    25,000 genes of the flowering plant, *Arabidopsis thaliana*.

-   **How insects develop immunity to plant toxins:** *Bacillus
    thurnigiensis* (Bt) is a bacterium that produces toxins with
    insecticidal activities. One of the early success stories of genetic
    engineering of plants was the introduction of the gene encoding the
    toxin into agriculturally important crops, such as cotton. These
    plants produced their own pesticide thereby reducing the need for
    chemical pesticides. Unfortunately, insects can "learn" (evolve
    mechanisms) to escape this clever trap, and develop resistance to
    the toxin but the molecular basis for resistance was unknown. In
    2001, two independent groups focusing on the molecular mechanism of
    Bt toxin resistance in two plant pests produced important results
    that were reported in back-to-back papers that appeared in
    *Science*, August 2001 and were highlighted by a "news and views"
    report.

***Cognitive neurosciences:***

-   **How bees learn:** Studies of the neuroanatomical basis of
    hormone-mediated behavioral development in the honeybee have shown
    that such bees learn olfactory associations as they emerge from the
    hive to forage for flower nectar. These studies are the first to use
    a social insect as a model system to assess the effects of a hormone
    on cognition, and provide a new perspective for understanding bee
    foraging behavior. They could also have substantial impact on
    agriculture. There has been a large response in the popular press to
    this work, with coverage from the New York Times, National Public
    Radio, the BBC and many other European news organizations.

-   **Mapping brain function:** A conformal map of one region to another
    preserves angles between intersecting curves, a property that is
    especially valuable in the study of regions such as the visual
    cortex of the brain. Conformal geometry is being used to map brain
    function using conformal mapping algorithms and other geometric
    ideas. These efforts at brain mapping have been described in several
    widely-read accounts, such as an article in the August 2001 issue of
    *Scientific American*.

**NSF OUTCOME GOALS**

![](media/image9.wmf){width="0.7208661417322835in"
height="0.5251968503937008in"}**C. TOOLS**

**PERFORMANCE OUTCOME GOAL III-3: Providing "broadly accessible,
state-of-the-art and shared research and education tools."**

**** **Goal Achieved**

NSF invests in tools to provide widely accessible, up-to-date science
and engineering infrastructure. This strategic outcome supports the
parts of NSF's mission directed at (1) programs to strengthen scientific
and engineering research potential and (2) an information base on
science and engineering appropriate for development of national and
international policy.

As emerging research opportunities increasingly involve phenomena at or
beyond the limits of our measurement capabilities, many research areas
can only be studied and problems solved through the use of new
generations of powerful tools. NSF investments provide state-of-the-art
tools for research and education, such as instrumentation and equipment,
multi-user facilities, digital libraries, research resources,
accelerators, telescopes, research vessels and aircraft and earthquake
simulators. In addition, resources support large surveys and databases
as well as computation and computing infrastructures for all fields of
science and engineering research and education. Support includes funding
for construction, upgrade, operations, and maintenance of facilities,
and for personnel to assist scientists and engineers in conducting
research and education at the facilities.

This is a new goal. Our performance is successful when, *in the
aggregate*, results reported in the period demonstrate significant
achievement for one or more of the following indicators:

-   Shared use platforms, facilities, instruments, and databases that
    enable discovery and enhance the productivity and effectiveness of
    the science and engineering workforce;

-   Networking and connectivity that take full advantage of the Internet
    and make SMET information available to all citizens;

-   Information and policy analyses that contribute to the effective use
    of science and engineering resources.

**RESULT:** Reports prepared by external experts during FY 2001 GPRA
reporting provided assessments and retrospective examples of
NSF-supported projects that document significant achievement. A sample
of these is provided for each of the performance indicators and areas of
emphasis for this goal.

There are very limited contributions and limited involvement of agency
programs, other than Science Resources Statistics (SRS), in developing
information and other materials fundamental to national policy debates.

**Implications for the FY 2002 Performance Plan:** This goal will be
continued in FY 2002. The performance indicators related to the Tools
goal has been expanded and modified to appropriately reflect the breadth
of NSF activities (see Section X. for details).

The **Incorporated Research Institutions for Seismology** (IRIS) seeks
to image, at high-resolution, the structure and composition of the
entire Earth. This image will form the basis for a new, physics-based
description of the dynamics of the whole Earth. Moreover, using Internet
technology, the data management program will make these state-of-the-art
data available to anyone in the world, promoting a powerful synergy of
research and education. The scale of the facility, like the scale of the
problem, is large, encompassing a partnership of 96 institutions and
hundreds of individual researchers and research projects.

The **national astronomy facilities,** consisting of the National Radio
Astronomy Observatory, the National Optical Astronomy Observatory, the
National Solar Observatory, the National Astronomy and Ionosphere Center
and Gemini Observatories, provide access to a broad scientific community
on the basis of scientific merit review. Each facility provides unique
capabilities to over 2,400 scientists and their students of whom over
half were pursuing their doctorates. The Atacama Large Millimeter Array
(ALMA), which will bring aperture synthesis techniques to radio
astronomy, is in the last year of design and development prior to
project construction involving an equal U.S.-European partnership.

**Materials Research Facilities:** The major-shared facilities include
the National High Magnetic Field Laboratory (NHMFL), the Center for High
Resolution Neutron Scattering (CHRNS), the Cornell High Energy
Synchrotron Source (CHESS), the Synchrotron Radiation Center (SRC), and
the National Nanofabrication Users Network (NNUN). Each of these
facilities operates to enhance the productivity and effectiveness of
scientists and engineers in a wide range of disciplines. For example,
the NHMFL is operated by a consortium composed of the University of
Florida, Florida State University and Los Alamos National Laboratory and
NHMFL and has established itself as the world's leading center for
multidisciplinary research and education using high magnetic fields.

**Polar Facilities:** NSF manages and operates the three permanent U.S.
stations in Antarctica (South Pole, McMurdo and Palmer). These stations
provide the facilities for U.S. scientists, including those from other
federal agencies, to conduct research in Antarctica in widely divergent
disciplines. NSF also supports both permanent and seasonal stations in
the Arctic. In the past two years the Summit Station (Greenland) has
operated year-round to support winter observations with a particular
emphasis on atmospheric chemistry.

**Forecasting Lightning Strikes:** A Lightning Mapping Array (LMA) has
been developed and uses very accurate Global Positioning System (GPS)
timing capabilities to measure the time of arrival of electrical
discharges at a dense array of receivers to provide a very accurate
three-dimensional map of the structure and evolution of cloud-to-ground
(CG) and intra-cloud (IC) lightning. Because IC discharges typically
occur about 15 minutes before the onset of CG strokes, the LMA provides
added lead time for use in forecasting the onset of lightning strokes
hitting the ground (one of the deadliest weather phenomena occurring
today). It also provides a natural complement to weather radar, and
together they offer the promise for improvements in the accuracy and
timeliness of short-term forecasts of severe convective weather.

**Gemini and Laser Guide Stars:** NSF serves as the executive agency for
the Gemini Observatories, an international project with seven partner
nations (the U.S., the United Kingdom, Canada, Australia, Chile, Brazil
and Argentina). The Gemini Observatories consist of two 8-meter
telescopes, one located in the southern hemisphere in Chile, and one
located in the northern hemisphere in Hawaii. NSF provides 50% of the
funding, enabling merit-based access to both telescopes for the U.S.
national community. The project has also played a pivotal role in sodium
laser tests sponsored by Gemini, the Cerro Tololo Inter-American
Observatories, and the European Southern Observatory.

# NSF is the major source of support for the Panel Study of Income Dynamics (PSID), a survey of a nationally representative sample of U.S. individuals and the family units in which they reside and was initiated in 1968. The major objective of the panel is to provide shared-use databases, research platforms and educational tools on cyclical, intergenerational and life-course measures of economic and social behavior. The PSID's innovative design and long-term panel have been central to the fundamental understanding of key social science issues with substantial broad impacts on society: income, poverty and wealth; cyclical behavior of wages, labor supply and consumption; savings, wealth accumulation and transfers; demographic events; labor market behavior; and the effects of neighborhoods. 

# PSID data transformed research on poverty from a static view of poor and rich to a dynamic one in which families experience episodes of poverty or affluence. PSID results have been replicated and validated. The enormous usefulness of decades of data on the sample families has made the PSID one of the most widely used social science data sets in the world. The project currently delivers more than 10,000 customized data sets a year to researchers via its Internet Data Center. Since 1968, over 2,000 journal articles, books and chapters, dissertations and other works have been based on PSID data. More information on the PSID is available at [**http://www.isr.umich.edu/src/psid**](http://www.isr.umich.edu/src/psid).

The NSF-funded **General Social Survey** (GSS) has been monitoring
social change in the United States, examining sub-groups in society, and
studying social processes since 1972. Over this period, 23
cross-sectional surveys of the U.S. adult household population have been
completed, involving face-to-face interviews with approximately 41,000
respondents. Surveys document fundamental social change in areas such as
uses of technology, social and cultural capital, neighborhoods and
communities, social networks, and racial and gender attitudes. The GSS
is widely used by the academic research community and scholars at
research centers, foundations, and in government. For example, as of
April 2000, there were 5,430 documented uses of the GSS data, including
2,676 journal articles, 1,201 books, 959 scholarly papers and 153
dissertations and theses. The GSS is used extensively in teaching, with
GSS having been utilized in 226 textbooks and contained in course
materials on 224 websites. The cumulative GSS data files are used in the
Inter-University Consortium for Political and Social Research (ICPSR)
Summer Program, which is annually attended by 150 scholars representing
over 100 colleges in the United States and Canada.

**The World Data Center for Paleoclimatology** (WDCP) is housed at the
National Oceanic and Atmospheric Administration's (NOAA) National
Geophysical Data Center at Boulder, Colorado. Data sets include
tree-rings, lake core records, ice core records, ocean sediment records
and corals, developed in part by NSF-supported researchers.

The objective of the National Science, Mathematics, Engineering and
Technology (SMET) Education **Digital Library** (NSDL) is to catalyze
and support significant advances in the quality of SMET education at all
levels by providing broad access to the rich collections of teaching and
learning resources of the nation in a distributed, digital environment.
The NSDL aims to create, organize and install high-quality education
resources onto the Internet. The program may strongly affect K-12 and
undergraduate educational by providing anytime, anywhere access to a
rich array of authoritative and reliable interactive materials and
environments. Some of the newly funded projects focus on gathering and
organizing content in areas such as geosciences, life sciences,
engineering and mathematics. Others will develop the processes to manage
and coordinate that content in the library's core collections, and
develop services for users and collection providers. The program
continues ongoing efforts of the national, multi-agency Digital
Libraries Initiative begun in 1994, which has involved NSF, the Defense
Advanced Research Projects Agency, the National Aeronautics and Space
Administration, and other federal agencies.

The **Performance Assessment Links in Science** (PALS) project has
created an on-line web-based assessment resource for teachers and test
developers. The teacher site includes science performance tasks, with
information on which national science standards are addressed, its
psychometric properties, scoring rubric, and examples of students' work.
This rich resource is used by teachers nationally, including Chicago,
Los Angeles, San Mateo (CA), and the Spring Independent School District
(TX). PALS also is used in several NSF-supported Urban Systemic
Initiative sites to help document program effectiveness through improved
student achievement.

The Space Science Institute has developed the **Space Weather Center web
site** as part of the National Space Weather Program (NSWP). The web
page can be viewed at <http://www.spacescience.org/>. It serves as a
central outlet for public information on space weather by providing a
collection of resources of interest to educators, the media, and the
general public. The web site includes introductory information on space
weather, an image archive of the best images from space weather research
programs, brief reports written by space weather researchers, links to
current solar and space weather data, and links to downloadable
curricula related to space weather. A new capability installed in the
past year allows visitors to the web site to take a virtual tour of any
of several space weather museum exhibits the Space Science Institute has
developed.

In FY 2001 NSF awarded **high performance network connections** to 19
universities, a research museum, and two research institutes, bringing
the total of institutions assisted through such grants to 221. Since
1995 the NSF High Performance Network Connections (HPNC) program has
provided scientists and engineers better access to research facilities
across the U.S., including those maintained by NSF through its
Partnerships for Advanced Computational Infrastructure (PACI) program.
The new awardees will join in connecting to a national grid of research
networks that operate at speeds up to 2.4 billion bits per second.

**Mathematics:** Software available through the Internet has been
developed for the solution of partial differential equations via
numerical analysis or object-oriented code, for the teaching of finite
element analysis, and for the solution of linear recurrence equations.
In addition, the conservation law package (CLAWPACK) has been freely
available on the Internet to researchers using multidimensional
high-resolution finite-volume methods for solving hyperbolic partial
differential equations. For example, the equations can simulate
ultrasound propagation in human tissue, elastic waves in heterogeneous
media, and gravitational waves of planetary bodies indicating
applicability to biosciences, materials science and astronomy.

Through the **WEB100** project, universities, research centers, and some
businesses today have connections capable of transmitting data at 100
megabits per second (Mbps) or higher. Research has shown, however, that
users rarely see performance greater than 3 Mbps. New WEB100 software,
developed jointly by the National Center for Atmospheric Research
(NCAR), the Pittsburgh Supercomputing Center, and the National Center
for Supercomputing Applications, with funding from NSF, will allow users
to take full advantage of available network bandwidth without the help
of a networking expert. WEB100 researchers traced the problem of poor
performance to software that governs the Transmission Control Protocol
(TCP). Networking experts are able to overcome this limit by fine-tuning
connections with adjustments to TCP. This type of "middleware" can help
us use existing resources more efficiently.

The **HomeNetToo** project focuses on the home Internet use of
low-income families, many of whom are first-time computer users.
Internet use is automatically computer-logged and surveys are
administered at five times over an 18-month period to address the
antecedents and consequences of Internet use. Preliminary findings
indicate that cognitive style is related to Internet use and influences
the relationship between race and use (as does socioeconomic status,
i.e., education and income). Subsequent analyses will identify
additional culturally-based factors that influence this relationship.
Twenty-three undergraduates served as technology facilitators during the
project's first year, nearly half (10) of whom were members of
underserved minority groups (nine African-Americans, one
Hispanic-American). Almost half (11) were female. All are majoring in
Computer Science. The graduate student who served as Project Director in
the first year is an African-American male, majoring in educational
technology.

**Digital archives:** Two important NSF-supported tools projects are
already available on the web. The first catalogs the development of
sanitary technologies for New York during the 19^th^ century in a
digital archive of documents, reports, and illustrations. With NSF
assistance, the Center for the History & Ethics of Public Health in the
Department of Sociomedical Sciences in the Mailman School of Public
Health at Columbia University has assembled *The Living City*
(<http://156.145.78.54/htm/home.htm>), which includes among other
things, an annotated timeline for New York from the 1860s through the
1920s. This project could become an essential reference tool for
historians and students interested in sanitation, technology, urban
history, and New York City history.

The other accomplishment is the *Perseus Digital Library* at Tufts
University devoted to ancient Greece (<http://www.perseus.tufts.edu/>).
The Perseus project is widely regarded as a model for digital archives.

The Division of Science Resources Statistics (SRS) is the unit within
NSF charged with collecting, analyzing and disseminating information on
the S&E enterprise. These activities fulfill the legislative mandate of
the National Science Foundation Act to "provide a central clearinghouse
for the collection, interpretation, and analysis of data on scientific
and engineering resources, and to provide a source of information for
policy formulation by other agencies of the Federal Government."

To carry out this mandate, SRS designs, supports, and directs 11
periodic surveys as well as a variety of other data collections and
research projects. These activities yield the materials for SRS staff to
compile, analyze, and disseminate quantitative information about
domestic and international resources devoted to science, engineering,
and technology. Each year SRS produces 40 to 50 publications, ranging
from short Data Briefs and Issue Briefs highlighting results from recent
surveys and analyses to Detailed Statistical Tables (DSTs) containing
extensive tabulated data from a particular survey; and large,
comprehensive \"overview\" reports, such as *Science and Engineering
Indicators* and *Women, Minorities, and Persons with Disabilities in
Science and Engineering.*

All new publications are placed on the SRS website and some material is
available only in an electronic format, not in a printed version. Many
tables on the web are also available in a spreadsheet format so they may
be downloaded and manipulated by users. The data are also provided to
users in a variety of formats, media, and in customized tabulations. In
addition, access to much of the data is provided through on-line
databases.

SRS has made improving the quality and usefulness of its data a high
priority in recent years. In previous years, SRS has had GPRA
performance goals related to the timeliness of the release of the data
the Division collects and establishment of data quality measures for its
surveys. For FY 2001, SRS worked to improve the relevance and quality of
the data it collects, the two dimensions rated most important by SRS
customers in a 1999 customer survey -- accuracy and the ability to find
and obtain needed information on S&E personnel and resources.

For FY 2001, SRS identified one or more aspects of each ongoing survey
in need of improvement and ways in which such improvement could be
pursued. Projects for half the surveys were begun in FY 2001. Choices as
to which of the projects to pursue in FY 2001 were based on a variety of
factors, including funding, feasibility, favorable timing in terms of
the survey cycle, and Division priorities. In nearly every case, these
projects will extend into FY 2002 and beyond, and it will be several
years before the results of these efforts will be evident in terms of
improved data quality. In the meantime, additional projects directed
toward improving data quality will be undertaken in FY 2002 and
subsequent years.

The projects undertaken in FY 2001 vary considerably in scope, ranging
from the complete redesign of the Survey of Scientific and Engineering
Research Facilities to review/improvement of questionnaire layout and
specific question wording of several surveys to adoption of a standard
method of computing response rates in the National Survey of Recent
College Graduates. Examples of specific FY 2001 projects include the
Facilities Survey, the Federal Funds Survey, the Academic Survey, the
Graduate Student Survey (GSS), and the National Survey of Recent College
Graduates (NSRCG)

In addition to these projects, all of which had been identified at the
beginning of FY 2001, SRS undertook a variety of other activities during
the year that will contribute to improved data quality in the future.
Several approaches toward improving questionnaire design (paper and
electronic) were started, including cognitive interviews with survey
respondents and assistance from a nationally recognized expert on
questionnaire design. Specifically, with regard to the public attitudes
survey, several avenues were explored as alternatives to the historical
Random-Digit-Dialing (RDD) telephone survey, about which there are
concerns related to both content and response rates. These included
placing items on other surveys, and web-based options for collecting
information.

SRS also provided data and assistance to a variety of offices throughout
the government in support of policy debates about science and
technology. For example, SRS continued working with the Immigration and
Naturalization Service in efforts to improve the scope of data capture
and nature of coding of immigrants' education levels, major fields, and
last occupation before entering the U.S.

In addition to conducting surveys and releasing reports/data based on
the surveys, SRS prepared and completed external reviews of the
Congressionally mandated report, *Women, Minorities and Persons with
Disabilities in Science and Engineering: 2002*, a source book of data on
the participation of these groups in science and engineering education
and employment, widely used by policy makers addressing the problems of
underrepresentation of these groups in science and engineering.

SRS released, on the Division of Science Resources Statistics website,
the Industrial Research & Development Information System (IRIS) which is
an online interface to the new Survey of Industrial Research and
Development Historical Database 1953-1998. The database contains all
industrial research and development (R&D) data published by NSF for the
years between 1953 and 1998 in over 2,500 statistical tables including
statistics on the levels of R&D support from company and federal sources
and sales and employment of R&D performers.

NSF provides funding for capital expenditures for the construction and
acquisition of major research facilities that provide unique
transformational capabilities at the cutting-edge of science and
engineering. Projects supported capitalize on technological innovation
to provide significant new research and education opportunities,
frequently in totally new directions. Continuing projects include South
Pole Station Modernization (SPSM), Atacama Large Millimeter Array (ALMA)
research and development, Network for Earthquake Engineering Simulation
(NEES), the Large Hadron Collider, and the Terascale Computing System
(highlighted in the next section).

The **Large Hadron Collider** (LHC) will be the frontier particle
accelerator in the world when it comes on-line in the second half of
this decade. There is now preliminary experimental evidence that the
Higgs particle, the key to understanding why everything in the universe
has mass, can be detected with the LHC. Recent work at the Brookhaven
National Laboratory on the magnetic moment of the muon, (the recently
reported results from the "g-2" experiment), provided results not
predicted by the Standard Model. Further tests of the Standard Model
will be done at the LHC. The U.S. ATLAS and CMS projects continue to
meet their goals and are reliable and influential partners in the
construction of the two detectors of the LHC machine. The European
Organization for Nuclear Research (CERN) expects to complete
construction of the LHC and initiate collider commissioning in 2005. The
U.S. schedules are consistent with this goal. The institutions
developing the two detectors have done an excellent job of meeting cost
and schedule goals.

At 10:00 AM, Friday, October 20, 2000, leaders of the **Laser
Interferometer Gravitational-wave (LIGO) Observatory** announced that
"First Lock" had been achieved with the two-kilometer long
interferometer at the Hanford Observatory. This marked achievement of a
major LIGO milestone. All mirrors were "locked" into their proper
positions to atomic-scale precision using a sophisticated computer-based
control system. First lock validated many aspects of the control system
design for the initial LIGO detectors, but it had even greater
significance -- the beginning of the process of tuning the
interferometer to its full sensitivity. Most importantly, this
achievement brought LIGO closer to its ultimate goal -- the first true
gravitational-wave observations.

NSF supports two ongoing MRE projects of enormous significance to
scientific and engineering research in the Antarctic region: 1) The
**South Pole Station Modernization**; and 2) Polar Support Aircraft
(LC-130) Upgrades. Both projects represent essential investments in the
health and vitality of on-continent and deep-field research in
Antarctica.

**Terascale Computing System**

NSF is continuing development of a *Terascale Computing System* to
enable U.S. researchers and educators to gain access to leading edge
computational systems.

-   **Supporting technologies:** Research performed under past and
    present NSF support has been incorporated as key components of the
    Terascale Computing System in the following key technology areas of
    parallelizing compilers; sequential compilers; numeric libraries;
    runtime libraries; and parallel software tools, input/output,
    visualization, and applications.

-   **Initial performance:** The new Terascale Computing System (TCS)
    funded by NSF in fiscal year 2000 has begun operation well ahead of
    schedule and is exceeding performance expectations. During an
    acceptance test in which the Pittsburgh Supercomputer Center staff
    evaluated its performance, TCS consistently surpassed speed
    expectations and operated virtually without interruption. The
    combined peak power of the full computer system will be 6 Teraflops,
    making it the most powerful computer available to academic
    scientists and engineers in the United States.

###### The Major Research Instrumentation Program 

The Major Research Instrumentation Program (MRI) is designed to improve
the condition of scientific and engineering equipment for research and
research training in our nation\'s academic institutions. This program
seeks to improve the quality and expand the scope of research and
research training in science and engineering, and to foster the
integration of research and education by providing instrumentation for
research-intensive learning environments.

-   The **Electronic Visualization Laboratory's (EVL)** research has
    focused on developing tools, techniques and hardware to support
    real-time, highly interactive visualization. Current efforts
    continue through the development of virtual reality (VR) devices,
    software libraries/toolkits and applications for collaborative
    exploration of data over national and global high-speed networks -
    called "tele-immersion." After building first and second-generation
    VR devices \[the Cave Automatic Virtual Environment (CAVE) in 1991
    and the ImmersaDesk in 1995\] to support tele-immersion
    applications, EVL is now conducting research in "third-generation"
    VR devices to construct variable resolution and desktop/office-sized
    displays. EVL continues to develop and refine a robust and
    VR-device-independent software library, as well as the software
    tools for building tele-immersion applications. This software
    infrastructure supports collaboration in design, training,
    scientific visualization, and computational steering in VR. Through
    advanced networking techniques, researchers can access distributed
    computing, storage and display resources more efficiently than ever.

###### S&E informational/reports/databases

-   **The General Social Survey:** The General Social Survey (GSS) has
    been monitoring social change in the United States, examining
    sub-groups in society, and studying social processes since 1972.
    Over this period, 23 cross-sectional surveys of the U.S. adult
    household population have been completed, involving face-to-face
    interviews with approximately 41,000 respondents. Surveys document
    fundamental social change in areas such as uses of technology,
    social and cultural capital, neighborhoods and communities, social
    networks, and racial and gender attitudes. The GSS is widely used by
    the academic research community and scholars at research centers,
    foundations, and in government. For example, as of April 2000, there
    were 5,430 documented uses of the GSS data, including 2,676 journal
    articles, 1,201 books, 959 scholarly papers and 153 dissertations
    and theses. The GSS is used extensively in teaching, with GSS having
    been utilized in 226 textbooks and contained in course materials on
    224 websites. The cumulative GSS data files are used in the
    Inter-University Consortium for Political and Social Research
    (ICPSR) Summer Program, which is annually attended by 150 scholars
    representing over 100 colleges in the United States and Canada. GSS
    data are made available to researchers and their students. The GSS
    Data and Information Retrieval System provides facilities for
    statistical analyses, hypertext viewing, customized extracts from
    data sets, and File Transfer Protocol for extracted data sets.

-   **Climate Change Policy:** NSF supports scientists who are engaged
    in the Intergovernmental Panel on Climate Change (IPCC) and U.S.
    National Assessment programs. The results of both of these
    assessments are used for policy analyses so that there is effective
    use of scientific resources. Beyond these scientific assessments
    with their specific policy-informing goal, there are specific
    examples of studies that provide policy analyses. One particular
    example involves an analysis of climate-change abatement policies.
    Most quantitative studies of climate-change policy attempt to
    predict a greenhouse-gas reduction plan that will have the optimum
    balance of long-term costs and benefits. However, it was found that
    large uncertainties associated with the climate-change problem could
    make the policy prescriptions of this traditional approach
    unreliable. An adaptive strategy with mid-course corrections was
    able to avoid significant errors.

***New types of scientific databases and tools for using them***

-   **Government statistical information** is essential in the
    day-to-day lives of all citizens. The importance of such data is
    illustrated by the efforts of multiple federal government agencies
    to create the National Statistical Information Infrastructure. Data
    from agencies such as Bureau of Labor Statistics, Census Bureau, and
    Bureau of Economic Analysis determine costs of everything from
    apples to zinc, the locations of new businesses, and the indexes for
    all government programs and payments. Web-based technologies offer
    citizens broader access to the vast array of statistical data so
    that they may make better personal decisions. Examples include
    baby-boomers planning for retirement, unemployed or underemployed
    individuals looking to relocate, and school children exploring
    careers. For broader segments of the population to take advantage of
    government statistical information, however, the data must both be
    easy to find and easy to interpret and use. Ease of search in this
    setting depends on helping users articulate needs, on distributing
    these articulations to different datasets across the federal
    government, unifying the results, and presenting them in forms most
    useful to user needs. NSF-supported researchers have successfully
    completed work on graphical representation, manipulation, browsing,
    and usability over the Web for federal statistical (tabular) data.
    As the system becomes commercially available to the users of
    federally collected and archived statistical data, the primary
    challenge is to ensure it will improve the usefulness of data in
    establishing, for example, the Consumer Price Index, the
    unemployment rate, and the determination of federal congressional
    districts.

-   **LAPACK and ScaLAPACK:** The LAPACK and ScaLAPACK libraries are the
    standard software for solving dense linear equations. With FY 1999
    funding NSF-supported release 3.0 of LAPACK, which improves error
    bound estimates. As important as the LAPACK software is the means by
    which it is disseminated. NSF has been a long-term supporter of
    Netlib, the standard Internet repository for numerical software.
    There have been over 129,000,000 requests from Netlib to date,
    indicating just how popular it is. Less obvious is the amount of
    effort that its search capabilities have saved countless
    investigators in locating the right software for the job. NSF
    support of Netlib also contributes to full use of the national
    networks (<http://www.netlib.org/lapack/index.html>),
    (<http://www.netlib.org/scalapack/index.html>).

-   This year NSF supported the establishment of the **National
    Historical Geographic Information System** (NHGIS) to upgrade and
    enhance U.S. Census databases from 1790 to the present, including
    the digitization of all census geography so that place-specific
    information can be readily used in geographic information systems.
    The NHGIS consists of three major components: (1) The data and
    documentation component will gather all extant machine-readable
    census summary data, fill holes in the surviving machine-readable
    data through data entry of paper census tabulations, harmonize the
    formats and documentation of all files, and produce standardized
    electronic documentation according to the recently developed Data
    Documentation Initiative (DDI) specification. (2) The mapping
    component will create consistent historical electronic boundary
    files for tracts, minor civil divisions, counties, and larger
    geographic units. (3) The data-access component will create a
    powerful but user-friendly, Web-based browser and extraction system
    based on the new DDI metadata standard. The system will provide free
    public access to both documentation and data and will present
    results in the form of tables or maps. Through these activities, the
    NHGIS will become a resource that can be used widely for social
    science training, by the media, for policy research at the state and
    local levels, by the private sector, and in secondary education.

**MANAGEMENT GOALS**

![](media/image10.png)

#   {#section-16 .unnumbered}

# V. MANAGEMENT GOALS

For FY 2001 we emphasized five performance goals (three of them new)
related to implementation of information technologies and human
resources development. These goals address the incorporation of advanced
technology into NSF business operations, and human capital involving
diversity and the NSF work environment.

We consider four factors to be especially critical to management
excellence at NSF:

-   Operating a viable, credible, efficient merit review system;

-   Exemplary use of and broad access to new and emerging technologies;

-   A diverse, capable, motivated staff that operates with integrity;
    and

-   Implementation of mandated performance assessment and management
    reforms in line with agency needs.

These critical factors were used in our developing annual performance
goals. Results for the management goals, most of which have quantitative
measures, are prepared and reviewed by NSF staff. For selected goals,
the auditing firm of PricewaterhouseCoopers LLP (PwC) reviewed the data
collection, maintenance, processing, and reporting procedures used in
our assessment.

A.  **Business Practices**

**Goal IV-1 -- Electronic Proposal Submission**

**** **Goal Achieved**

![](media/image9.wmf){width="0.5in" height="0.5in"}

**Goal IV-1: Ninety-five percent of full proposals will be received
electronically through FastLane.**

Astate-of-the-art communications and technology infrastructure has been
essential to our successful management of an increasing workload. This
investment also provides incentives for the recruitment and retention of
high-quality employees. We continue to experiment with new means to do
business electronically.

For example, FastLane, our web-based interface with grantee
institutions, was developed in close consultation with the research and
education communities. The most complex use of FastLane is for the
submission of full technical proposals via the web. NSF is the *only*
federal research agency currently receiving proposals electronically on
a routine basis.

  ---------------- ------------ ------------ ------------ ------------ ---------------
  **percent of                                                         
  proposals                                                            
  submitted                                                            
  electronically                                                       
  in fastlanE**                                                        

                   **FY 1997**  **FY 1998**  **FY 1999**  **FY 2000**  **FY 2001**

  **Baseline**     4.4%                                                

  **Goal**                                   25%          60%          95%

  **Actual**                    17%          44%          81%          **99%**[^15]
  ---------------- ------------ ------------ ------------ ------------ ---------------

**Results:** NSF is successful for this goal. Of the 32,141 full
proposals that were submitted in FY 2001, only 286 were not submitted
through FastLane. *This equates to 99% of full proposal submissions
received and processed through FastLane* -- well in excess of the 95%
goal.

The following chart illustrates progress over the past five years:

[Outreach]{.underline}: We have continued aggressive outreach to the
research and education community to promote the use and advantages of
FastLane. We conducted over 30 FastLane presentations and workshops at
scientific meetings, research administration conferences and educational
institutions, and through videoconferencing.

[Customer Service]{.underline}: Throughout FY 2001, our FastLane
Helpdesk continued to assist our external customers.

[Implementation]{.underline}: In view of our success with electronic
submission, in September 2000, we issued Important Notice 126 to
Presidents of Universities and Colleges and Heads of other NSF grantee
institutions reaffirming that effective October 1, 2000, specified
transactions with NSF had to be accomplished electronically via use of
the FastLane system[^16].

[Electronic Signatures]{.underline}: We implemented acceptance of
electronic signatures on incoming proposals in June 2001. In the
remaining three months of FY 2001, and the first three months of FY
2002,19,982 (97%) of 20,451 submissions were received with electronic
signatures.

**Implications for the FY 2002 Performance Plan:** This goal will not be
continued in FY 2002. Electronic submission of proposals via FastLane is
now standard operating procedure at NSF.

**Goal IV-2 -- Electronic Proposal Processing**

**** **Goal Achieved**

**Goal IV-2: In FY 2001 NSF will conduct ten pilot paperless projects
that manage the competitive review process in an electronic
environment.**

This was a new goal in FY 2001, and represented a logical step in our
evolution towards a paperless environment. As we noted above, more than
99% of all proposals are now submitted electronically. But once
proposals are received, our current processes still involve paper.
Ultimately, we want to make our entire proposal and award process an
electronic, or paperless, process.

The pilot paperless project goal was intended to develop the
technological capability to process electronically submitted proposals
through the entire review process *without generating paper within NSF*.

Programs within four of our Directorates participated in the pilot.
Successful accomplishment of the pilot project required all of the
following tasks to be accomplished for proposals in the program:

-   Receipt of all proposals through FastLane

-   Electronic capture of proposal data into the Proposal, Principal
    Investigator and Reviewer System (PARS)

-   Assignment of reviewers accomplished in PARS (if using ad hoc
    reviewers)

-   Submission of reviewer requests through PARS (if using ad hoc
    reviewers)

-   Printed proposals not sent to reviewers

-   Submission of reviews in FastLane

-   Panels, if conducted, utilize the electronic Interactive Panel
    System

-   Funding recommendation by the Program Officer accomplished in PARS

**Results:** NSF is successful for this goal. All ten participants in
the FY 2001 pilot project successfully managed the review process in
their programs electronically, demonstrating the capability for a
paperless review process within NSF[^17].

**Implications for the FY 2002 Performance Plan:** In FY 2002 we will
make various enhancements to several modules used in the paperless
process. The FY 2002 goal doubles the number of projects, will include
more directorates, divisions, and larger programs, and will incorporate
the e-signature in the definition of a paperless project. The intent of
making the FY 2002 pilots broader and more complex is to further
demonstrate the capability and benefits of an internal paperless
process, thus encouraging the majority of our programs to transition to
paperless processing.

**Goal IV -- Video Conferencing**

**** **Goal Achieved**

**Goal IV-3: By the end of FY 2001 NSF will increase usage of a
broad-range of video-conferencing/long distance communications
technology by 100% over the FY 1999 level.**

This was a new goal in FY 2001 and was designed to illustrate and
promote the use of innovative business technologies within NSF. Over the
past several years efforts have been underway to increase our ability to
collaborate worldwide via videoconference. In FY 1999 we monitored
videoconference usage and established a baseline of 50 videoconferences.
During FY 2000 we focused efforts on increasing our technical capability
in this area and marketing the technology to staff and the communities
we serve. For FY 2001 we established a goal of conducting 100
videoconferences--double the FY 1999 baseline.

**Results:** NSF is successful for this goal. We held a total of 142
videoconferences, a 184% increase over FY 1999, and logged about 342
hours of videoconferencing time (both point-to-point and multi-point)
during the course of the year. Staff from six Directorates, seven Staff
Offices, and members of the National Science Board conducted
videoconferences during the year. Our staff used videoconferencing to
hold Advisory Committee meetings, conduct FastLane training sessions for
institutions across the country, conduct site visits, meet with PIs, and
conduct peer review panels. We also held videoconferences with foreign
sites.

+------------+-------------+-------------+-------------+------------+
| **number   |             |             |             |            |
| of         |             |             |             |            |
| videoc     |             |             |             |            |
| onferences |             |             |             |            |
| held**     |             |             |             |            |
+------------+-------------+-------------+-------------+------------+
|            | **FY 1999** | **FY 2000** | **FY 2001** | **FY       |
|            |             |             |             | 2002**     |
+------------+-------------+-------------+-------------+------------+
| **         | ### 50 {#se |             |             |            |
| Baseline** | ction-17 .B |             |             |            |
|            | ox-Heading} |             |             |            |
+------------+-------------+-------------+-------------+------------+
| **Goal**   |             | ###         | 100         | N/A        |
|            |             | N/A {#na .B |             |            |
|            |             | ox-Heading} |             |            |
+------------+-------------+-------------+-------------+------------+
| **Actual** |             | 74          | **          |            |
|            |             |             | 142**[^18] |            |
+------------+-------------+-------------+-------------+------------+

**Implications for the FY 2002 Performance Plan:** In FY 2001
videoconferencing was viewed as a functioning, rather than experimental,
technology. Because videoconferencing is an established practice for us
it will not be continued as a goal. We will, however, continue to
emphasize this technology for current and emerging business
applications. For example, we intend to pilot videoconferencing for
telecommuters and facilitate communication between visiting staff
scientists and engineers and students and colleagues at their home
institutions.

# MANAGEMENT GOALS

# B. NSF Staff

![](media/image9.wmf){width="0.5in" height="0.5in"}

**Goal IV-4 -- Staff Diversity**

**** **Goal Achieved**

**Goal IV-4:** **NSF will show an increase over 1997 in the total number
of hires to S&E positions from underrepresented groups.**

The NSF Strategic Plan notes that a diverse, capable, and motivated
staff is one of the critical factors for our success. We are committed
to diversifying our staff of scientists and engineers (S&E) in both
permanent and visiting positions.

We continue to make progress in diversifying our NSF workforce.

During FY 2001 we conducted a number of activities to increase the
numbers of minorities and underrepresented groups in the S&E staff.
These activities included:

-   Developing internal and external strategies for recruitment efforts,
    with direction and focus provided by the NSF Director and Deputy
    Director.

-   Requiring supervisors to implement recruitment and selection plans
    to address diversity issues specific to their organizations.

-   Soliciting input from senior officials regarding best practices and
    impediments to attracting and maintaining a diverse, professional
    staff.

-   Ensuring diversity goals are reflected in individual performance
    plans.

-   Endorsing comprehensive, nationwide searches for executive
    positions, as well as for the science and engineering staff.

-   Broadening recruitment and outreach efforts through a variety of
    electronic websites and paid advertisements. Vacancy announcements
    are now listed on NSF's website and Directorate home pages, the
    Office of Personnel Management's (OPM) electronic listing, and the
    Department of Labor's Office of Disability Employment Programs Job
    link.

-   Expanding NSF's mailing list for vacancy announcements to over 1,500
    organizations, including Historically Black Colleges and
    Universities (HBCU), Tribal Affiliations, Hispanic Serving
    Organizations, and others.

We now encourage our Custom News Service subscribers to access our
vacancy announcements and to request e-mail notification of specific
vacancies as announcements are posted. These electronic advancements in
disseminating information have revolutionized the ability of the
Foundation to bring its announcements to the attention of the science
and engineering community and the general public.

**Results:** NSF is successful for this goal. In comparison to the
hiring statistics for FY 1997, during FY 2001 we achieved a 138%
increase in the number of female hires and a 47% increase in the number
of minority hires. FY 2001 is the 2^nd^ year we exceeded our goal.

+--------+---------+---------+----------+---------+---------+--------+
| **     |         |         |          |         |         |        |
| appoin |         |         |          |         |         |        |
| tments |         |         |          |         |         |        |
| to     |         |         |          |         |         |        |
| S      |         |         |          |         |         |        |
| cience |         |         |          |         |         |        |
| &      |         |         |          |         |         |        |
| Engin  |         |         |          |         |         |        |
| eering |         |         |          |         |         |        |
| posit  |         |         |          |         |         |        |
| ions** |         |         |          |         |         |        |
|        |         |         |          |         |         |        |
| **from |         |         |          |         |         |        |
| unde   |         |         |          |         |         |        |
| rrepre |         |         |          |         |         |        |
| sented |         |         |          |         |         |        |
| gr     |         |         |          |         |         |        |
| oups** |         |         |          |         |         |        |
+--------+---------+---------+----------+---------+---------+--------+
|        | **FY    | **FY    | **FY     | **FY    | **FY    | **FY   |
|        | 1997**  | 1998**  | 1999**   | 2000**  | 2001**  | 2002** |
+--------+---------+---------+----------+---------+---------+--------+
| **Base | 16      |         |          |         |         |        |
| line** | Female  |         |          |         |         |        |
|        |         |         |          |         |         |        |
|        | 15      |         |          |         |         |        |
|        | M       |         |          |         |         |        |
|        | inority |         |          |         |         |        |
+--------+---------+---------+----------+---------+---------+--------+
| **     |         |         | Efforts  | More    | More    | More   |
| Goal** |         |         | to       | than    | than    | than   |
|        |         |         | attract  |         |         |        |
|        |         |         | underrep | 16      | 16      | 35     |
|        |         |         | resented | Female, | Female, | F      |
|        |         |         | groups   |         |         | emale, |
|        |         |         |          | 15      | 15      |        |
|        |         |         |          | M       | M       | 19     |
|        |         |         |          | inority | inority | Mi     |
|        |         |         |          |         |         | nority |
+--------+---------+---------+----------+---------+---------+--------+
| **Ac   |         |         | Achie    | 35      | **38   |        |
| tual** |         |         | ved[^19] | Female  | Female  |        |
|        |         |         |          |         | **[^20] |        |
|        |         |         |          | 19      |         |        |
|        |         |         |          | M       | **22    |        |
|        |         |         |          | inority | M       |        |
|        |         |         |          |         | inority |        |
|        |         |         |          |         | **[^21] |        |
+--------+---------+---------+----------+---------+---------+--------+

The success we have achieved over the past several years in the
employment of women and minorities is reflected in the total on-board
strength in our S&E corps. Women and minorities now comprise 39% and
21%, respectively, of our current S&E workforce, in comparison to 34%
and 18% in 1997.

**Implications for the FY 2002 Performance Plan:** For FY 2002 we will
increase the baseline against which we measure success to reflect more
recent hiring practices. In lieu of the 1997 baseline (16 women and 15
minorities) the baseline will become the results achieved in FY 2000 (35
women and 19 minorities). The new goal is much more challenging and will
necessitate a more proactive approach in recruiting and retaining
individuals from underrepresented groups.

**Goal IV-5 -- Work Environment**

 **Goal Not Achieved**

**Goal IV-5:** **NSF will establish various baselines that will enable
management to better assess the quality of worklife and work environment
within the Foundation.**

**Performance Indicator: Development of an employee survey.**

This was a new goal for FY 2001. We realized that a systematic method of
collecting relevant information on our human capital and work
environment was needed in order to promote a more effective and
efficient workplace and ensure that the needs of our staff were being
addressed. We are committed to enhancing employee potential and
promoting higher performance.

Fundamental to achieving this goal was the establishment of an
organizational assessment tool (workforce survey) that would be
distributed to our staff. We:

-   Researched, reviewed and analyzed numerous workforce assessment
    tools to help identify an appropriate survey that could be
    administered to staff.

-   Selected, after careful and extensive review, an assessment survey
    prepared by the Office of Personnel Management.

```{=html}
<!-- -->
```
-   Established an internal working group to develop an implementation
    process for this survey within NSF. We also identified employee
    focus groups to review questions and discuss possible concerns and
    responses.

```{=html}
<!-- -->
```
-   Solicited extensive involvement of all levels of staff to determine
    whether the survey needed to be adapted to reflect our academic
    culture and organization. Our senior management provided input
    regarding the scope of the survey questions and the timing of
    implementation. We briefed union officers and stewards on the survey
    process and management's expectations for future actions.

**Results:** NSF was not successful for this goal. While significant
progress was made towards achieving this goal we did not achieve the
goal because the survey was not administered.

**Why we did not achieve this Goal:** Implementation of the survey was
delayed for several reasons, including an extensive process to identify
an appropriate survey instrument, adapting the survey to reflect NSF\'s
culture, and carefully addressing employee, supervisory and policy level
input.

**Steps we will take in FY 2002 to achieve this Goal:** A survey will be
conducted during FY 2002.

**Implications for the FY 2002 Performance Plan:** This goal is
contained in the FY 2002 Performance Plan. This survey will inform the
agency's approach to meeting the standards for success outlined in the
Human Capital initiative of the President's Management Agenda.

**INVESTMENT PROCESS GOALS**

![](media/image11.png)

**VI. INVESTMENT PROCESS GOALS**

Success in achieving our outcome goals is dependent upon the award
portfolio developed by our program staff. The following sections provide
information on how our investment process shapes the awards portfolio
and supports our outcome goals. Investment process goals focus on means
and strategies for successful performance -- in merit review and award
oversight and management processes, broadening participation, and
facilities oversight.

The goals included within this section focus on merit review, customer
service, awards managements and oversight, broadening participation of
our reviewer pool, and facilities. Success in achieving these goals is
dependent upon factors such as high quality merit review, sufficient
staff resources and operating expenses, constraints imposed by
administrative requirements, and electronic information systems that
support the various management processes.

**INVESTMENT PROCESS GOALS**

**A. MERIT REVIEW**

![](media/image9.wmf){width="0.5in" height="0.5in"}

Merit review is the keystone to identification of the most promising
People, Ideas, and Tools and is critical to fostering the highest
standards of excellence and accountability---standards for which NSF is
globally recognized. We evaluate proposals for research and education
projects using two criteria---the intellectual merit of the proposed
activity and the broader impacts of the proposed activity.

Evaluations of proposals and funding decisions made through the process
of merit review rely on evaluation by experts. Each year, more than
200,000 merit reviews are conducted to help program officers evaluate
the proposals submitted for consideration.

The two merit review criteria are:

> **What is the intellectual merit of the proposed activity?**\
> How important is the proposed activity to advancing knowledge and
> understanding within its own field or across different fields? How
> well qualified is the proposer (individual or team) to conduct the
> project? (If appropriate, the reviewer will comment on the quality of
> the prior work.) To what extent does the proposed activity suggest and
> explore creative and original concepts? How well conceived and
> organized is the proposed activity? Is there sufficient access to
> resources?
>
> **What are the broader impacts of the proposed activity?**\
> How well does the activity advance discovery and understanding while
> promoting teaching, training, and learning? How well does the proposed
> activity broaden the participation of underrepresented groups (e.g.,
> gender, ethnicity, disability, geographic, etc.)? To what extent will
> it enhance the infrastructure for research and education, such as
> facilities, instrumentation, networks, and partnerships? Will the
> results be disseminated broadly to enhance scientific and
> technological understanding? What may be the benefits of the proposed
> activity to society?

A merit-based review involves a scientist, engineer, or educator serving
as an NSF Program Officer who reviews all proposals within his/her
program, and includes review by three to ten other persons outside NSF
who are experts in the particular fields represented by the proposal and
are without conflicts of interest. Proposers are invited to suggest
names of persons they believe are especially well qualified to review
the proposal and/or persons who should not review the proposal. These
suggestions may serve as an additional source in the reviewer selection
process at the Program Officer's discretion. Program Officers may obtain
comments from assembled review panels or from site visits before
recommending final action on proposals. Senior NSF staff further review
recommendations for awards and declines. When a decision has been made
(whether an award or a declination), verbatim copies of reviews,
excluding the names of the reviewers, and summaries of review panel
deliberations, if any, are provided to the proposer.

**Goal V-1 -- Use of Merit Review**

**** **Goal Achieved**

**Goal V-1: At least 85% of basic and applied research funds will be
allocated to projects that undergo merit review.**

The vast majority of proposals we receive undergo external merit review.
The Foundation makes a few exceptions to this general requirement in
situations where timeliness is crucial such as for studies of volcanic
eruptions or earthquakes or where objective external reviewers may be
difficult to find. It also considers exceptions when researchers propose
such new ideas that knowledgeable external reviewers do not exist.

Data is collected using OMB's government-wide merit-review
definition[^22] that measures merit-reviewed scientific research as a
percentage of basic and applied research[^23]. This performance goal
applies to federal science, space, and technology agencies. NSF has
established the 85% target to be consistent with the OMB recommended
range of 70% to 90%.

**Results:** NSF is successful for this goal. In FY 2001 we revised our
goal from having 80% of funds allocated to merit-reviewed projects to
85% of funds allocated to merit-reviewed projects. We exceeded that goal
by 3%.

+---------+---------+---------+---------+---------+---------+---------+
| # Perce |         |         |         |         |         |         |
| nt of f |         |         |         |         |         |         |
| unds to |         |         |         |         |         |         |
|  projec |         |         |         |         |         |         |
| ts that |         |         |         |         |         |         |
|  underg |         |         |         |         |         |         |
| o merit |         |         |         |         |         |         |
|  review |         |         |         |         |         |         |
+---------+---------+---------+---------+---------+---------+---------+
|         | **FY    | **FY    | **FY    | **FY    | **FY    | **FY    |
|         | 1997**  | 1998**  | 1999**  | 2000**  | 2001**  | 2002**  |
+---------+---------+---------+---------+---------+---------+---------+
| #       | 85%     |         |         |         |         |         |
| ##### B |         |         |         |         |         |         |
| aseline |         |         |         |         |         |         |
+---------+---------+---------+---------+---------+---------+---------+
| ####    |         |         | N/A     | 80%\*   | 85%     | 85%     |
| ## Goal |         |         |         |         |         |         |
+---------+---------+---------+---------+---------+---------+---------+
| ######  |         | 86%     | 86%     | 87%     | *       |         |
|  Result |         |         |         |         | *88%** |         |
+---------+---------+---------+---------+---------+---------+---------+

\* The 80% estimated goal, recalculated from NSF\'s original goal of
90%, is based on the FY 2000 OMB definition of merit-reviewed scientific
research.

#### Implications for the FY 2002 Performance Plan: An examination of our performance over the last four years shows that we have consistently exceeded our current goal of 85%. Furthermore, we are showing a small increase in the funds allocated to merit-reviewed proposals each year. We will continue to maintain the goal of at least 85% in FY 2002.

\* Goal not established for FY 1997 -- FY 1999.

![](media/image12.wmf)

**Goal V-2 -- Reviewer Use of Both Merit Review Criteria**

 **Goal Not Achieved**

**Goal V-2: NSF performance in implementation of the merit review
criteria is successful when reviewers address the elements of both
generic review criteria.**

On September 20, 1999, NSF issued Important Notice #125 to Presidents of
Universities and Colleges, encouraging Principal Investigators to
address the merit review criterion, "the broader impacts of the proposed
activity", in their proposals and reviews. This criterion addresses the
extent to which proposed activities will: advance discovery and
understanding while promoting teaching, training, and learning; broaden
participation of underrepresented groups; enhance the infrastructure for
research and education; enhance scientific and technological
understanding; and benefit society.

**Results:** This goal was revised for FY 2001[^24]. For FY 2001
external groups of experts reviewed 70 NSF programs with respect to this
performance goal. In analyzing these reports we concluded we were
unsuccessful in achieving this goal[^25].

**Why we did not achieve this Goal:** The two merit review criteria were
not implemented until FY 1998. The FY 2001 assessment includes proposals
reviewed in FY 1998, FY 1999, and FY 2000, and is the first assessment
to review the full implementation of the two criteria.

We believe that a critical factor in our failure to achieve this goal is
the time required for our community to become aware of the importance
that we assign to discussing both merit review criteria within proposals
and within reviews. There are indications that discussion of both
criteria by reviewers has increased since the criteria were implemented
in FY 1998. During FY 2001 we examined a random sample of FY 2001
reviews to determine the extent of reviewer response to the broader
impacts criterion. We found, overall, that approximately 69% of reviews
provided evaluative comments on proposer attention to the broader
impacts criterion. We expect, therefore, that full usage should become
more apparent in the FY 2002 assessments.

**Steps we will take in FY 2002 to achieve this Goal:** In response to a
directive by the Senate Appropriations Committee that NSF review the
procedure and criteria for merit review once the new criteria had been
in place for a year, we issued a contract to the National Academy of
Public Administration (NAPA) to conduct a study of the impact of the new
merit review criteria on the nature of the projects NSF supports. Their
key finding was that it is too soon to make valid judgments about the
impact and effectiveness of the new criteria. The NAPA report also
highlighted the need to improve quantitative measures and performance
indicators to track the objectives and implementation of the new
criteria.

We are continuing to educate reviewers and proposers on the use of both
merit review criteria. We have clarified the meaning of the broader
impacts criterion and stressed the importance of using both criteria. We
will also collect examples of broader impacts and develop a plan to
disseminate them. We have modified program announcements to encourage
proposers to provide information on all relevant aspects of the merit
review criteria in their proposals.

We have added separate screens in FastLane to enable reviewers to
address each merit-review criterion separately. Information for this
goal will be collected from the FastLane database. In FY 2002, we expect
most reviews to be submitted electronically via FastLane. Since there
are separate sections for responses to each of the merit review
criteria, we expect to see an increase in the response rate by reviewers
to both criteria. This will also significantly increase the ease and
reliability with which we will be able to track and count reviews that
address both criteria. External expert judgment will also be used to
enable assessment of our progress towards achieving this goal.

**Implications for the FY 2002 Performance Plan:** This goal will be
modified to reflect our expectation of increasing use of both criteria
in FY 2002.

**Goal V-3 -- Program Officer Use of Merit Review Criteria**

**** **Goal Achieved**

Goal V-3: NSF performance in implementation of the merit review criteria
is successful when Program Officers address the elements of both generic
review criteria when making their award decisions.

After a proposal has been subjected to external peer review a NSF
Program Officer makes a recommendation concerning support of the
proposal. The matters to be discussed in this recommendation are
described in our Proposal and Award Manual, Chapter VI, Section B-4. We
state that "*Program Officers must comment on the intellectual merit of
the proposed activity and the broader impacts of the proposed
activity*."

**Results:** NSF is successful for this goal. This goal was revised for
FY 2001[^26]. For FY 2001 external groups of experts reviewed 70 NSF
programs with respect to this performance goal. Program reports prepared
by external experts during FY 2001 GPRA reporting indicated an
assessment of successful for the Foundation in implementation of both
merit review criteria by Program Officers.

**Implications for the FY 2002 Performance Plan:** This goal will be
continued for FY 2002, and we will take initial steps towards
quantifying this goal.

**Investment Process Goals**

**B. Customer Service**

# 

> ![](media/image9.wmf){width="0.5in" height="0.5in"}

Customer service has a potential impact on the number and quality of
proposals received and thus on our ability to meet all Outcome goals. In
1995, we adopted a set of customer service standards, primarily related
to the merit review process, treating grantees and potential grantees
(*applicants*) as the primary *customers* for NSF's administrative
processes. In a survey, applicants valued three standards most highly:
(1) clear guidelines for proposal content and preparation, (2) a minimum
of three months between release of program announcements and proposal
deadlines, and (3) notification of proposal funding recommendation
within six months of proposal submission.

For our FY 2001 Performance Plan, we focused on the latter two of these
standards, ones to which our staff have devoted special attention since
the standards were adopted. The first of these standards (provision of
clear guidelines) is being addressed in internal processes.

**Goal V-4: -- Time to Prepare Proposals**

**** **Goal Achieved**

**Goal V-4: Ninety-five percent of program announcements will be
available to relevant individuals and organizations at least three
months prior to the proposal deadline or target date.**

We realize that researchers and educators require sufficient time to
prepare submissions. To encourage new investigators and solicit quality
proposals, and based on responses to customer surveys, program
announcements and solicitations should be available a minimum of 90 days
prior to the deadline for submission. We define this time as the time
between the posting of the announcement on the web and the deadline for
proposal submission given in the web posting. This goal is identical to
the FY 1999 and FY 2000 goals.

**Results:** NSF is successful for this goal. After two years of failing
to achieve this goal, we exceeded our goal. All of our program
announcements and solicitations were made available at least 90 days
before the proposal deadline[^27].

+-----------+----------+-----------+-----------+-----------+---------+
| **Percent |          |           |           |           |         |
| of        |          |           |           |           |         |
| program   |          |           |           |           |         |
| anno      |          |           |           |           |         |
| uncements |          |           |           |           |         |
| and       |          |           |           |           |         |
| soli      |          |           |           |           |         |
| citations |          |           |           |           |         |
| available |          |           |           |           |         |
| at least  |          |           |           |           |         |
| 3 months  |          |           |           |           |         |
| prior to  |          |           |           |           |         |
| proposal  |          |           |           |           |         |
| deadline  |          |           |           |           |         |
| or target |          |           |           |           |         |
| dates**   |          |           |           |           |         |
+-----------+----------+-----------+-----------+-----------+---------+
|           | **FY     | **FY      | **FY      | **FY      | **FY    |
|           | 1998**   | 1999**    | 2000**    | 2001**    | 2002**  |
+-----------+----------+-----------+-----------+-----------+---------+
| **B       | 66%      |           | ###       |           |         |
| aseline** |          |           |  {#sectio |           |         |
|           |          |           | n-19 .Box |           |         |
|           |          |           | -Heading} |           |         |
+-----------+----------+-----------+-----------+-----------+---------+
| **Goal**  |          | 95%       | 95%       | 95%       | 95%     |
+-----------+----------+-----------+-----------+-----------+---------+
| *         |          | 75%       | 89%       | **100%** |         |
| *Actual** |          |           |           |           |         |
+-----------+----------+-----------+-----------+-----------+---------+

![](media/image13.wmf)**Implications for the FY 2002 Performance Plan:**
This goal will be maintained in FY 2002.

We are also considering enhancement of one of our corporate systems in
order to track data for this goal. The Foundation is developing a
Program Information Management System (PIMS), which is a relational
database designed to collect information and could be used to track the
progress of publications such as program announcements and
solicitations.

**Goal V-5 -- Time to Decision**

 **Goal Not Achieved**

**Goal V-5: For 70 percent of proposals, be able to tell applicants
whether their proposals have been declined or recommended for funding
within six months of receipt.**

One of the most significant issues raised in customer satisfaction
surveys is the amount of time it takes us to process proposals. We
recognize the importance of this issue, and we are continually reviewing
the steps needed to decrease proposal processing time.

**Results:** We were not successful in achieving this goal. In FY 2001
we processed 62% of all proposals within six months of receipt, a
significant improvement over FY 2000. Nevertheless, we fell short of the
70% goal.

![](media/image14.wmf)One factor leading to delay in processing is that
some programs prefer to conduct merit review by mail rather than by
convening review panels. Mail reviews often take longer to complete. For
example, in FY 2001, 70% of all proposals reviewed by panel-only were
processed within six months, compared to 58% for mail-plus-panel review
and 52% for mail-only review. Another factor is that some programs tend
to hold a few highly rated proposals until the end of the fiscal year,
or even into the next fiscal year, in anticipation that more funds might
become available.

**Steps we will take in FY 2002 to achieve this Goal:** This represents
the third consecutive year we have not achieved this goal. However, we
are encouraged by the fact that in FY 2001 we processed a greater
percentage of our proposals within six months than in each of the
preceding three years. Furthermore, a review of our FY 2001 data shows
that 77% of our proposals were processed in less than seven months.
Thus, an additional 15% of our proposals came within one month of being
processed within our goal.

In FY 2002, we will continue to focus on improving the efficiency of
proposal processing, including the dissemination of best practices to
program staff. We have implemented a series of new electronic processes
designed to improve the efficiency and effectiveness of the merit review
process. New FastLane modules such as the Interactive Panel System and
Electronic Declination, as well as the pilot project to provide
proposals to reviewers electronically (with print-on-demand available),
are reducing processing time and helping our staff to cope with
increasing workloads.

We have sponsored a series of brainstorming sessions for staff at all
levels within the organization to discuss issues, concerns, and
effective practices related to proposal processing time. The results of
these sessions, including effective practices employed by organizations
with excellent processing times, have been widely disseminated
throughout NSF. The sessions also identified a number of key management
issues related to processing time such as the need for timely processing
of declinations and better tracking information on proposals in process.
We have developed a report that tracks proposals through major
processing stages and identifies those that are close to exceeding
recommended timeframes for each stage. This report is produced centrally
and periodically distributed to division directors throughout NSF.

In FY 2002 NSF staff will work towards shortening the award processing
time by making more effective use of electronic mechanisms in conducting
the review, working cooperatively to reduce overloads and bottlenecks,
and by carefully tracking the stage of processing and receipt date of
all proposals. Some internal organizations are considering eliminating
the practice of holding over proposals for potential funding until the
next fiscal year. Some have added "performance on prompt handling of
proposals" to the performance evaluation criteria of their staff.

**Implications for the FY 2002 Performance Plan:** This goal will be
maintained in FY 2002. We believe the expanded use of key electronic
processes and adoption of effective practices identified in the FY 2001
brainstorming sessions will lead to our meeting or exceeding the 70%
goal.

# INVESTMENT PROCESS GOALS 

# 

# C. AWARDS

> ![](media/image9.wmf){width="0.5in" height="0.5in"}

The size and duration of NSF awards impact research and education
activities at many institutions. Increasing award size and duration will
allow scientists and engineers to devote more time to productive
research and education in comparison to the time spent preparing
proposals. Adequate award size and duration are important both to
obtaining high quality proposals and to ensuring that proposed work can
be accomplished as planned.

In FY 2002, NSF will continue efforts to address Foundation-wide
concerns about research and education grant size and duration -- this
priority is highlighted in NSF's Strategic Plan and is one of the new
management reform activities for NSF highlighted by OMB.

**Goal V-6a -- Increased Average Annualized Award Size**

**** **Goal Achieved**

**Goal V-6a: NSF will increase the average annualized award size for
research projects to \$110,000.**

Increasing award size is a new goal[^28]. We want to reach an average
annualized award size of \$150,000 by FY 2005.

Adequate award size is important both for attracting high-quality
proposals and for ensuring that proposed work can be accomplished as
planned. Larger awards increase the efficiency of the system by allowing
scientists and engineers to devote a greater portion of their time to
actual research rather than to proposal writing and other administrative
work.

**Results:** We were successful in achieving and exceeding this goal.

+----------+-----------+-----------+-----------+-----------+----------+
| *        |           |           |           |           |          |
| *average |           |           |           |           |          |
| an       |           |           |           |           |          |
| nualized |           |           |           |           |          |
| award    |           |           |           |           |          |
| size for |           |           |           |           |          |
| research |           |           |           |           |          |
| pr       |           |           |           |           |          |
| ojects** |           |           |           |           |          |
+----------+-----------+-----------+-----------+-----------+----------+
|          | **FY      | **FY      | **FY      | **FY      | **FY     |
|          | 1998**    | 1999**    | 2000**    | 2001**    | 2002**   |
+----------+-----------+-----------+-----------+-----------+----------+
| **Ba     | \$90,000  |           |           |           |          |
| seline** |           |           |           |           |          |
+----------+-----------+-----------+-----------+-----------+----------+
| **Goal** |           |           |           | \$110,000 | \        |
|          |           |           |           |           | $113,000 |
+----------+-----------+-----------+-----------+-----------+----------+
| **       |           | \$94,000  | \$105,800 | ### *     |          |
| Actual** |           |           |           | *\$113,6 |          |
|          |           |           |           | 01**[^29] |          |
|          |           |           |           |  {#sectio |          |
|          |           |           |           | n-21 .Box |          |
|          |           |           |           | -Heading} |          |
+----------+-----------+-----------+-----------+-----------+----------+

**Implications for the FY 2002 Performance Plan:** Our new goal for FY
2002 is based on our performance in FY 2001. Our goal for FY 2002 will
be an average annualized award size of \$113,000.

**Goal V-6b -- Increased Average Award Duration**

 **Goal Not Achieved**

**Goal V-6b: NSF will increase the average duration of awards for
research projects to at least three years.**

Our goal is to reach an average award duration of 4 years by FY
2005[^30]. Increasing award duration was a new goal in FY 2001. The
award duration goal built on a FY 1999 goal (the duration goal was
dropped in FY 2000 and reinstated in FY 2001).

Longer award terms are important in increasing the effectiveness of
Principal Investigators and graduate students. Less time is spent
preparing proposals, and graduate students are able to have more time to
do their thesis work.

**Results:** We were not successful in achieving this goal.

+----------+-----------+-----------+-----------+-----------+----------+
| *        |           |           |           |           |          |
| *average |           |           |           |           |          |
| award    |           |           |           |           |          |
| DURATION |           |           |           |           |          |
| for      |           |           |           |           |          |
| research |           |           |           |           |          |
| pr       |           |           |           |           |          |
| ojects** |           |           |           |           |          |
+----------+-----------+-----------+-----------+-----------+----------+
|          | **FY      | **FY      | **FY      | **FY      | **FY     |
|          | 1998**    | 1999**    | 2000**    | 2001**    | 2002**   |
+----------+-----------+-----------+-----------+-----------+----------+
| **Ba     | 2.7 years |           |           |           |          |
| seline** |           |           |           |           |          |
+----------+-----------+-----------+-----------+-----------+----------+
| **Goal** |           | 2.8 years | N/A       | 3.0 years | 3.0      |
|          |           |           |           |           | years    |
+----------+-----------+-----------+-----------+-----------+----------+
| **       |           | 2.8 years | 2.8 years | ### **2. |          |
| Actual** |           |           |           | 9 years** |          |
|          |           |           |           | [^31] {#y |          |
|          |           |           |           | ears .Box |          |
|          |           |           |           | -Heading} |          |
+----------+-----------+-----------+-----------+-----------+----------+

![](media/image15.wmf){width="3.5in" height="2.39375in"}

**Why we did not achieve this Goal:** Sufficient resources were not
available to achieve both the award size and award duration goals. NSF
focused its efforts on increasing average annualized award size and
reached its goal for FY 2001.

**Steps we will take in FY 2002 to achieve this Goal:** Progress on this
goal is budget dependent. Program Directors must balance
competing/multiple requirements: increasing award size, increasing
duration of awards, or making fewer awards. We will continue to focus on
increasing award size and duration in order to improve the efficiency of
the research process.

We have contracted with Mathematica Policy Research, Inc. to assist in
the development and administration of two surveys -- one for Principal
Investigators and one for institutions -- on issues related to the
appropriate size and duration of awards. The goal of the study is to
understand how to improve the overall efficiency of the research process
and to understand the impact of NSF research and education awards.

**Implications for the FY 2002 Performance Plan:** We will maintain the
FY 2001 goal of 3.0 years for the average duration of awards for
research and education grants.

**Goal V-7 -- Maintaining Openness in the System**

 **Goal Not Achieved**

**Goal V-7: NSF will award 30% of its research grants to new
investigators.**

We believe it is important that the proposal and award process be open
to new people and new ideas to help ensure that NSF is supporting
research and education at the frontier of science and engineering. We
are committed to maintaining openness in the system and will strive to
increase the percentage of awards to new investigators.

**Results:** We were not successful in achieving this goal. The
percentage of competitive research and education grants issued to new
investigators was 28%, the same as in FY 2000, and one percent higher
than in FY 1999.

+----------+-----------+-----------+-----------+-----------+----------+
| **pe     |           |           |           |           |          |
| rcentage |           |           |           |           |          |
| of       |           |           |           |           |          |
| com      |           |           |           |           |          |
| petitive |           |           |           |           |          |
| research |           |           |           |           |          |
| grants   |           |           |           |           |          |
| issued   |           |           |           |           |          |
| to new   |           |           |           |           |          |
| investi  |           |           |           |           |          |
| gators** |           |           |           |           |          |
+----------+-----------+-----------+-----------+-----------+----------+
|          | **FY      | **FY      | **FY      | **FY      | **FY     |
|          | 1997**    | 1998**    | 1999**    | 2000**    | 2001**   |
+----------+-----------+-----------+-----------+-----------+----------+
| **Ba     | 27%       |           |           |           |          |
| seline** |           |           |           |           |          |
+----------+-----------+-----------+-----------+-----------+----------+
| **Goal** |           |           | 30%       | 30%       | 30%      |
+----------+-----------+-----------+-----------+-----------+----------+
| **       |           | 27%       | 27%       | 28%       | ### **  |
| Actual** |           |           |           |           | 28%** {# |
|          |           |           |           |           | section- |
|          |           |           |           |           | 22 .Box- |
|          |           |           |           |           | Heading} |
+----------+-----------+-----------+-----------+-----------+----------+

![](media/image16.wmf){width="3.75in" height="2.464583333333333in"}**Why
we did not achieve this Goal:** This has been a difficult and
challenging goal for NSF. In spite of our focused efforts, we have been
unable to achieve this goal, although we have come close in the past few
years. It is not clear why we have not attained the goal, as budgets,
quality of proposals, experience of Principal Investigators, and other
factors all come into the equation.

**WHAT WE WILL CONTINUE TO DO IN FY 2002:** We will continue to actively
seek creative and innovative proposals from new investigators. We will
continue our outreach efforts. Program staff will continue to attend
scientific meetings, conferences, and conventions and will conduct site
visits to promote awareness of the research opportunities at NSF and to
encourage new investigators to submit proposals. We will examine trends,
such as whether the pool of new investigators is smaller than in
previous years or whether they are submitting fewer proposals.

**Implications for the FY 2002 Performance Plan:** This openness goal is
not included in the FY 2002 Performance Plan because we wish to fully
consider whether this goal provides a good measure of openness in the
system. Thus, we intend to examine a variety of interrelated parameters
that affect it, and on that basis consider another.

# INVESTMENT PROCESS GOALS 

# 

# D. BROADENING PARTICIPATION

> ![](media/image9.wmf){width="0.5in" height="0.5in"}

We are strongly committed to increasing the participation of science and
engineering researchers, educators and students from groups currently
underrepresented in the science and engineering enterprise in all NSF
activities. Congress has enacted legislation giving NSF explicit
responsibility for addressing issues of equal opportunity in science and
engineering. This assignment of responsibility reflected the serious
underrepresentation of women, minorities, and persons with disabilities
in the science and engineering workforce, and, although progress has
been made, underrepresentation persists.

Recognizing that progress toward all outcome goals for research and
education requires maximum diversity of intellectual thought, NSF is
focusing its attention on enhancing the participation of groups
currently underrepresented in science and engineering in all its
programs. In order to realize this increased participation, and so
contribute to the development of a dynamic, diverse, human resource pool
in science and engineering over the next decade NSF seeks to:

-   Increase the participation of scientists and engineers from
    underrepresented groups in NSF\'s merit review process;

-   Increase the participation of scientists and engineers from
    underrepresented groups in NSF\'s workshops and conferences;

-   Increase the number of proposals submitted by and awards made to
    scientists and engineers from underrepresented groups; and

-   Increase the number of scientists and engineers from
    underrepresented groups appointed by NSF to its staff.

At present we are focusing on the first and fourth of these efforts. NSF
is committed to maintaining openness in the system and strives to
increase the percentage of awards to new investigators.

**Goal V-8 -- Broadening Participation: Reviewer Pool Diversity**

**** **Goal Achieved**

**Goal V-8: NSF will begin to request voluntary demographic data
electronically from all reviewers to determine participation levels of
members of underrepresented groups in the NSF reviewer pool.**

NSF is strongly committed to increasing the participation of science and
engineering researchers, educators and students from groups currently
underrepresented in the science and engineering enterprise in all NSF
activities. Congress has enacted legislation giving NSF explicit
responsibility for addressing issues of equal opportunity in science and
engineering.

FY 2001 marks the first time we have focused attention on reviewer pool
data. To establish the baseline, we have begun to gather the appropriate
voluntary data from the reviewers. A baseline for FY 2002 will be
derived from this data.

**Results:** We were successful in achieving this goal. The reviewer
system in FastLane was revised to gather voluntary demographic data.

**Implications for the FY 2002 Performance Plan:** We will continue a
related goal in FY 2002. Our FY 2002 goal is to establish a baseline for
participation of members of underrepresented groups in NSF proposal
review activities. We will seek voluntary demographic data from all
reviewers electronically, and encourage increased participation of
members of underrepresented groups in NSF conferences and workshops
where they may come into contact with NSF program staff. We will
continue to encourage members of underrepresented groups in science and
engineering fields to participate in the NSF merit review system as
reviewers and widely disseminate information about opportunities to
participate in our merit review process as a reviewer or panel member.

**INVESTMENT PROCESS GOALS**

**E. FACILITIES**

![](media/image9.wmf){width="0.5in" height="0.5in"}

NSF has responsibility for supporting the operation of multiple user
facilities that provide state-of-the-art equipment with unique
capabilities. In addition, we put a high premium on initial planning for
construction and upgrade of facilities. Planning for unique,
state-of-the-art facilities must take into account the exploratory
nature of the facilities themselves as such facilities test the limits
of technological capability.

Every year, in the President's Budget Request, we set out a cost plan
and schedule for major construction and upgrade projects currently
underway or planned for initiation in the Major Research Equipment and
Facilities Construction account. Cost plans and schedules are also
developed for other construction and upgrade projects funded through the
Research and Related Activities Account. We have established performance
goals and measurements with respect to these plans and expect each
construction and upgrade activity to meet these performance goals. We
consult with other agencies to avoid duplication and to optimize
capabilities available to American researchers and educators, and
cooperate with other agencies and international partners in construction
of facilities where it will facilitate use across broad communities of
researchers. We manage facilities in the Antarctic that are used by all
federal agencies.

In FY 2001 24% of our budget was allocated to the support of "Tools."
Within Tools, FY 2001 funding for the Major Research Equipment (MRE)
account was approximately \$119 million, an increase of \$14 million
over FY 2000.

Although we have done well in the past in keeping large projects on
schedule and within budget, OMB asked us to develop a plan for costing,
approval, and oversight of major facility projects. In response, we have
completed a Large Facility Projects Management and Oversight Plan that
was submitted to OMB in September 2001. This new facilities plan has
four major foci:

-   Enhance organizational and staff capabilities to improve
    coordination, collaboration, and shared learning among our staff and
    external partners;

-   Implement comprehensive guidelines and procedures for all aspects of
    facilities planning, management, and oversight;

-   Improve the process for reviewing and approving Large Facility
    Projects; and

-   Practice coordinated and proactive oversight of all facility
    projects to ensure success.

Further development and implementation of the plan is continuing.

We have established a new position--Deputy, Large Facility Projects--to
enable the efficient and effective evolution of our large facility
projects from their pre-formulation through operations. This position
will be filled in FY 2002.

In order to report on the government performance goals related to
Facility Operations and Construction and Upgrades, we initiated, in FY
1999, development of a new Facilities Reporting System. This is linked
to the Performance Reporting System, a module of the existing FastLane
system. The module is used to collect information on operations and
construction from Facilities Managers external to NSF. As is the case
with any new data collection effort, we expect the quality of the
information provided to improve as NSF's Program Officers and external
facilities managers gain experience with gathering and reporting the
required data.

In FY 2001 NSF engaged PricewaterhouseCoopers LLP (PwC) to review the
process for collection and reporting of GPRA data for the facilities
goals. PwC's recommendations, along with NSF's own review of the
facilities goals and associated data collection methods, will be
examined in FY 2002. Necessary changes will be identified and an
implementation plan for the changes will be developed.

The facility goals that follow are organized in two categories -- (1)
*Construction and Upgrade of Facilities and (2) Operations and
Management of Facilities*. Our goals are based on the general goals for
facilities construction and operations outlined in the "General Science,
Space and Technology" (Function 250) chapter of the President's FY 2001
Budget Request. These goals apply to the federal science, space and
technology agencies (NSF, NASA, DoE).

**Goal V-9a -- Annual Construction and Upgrade Expenditures**

**** **Goal Achieved**

**Goal V-9a: For 90 percent of facilities, keep construction and
upgrades within annual expenditure plan, not to exceed 110 percent of
estimates.**

This FY 2001 goal was slightly revised from the FY 2000 goal. In FY 2000
one hundred percent of facilities were required to meet the goal for NSF
to be considered successful. In FY 2001 the goal was revised so that we
were considered successful if at least 90% of facilities kept
construction and upgrade expenditures within 110% of their estimates.
This change was made because while we place great importance on accurate
planning for construction and upgrade of facilities we recognize that
the unique, state-of-the-art projects being supported stretch the limits
of technological capability. As a result there may be unforeseen
expenditures. Nevertheless, we expect that the vast majority of our
projects will be within budget. However, we do not believe the agency
should be considered unsuccessful overall in this area if a small
percentage of facilities are unable to meet the goal. Therefore, to
assure that we have realistic and achievable goals, we reestablished the
target level of success at 90% of the facilities for FY 2001. We will
evaluate this goal over time to determine if 90% is the appropriate
level.

**Results:** We were successful in achieving this goal. Of the
twenty-five construction and upgrade projects supported by NSF,
twenty-four (96%) were within 110% of annual expenditure plans. The
expenditures of nine projects were equal to the planned annual cost,
twelve projects' expenditures were less than the estimated cost and
three projects had annual costs greater than but within 110% of their
estimate.

+------------+-------------+-------------+-------------+------------+
| **annual   |             |             |             |            |
| co         |             |             |             |            |
| nstruction |             |             |             |            |
| and        |             |             |             |            |
| upgrade    |             |             |             |            |
| expe       |             |             |             |            |
| nditures** |             |             |             |            |
+------------+-------------+-------------+-------------+------------+
|            | **FY 1999** | **FY 2000** | **FY 2001** | **FY       |
|            |             |             |             | 2002**     |
+------------+-------------+-------------+-------------+------------+
| **Goal**   | Keep within | Keep within | For 90% of  | For 90% of |
|            | annual      | annual      | facilities, | f          |
|            | expenditure | expenditure | keep within | acilities, |
|            | plan, not   | plan, not   | annual      | keep       |
|            | to exceed   | to exceed   | expenditure | within     |
|            | 110% of     | 110% of     | plan, not   | annual     |
|            | estimates.  | estimates.  | to exceed   | e          |
|            |             |             | 110% of     | xpenditure |
|            |             |             | estimates.  | plan, not  |
|            |             |             |             | to exceed  |
|            |             |             |             | 110% of    |
|            |             |             |             | estimates. |
+------------+-------------+-------------+-------------+------------+
| **Actual** | Majority of | 11 of 11    | ### **24 o |            |
|            | projects    | (100%)      | f 25 (96%)  |            |
|            | were within | projects    | projects we |            |
|            | 110% of     | were within | re within 1 |            |
|            | estimates.  | 110% of     | 10% of esti |            |
|            |             | estimates.  | mates**[^32 |            |
|            |             |             | ]. {#of-25- |            |
|            |             |             | 96-projects |            |
|            |             |             | -were-withi |            |
|            |             |             | n-110-of-es |            |
|            |             |             | timates. .B |            |
|            |             |             | ox-Heading} |            |
+------------+-------------+-------------+-------------+------------+

**implications for the fy 2002 performance plan:** This goal will be
maintained in FY 2002. We have established a new position--Deputy, Large
Facility Projects--to enable the efficient and effective evolution of
our large facility projects from their pre-formulation through
operations.

**Goal V-9b -- Meeting Annual Schedule Milestones**

 **Goal Not Achieved**

**Goal V-9b: Ninety percent of facilities will meet all annual schedule
milestones by the end of the reporting period.**

The FY 2001 goal is slightly revised from the FY 2000 goal. In FY 2000,
for NSF to be considered successful, one hundred percent of facilities
were required to meet all annual schedule milestones within 110% of
estimates. In FY 2001 we have modified this goal and consider successful
achievement to be when at least 90% of facilities meet all major
schedule milestones by the end of the reporting period. This change was
made because while we place great importance on accurate planning for
construction and upgrade of facilities we recognize that the unique,
state-of-the-art projects being supported stretch the limits of
technological capability and there may be unexpected construction
delays. While we expect the vast majority of projects to be on schedule,
we do not believe we should be considered unsuccessful overall in this
area if a small percentage of facilities are unable to meet the goal.
Therefore, to assure that we had realistic and achievable goals, we
reestablished the target level of success at 90% of the facilities for
FY 2001. We will evaluate this over time to determine if 90% is the
appropriate level.

**Results:** For FY 2001, of the 25 construction and upgrade projects we
supported, 21 (84%) met all annual schedule milestones by the end of the
reporting period.

+------------+-------------+-------------+-------------+------------+
| **annual   |             |             |             |            |
| schedule   |             |             |             |            |
| mi         |             |             |             |            |
| lestones** |             |             |             |            |
+------------+-------------+-------------+-------------+------------+
|            | **FY 1999** | **FY 2000** | **FY 2001** | **FY       |
|            |             |             |             | 2002**     |
+------------+-------------+-------------+-------------+------------+
| **Goal**   | ### Cons    | C           | 90% of      | 90% of     |
|            | truction an | onstruction | facilities  | facilities |
|            | d upgrades  | and         | will meet   | will meet  |
|            | within annu | upgrades    | all major   | all major  |
|            | al schedule | within      | annual      | annual     |
|            | , time requ | annual      | schedule    | schedule   |
|            | ired for ma | schedule,   | milestones  | m          |
|            | jor compone | time        | by the end  | ilestones. |
|            | nts within  | required    | of the      |            |
|            | 110% of est | for major   | reporting   |            |
|            | imates. {#c | components  | period.     |            |
|            | onstruction | within 110% |             |            |
|            | -and-upgrad | of          |             |            |
|            | es-within-a | estimates.  |             |            |
|            | nnual-sched |             |             |            |
|            | ule-time-re |             |             |            |
|            | quired-for- |             |             |            |
|            | major-compo |             |             |            |
|            | nents-withi |             |             |            |
|            | n-110-of-es |             |             |            |
|            | timates. .B |             |             |            |
|            | ox-Heading} |             |             |            |
+------------+-------------+-------------+-------------+------------+
| ######     | Majority of | 7 of 11     | ### **21   |            |
| ### Actual | projects    | (64%)       |  of 25 (84% |            |
|            | were within | projects    | ) projects  |            |
|            | 110% of     | were within | met all maj |            |
|            | estimates.  | 110% of     | or annual s |            |
|            |             | estimates.  | chedule mil |            |
|            |             |             | estones by  |            |
|            |             |             | the end of  |            |
|            |             |             | the reporti |            |
|            |             |             | ng period** |            |
|            |             |             | [^33]**.**  |            |
|            |             |             | {#of-25-84- |            |
|            |             |             | projects-me |            |
|            |             |             | t-all-major |            |
|            |             |             | -annual-sch |            |
|            |             |             | edule-miles |            |
|            |             |             | tones-by-th |            |
|            |             |             | e-end-of-th |            |
|            |             |             | e-reporting |            |
|            |             |             | -period. .B |            |
|            |             |             | ox-Heading} |            |
+------------+-------------+-------------+-------------+------------+

**WHY WE DID NOT ACHIEVE THIS GOAL:** In some cases, projects were
unable to meet all major annual schedule milestones within the reporting
period due to circumstances beyond the control of the facility manager,
such as an earthquake. Other examples of why some projects were not able
to meet annual schedule milestones are underestimates of project
complexity, technical problems and personnel vacancies.

**Steps we will take in FY 2002 to achieve this Goal:** We will continue
to work with our awardees to identify obstacles to successful
performance and together implement plans to avoid these same obstacles
or to mitigate their consequences in the future.

**Implications for the FY 2002 Performance Plan:** The FY 2002 goal will
be revised because of our experiences during FY 2001. We found that the
definition of "reporting period" was somewhat ambiguous. As a result,
the goal for FY 2002 has been changed to "Ninety percent of
construction/upgrade projects will meet all major annual schedule
milestones." Relevant definitions of terms used in reporting will be
clarified.

**Goal V-9c -- Total Cost**

**** **Goal Achieved**

**Goal V-9c: For all construction and upgrade projects initiated after
1996, when current planning processes were put in place, keep total cost
within 110 percent of estimates made at the initiation of
construction.**

We recognize that construction and upgrade projects may experience both
cost and schedule overruns. Our goal, since FY 1999, is that all
projects/upgrades will keep within 110% of their initial estimated total
costs.

**Results:** We were successful in achieving this goal. One project was
completed in FY 2001. The total cost of the project was equal to the
estimated total cost. This goal was not applicable in FY 1999 and FY
2000 since no projects were completed.

  ---------------- --------------- --------------- ------------------ ---------------
  **construCtion                                                      
  and upgrade                                                         
  total cost**                                                        

                   **FY 1999**     **FY 2000**     **FY 2001**        **FY 2002**

  **Goal**         For all         For all         For all            For all
                   construction    construction    construction and   construction
                   and upgrade     and upgrade     upgrade projects   and upgrade
                   projects        projects        initiated after    projects
                   initiated after initiated after 1996, keep total   initiated after
                   1996, keep      1996, keep      cost within 110%   1996, keep
                   total cost      total cost      of estimates made  total cost
                   within 110% of  within 110% of  at the initiation  within 110% of
                   estimates made  estimates made  of construction.   estimates made
                   at the          at the                             at the
                   initiation of   initiation of                      initiation of
                   construction.   construction.                      construction.

  **Actual**       No projects     No projects     **** **One        
                   completed.      completed.      project was        
                                                   completed. The     
                                                   actual total cost  
                                                   was equal to the   
                                                   estimated total    
                                                   cost**[^34]**.**   
  ---------------- --------------- --------------- ------------------ ---------------

**Implications for the FY 2002 Performance Plan:** This goal will be
maintained in FY 2002. We have established a new position -- Deputy,
Large Facility Projects -- to enable the efficient and effective
evolution of our large facility projects from their pre-formulation
through operations.

**Goal V-10 -- Operating Time**

 **Goal Not Achieved**

**Performance Goal V-10: For 90 percent of facilities, keep operating
time lost due to unscheduled downtime to less than 10 percent of the
total scheduled operating time.**

Our FY 2001 goals are based on government-wide goals established by OMB
for science and technology agencies (NSF, NASA and the Department of
Energy) that support construction projects and have responsibility for
managing facilities.

The "operating time" goal has been revised from 100% of facilities to
90% because we recognize that while some facilities may occasionally
have a failure rate greater than 10%, this is balanced overall by
facilities that operate more reliably. We expect that the vast majority
of facilities will keep operating time lost due to unscheduled downtime
to less than 10% of the total operating time. We do not believe the
agency should be considered unsuccessful if a small percentage of the
facilities are, at times, unable to meet this goal. Therefore, to
provide the flexibility necessary for NSF to report realistic goals, we
reestablished the level deemed "successful" at 90% of the facilities.
This change will be evaluated over time to determine if 90% is the
appropriate level for this goal.

**Results:** We were not successful in achieving this goal. Of the 29
reporting facilities, 25 (86%) met the goal of keeping unscheduled
downtime to below 10% of the total scheduled operating time. Four
reported unscheduled downtime greater than 10%.

+------------+-------------+-------------+-------------+------------+
| *          |             |             |             |            |
| *operating |             |             |             |            |
| time**     |             |             |             |            |
+------------+-------------+-------------+-------------+------------+
|            | **FY 1999** | **FY 2000** | **FY 2001** | **FY       |
|            |             |             |             | 2002**     |
+------------+-------------+-------------+-------------+------------+
| **Goal**   | ### K       | Keep        | For 90% of  | For 90% of |
|            | eep operati | operating   | facilities, | f          |
|            | ng time los | time lost   | keep        | acilities, |
|            | t due to un | due to      | operating   | keep       |
|            | scheduled d | unscheduled | time lost   | operating  |
|            | owntime to  | downtime to | due to      | time lost  |
|            | less than 1 | less than   | unscheduled | due to     |
|            | 0% of the t | 10% of the  | downtime to | u          |
|            | otal schedu | total       | less than   | nscheduled |
|            | led operati | scheduled   | 10% of the  | downtime   |
|            | ng time. {# | operating   | total       | to less    |
|            | keep-operat | time.       | scheduled   | than 10%   |
|            | ing-time-lo |             | operating   | of the     |
|            | st-due-to-u |             | time.       | total      |
|            | nscheduled- |             |             | scheduled  |
|            | downtime-to |             |             | operating  |
|            | -less-than- |             |             | time.      |
|            | 10-of-the-t |             |             |            |
|            | otal-schedu |             |             |            |
|            | led-operati |             |             |            |
|            | ng-time. .B |             |             |            |
|            | ox-Heading} |             |             |            |
+------------+-------------+-------------+-------------+------------+
| **Actual** | Majority of | 22 of 26    | **25 of 29 |            |
|            | facilities  | (85%)       | (86%)       |            |
|            | successful. | reporting   | reporting   |            |
|            |             | facilities  | facilities  |            |
|            |             | met goal.   | met         |            |
|            |             |             | goal*       |            |
|            |             |             | *[^35]**.** |            |
+------------+-------------+-------------+-------------+------------+

**Why we did not achieve this Goal:** Some causes of unscheduled
downtime in excess of 10% of total scheduled operating time were outside
the control of the facility manager, such as electric power supply
interruption and equipment failure. Other causes ranged from sub-par
performance of new instruments early in their commissioning to
unanticipated failure and downtime for repair.

**Steps we will take in FY 2002 to achieve this Goal:** NSF program
staff will continue to work with project managers to identify obstacles
to successful performance and to ensure that progress will be made
toward the achievement of this goal in FY 2002.

**Implications for the FY 2002 Performance Plan:** This goal will be
maintained in FY 2002. We have established a new position -- Deputy,
Large Facility Projects -- to enable the efficient and effective
evolution of our large facility projects from their pre-formulation
through operations.

**OTHER INFORMATION**

![](media/image17.png)

# VII. Assessment and Evaluation Process

We employ a mix of both qualitative and quantitative goals, and make use
of both qualitative information and quantitative data in determining
annual progress towards achieving goals. Our outcome goals are generally
expressed in a qualitative form, and most management goals and
investment process goals are quantitative.

**MANAGEMENT AND INVESTMENT PROCESS GOALS**

## 

We make use of internal data systems to monitor and report progress in
achieving the quantitative management goals and investment process
goals. For these goals, performance results are assessed and reviewed by
our administrative staff and managers, with selected goals audited by
external third parties. The two qualitative investment process goals
(Goals V-2 and V-3) are addressed by external experts who participate in
COV and AC reviews. Selected results are verified and validated by a
third party.

## 

The assessment process for the quantitative goals is straightforward. We
collect relevant data using internal corporate data systems and compare
the result with the performance level targeted for the fiscal year.
Progress towards achievement of most quantitative goals is reviewed by
senior management on a quarterly basis. In FY 2000, an agency-wide GPRA
module that collects data relevant to the quantitative goals was created
to allow staff to track progress throughout the year.

## **OUTCOME GOALS**

## 

We have traditionally made use of various types of assessments and
evaluations to monitor non-quantitative research and education outcomes,
the quality of our investments, and the processes we use. Formalized
examination takes place during merit review of proposals, COV and AC
assessments, and GPRA reporting. Additionally, programs and plans are
assessed and evaluated throughout the year on a continuing basis by NSF
staff. Elements of GPRA reporting are highlighted in the figure below.

**Project Assessment During NSF Merit Review**

Applicants and grantees provide results from previous NSF support,
information about existing facilities and equipment available to conduct
the proposed activity, biographical information on the Principal
Investigators, other sources of support, federally required
certifications and certifications specific to NSF. Such information is
required at the time of application, at the time of an award, and in
annual and final project reports. It is reviewed by NSF staff, is
utilized during merit review, and is available to external committees
(COVs and ACs) conducting performance assessment. The merit review
process provides a rigorous, first phase of assessment of NSF's research
and education portfolio. Thus, at the onset, this process selects for
support only the most competitive one-third of proposals submitted for
consideration.

## Program Officers review the annual progress of awards. The progress report includes information on significant accomplishments, progress achieved in the prior year, plans for the next year, and points out issues that may impact progress or completion of the project on schedule and within budget. On approval of this report by the Program Officer, NSF releases funds for the ensuing year. 

All materials associated with the review of a proposal as well as
subsequent annual reports are available to Committees of Visitors. Our
staff also prepare materials (reports, evaluations, highlights) for use
by COVs and ACs in developing their reports and making their
assessments.

![](media/image1.wmf)

**Program Assessment by Committees of Visitors (COVs)**

NSF's Committees of Visitors provide program assessments that are used
both in program management and in our annual GPRA reporting. Included
are assessments for outcome goals and for the two qualitative investment
process goals dealing with the implementation of the merit review
criteria. In the past, COVs have traditionally assessed the integrity
and efficiency of the processes for proposal review. With the full
implementation of GPRA in FY 1999, we added a retrospective GPRA
assessment component (both outputs and outcomes) to their
responsibilities.

Each COV typically consists of five to twenty external experts who
review one or more programs over a two or three day period. These
experts are selected to ensure independence, programmatic coverage, and
balanced representation. They typically represent academia, industry,
government, and the public sector.

Each year, COVs assess approximately one-third of our progams, and
review their performance over the previous three years. In FY 2001,
about 32% of NSF's portfolio of 220 programs was evaluated by COVs for
quality of process and progress made in achieving NSF's FY 2001 goals.
In FY 2000, about 37% of NSF's portfolio was evaluated by COVs. The
remaining portion of NSF's portfolio will be evaluated by COVs in FY
2002 to complete the full three-year cycle of assessment of NSF
programs.

In FY 2001, approximately 250 COV members participated in program review
and performance assessment. The 19 COV reports generated covered 70 of
our approximately 220 programs (see Appendix II for a schedule of
program evaluations). Typically, there are fewer COV reports than
programs because some reports evaluate multiple programs.

All COVs are asked to complete a report template with questions
addressing how programs contribute to NSF's goals. Committees of
Visitors are asked to address (A) the integrity and efficiency of the
*processes* involved in proposal review; and (B) the results, including
quality and other factors, of NSF's investments. In determining whether
there has been significant achievement with respect to the prescribed
performance indicators, COV members use their collective
experienced-based norms.

The FY 2001 COVs were asked to judge whether our programs were
successful or not in achieving Outcome Goals III-1a, III-2, and III-3,
and in implementing the merit review criteria (Investment Process Goals
V-2 and V-3). COVs are asked to justify their judgements and provide
supporting examples or statements illustrating success and progress
toward GPRA goals.

COVs are generally subcommittees of NSF Directorate Advisory Committees.
As such, their reports, along with responses from the responsible
Directorate addressing recommendations made by the COVs, are submitted
to the parent Advisory Committee. The reports are also reviewed by NSF
staff.

**Advisory Committee (AC) Reporting on Directorate/Office Performance**

Advisory Committees advise the seven directorates and the Office of
Polar Programs. They are typically composed of 18-25 external experts
who have broad experience in academia, industry, and government.
Advisory Committees are chartered and hence are subject to Federal
Advisory Committee Act (FACA) rules. The role of the ACs is to provide
advice on priorities, address program effectiveness, review COV reports
and directorate responses to COV recommendations, and assess directorate
progress in achieving NSF-wide GPRA goals.

The ACs have full access to all available data sources to carry out
their assessments. Their review and assessment process culminates with
an AC report that incorporates the results of all external COV and
directorate reporting and highlights the annual progress of the
directorate toward achieving NSF's qualitative goals.

At the close of the fiscal year, each directorate submits all
GPRA-related materials (COV and AC reports, directorate annual reports,
and responses to recommendations made by COVs) to agency senior
management, including the Office of the Director. Simultaneously,
quantitative data relevant to the management goals and investment
process goals are finalized by our staff and submitted to senior
management, including the Office of the Director.

**Agency GPRA Reporting**

The COV and AC reports prepared by external experts address a broad set
of issues ranging from staffing and quality of merit review to specifics
of a scientific project. The GPRA components of these reports are used
in assessing NSF's progress toward achieving its People, Ideas, and
Tools outcome goals (Goals III-1a, III-2 and III-3.) These reports also
contain discussions of investment process goals related to use of merit
review criteria by reviewers (Goal V-2) and Program Officers (Goal V-3).
Both are stated in the alternative form. Two quantitative goals (Goals
III-1b and III-1c) associated with the People outcome goal are evaluated
using relevant quantitative data.

The criterion for success for each of the People, Ideas, and Tools
outcome goals can be stated:

> *"NSF is successful when, in the aggregate, results reported in the
> period demonstrate significant achievement in one of more of the
> \[associated indicators\]."*

This criterion is utilized for judgements about both program and agency
success for GPRA P-I-T outcome goals. For program assessment, only
relevant goals and indicators are used. For agency assessment, all goals
and indicators are relevant and all are used in determining agency
success. The agency decision for NSF is based on analysis of the
successful/not successful judgements contained within the AC and COV
reports. Each successful rating requires supporting evidence or
retrospective examples supporting such a judgement.

NSF staff examine individual ratings (Successful or Not Successful)
included in COV and AC reports to ensure that ratings for the
qualititave outcome goals and indicators are justified. Each rating
assigned by the committees is evaluated by NSF staff utilizing
well-defined, internally developed criteria. In order to verify and
validate staff judgements regarding AC or COV ratings, an external firm,
PricewaterhouseCoopers LLP (PwC), was engaged to review the COV and AC
reports using the same criteria as NSF staff. PwC was asked to
independently assess ratings and justifications contained within the COV
and AC reports. NSF staff and PwC then met to compare and reconcile
their conclusions. In almost all cases there was consensus. The
differences were minor and had no impact on the final results.

Principal factors contributing to NSF's decision that the agency is
successful for our outcome goals related to People, Ideas, and Tools
include:

-   the consistently high ratings for each of the eight directorates and
    offices, as contained in Advisory Committee reports -- the external
    experts on the eight advisory committees judged all relevant outcome
    goals and indicators as successful.

-   the lack of significant numbers of "not successful" assessments (in
    both COV and AC reports).

-   the extensive number and quality of retrospective examples
    demonstrating significant achievement for the 11 indicators
    associated with the three outcome goals.

For agency GPRA reporting, we generally placed more emphasis on results
contained in the AC reports because these reports are more complete and
comprehensive compared to the COV reports, which cover only a third of
our programs. The AC reports took into account all of the material
provided by the COVs and Directorates/Office staff

**VIII. Verification and Validation (V&V)**

##### 

The Foundation has both qualitative and quantitative GPRA goals. Its
qualitative goals include the three broad strategic outcome goals
related to People, Ideas, and Tools and two investment process goals
related to implementation of merit review criteria. The outcome goals
are presented in a format that requires qualitative assessment of
achievement. These assessments are based largely on information included
in reports prepared by committees of independent, external experts (e.g.
Committees of Visitors and Advisory Committees) who assess the quality
of program results based on their collective experience-based norms. Our
quantitative goals focus on management activities, with the majority
presented in a format that enables quantitative assessment of progress
toward goal achievement. Assessment for these goals is based on data
collected with NSF's central data systems.

###### Quality of Reported Performance Information

NSF recognizes the ongoing need to improve data systems for collecting
performance information and data, especially that related to facilities.
We view the improvement of the quality of data and data systems as an
evolutionary process and intend to maintain it as a priority as budget
and time allow. Implementing GPRA has enabled NSF to gather information
in a structured way and to address issues in a more formal, focused
manner than in the past.

In their January 2002 report PricewaterhouseCoopers LLP (PwC) addressed
system aspects of NSF data quality for the Awards system, Enterprise
Information System, Financial Accounting System, FastLane, Integrated
Personnel System, and the Proposal, PI, and Reviewer System. PwC
"*reviewed NSF's information systems to ensure that adequate internal
controls are in place to produce reliable data. The techniques presented
are based on interviews with NSF managers and staff, rather than a full
application review. Pursuant to GAO's assessment guide, we relied on
previously conducted work and on departmental sources to determine
whether there were any known problems with the data sources or the data
itself that would cast doubt on the credibility of the information. One
external report that we referenced was the House Subcommittee on
Government Efficiency, Financial Management, and Intergovernmental
Relations' computer security report card, which it released in October
2001. The report card rated NSF with the highest grade (B+) of 24 major
federal agencies.*"

In FY 2002, NSF's data quality program has the following objectives:

-   Complete the evaluation of data elements with primary focus on data
    supporting GPRA goals;

-   Complete the population of validated data elements into the data
    dictionary for all GPRA data elements;

-   Enhance the functionality of the data dictionary for all NSF-wide
    information systems to ensure that meta-data describing the data is
    identified and thoroughly documented;

-   Continue to ascertain the causes of the data quality problems and
    develop systematic methods for correction; and

-   Develop and promulgate data quality policies and procedures
    NSF-wide.

A COV data project initiated in FY 2001 will substantially improve the
quality, consistency and availability of data, reports and charts that
are used by external NSF committees. These committees, in addition to
providing advice to the NSF organization, provide assessments used in
NSF's annual GPRA reporting. Currently, each NSF organization produces
its own reports and charts for each of its committees. As a result of
this new project, the reports will be generated centrally to reduce
costs and improve quality and consistency across NSF. The initial
planning for the project begins in FY 2001 with the majority of the
implementation to be completed in FY 2002. The project will be completed
in FY 2003.

###### Data V&V Activities

We used a process similar to the one used in FY 2000 to verify and
validate selected FY 2001 GPRA performance information. In FY 2000 and
FY 2001, we engaged an external third party, PricewaterhouseCoopers LLP,
to verify and validate selected GPRA performance results as well as the
process through which supporting data was compiled. PwC documented the
processes we follow to collect, process, maintain, and report selected
performance data. They identified relevant controls and commented on
their effectiveness. Based on GAO guidance, they provided an assessment
of the validity and verifiability of the data, policies, and procedures
we used to report results for the FY 2001 goals. For the outcome goals,
PwC confirmed the ratings and interpretations contained in the COV and
AC reports. PwC also provided high-level review of NSF's information
systems based on GAO standards for application controls. We expect to
use a similar process in FY 2002.

In their report (January 2002), PwC concluded "From our review, we
determined that NSF has reported on ten of the quantitative goals and
all five qualitative goals in a manner such that any errors, should they
exist, would not be significant enough to change the reader's
interpretation of the Foundation's success in meeting the supporting
performance goal. For these goals, NSF relies on sound business
processes, system and application controls, and manual checks of system
queries to report performance. We believe that these processes are valid
and verifiable. For the four goals related to facilities management, we
identified significant data limitations, which impaired our ability to
verify the processes. However, we believe that NSF's reported outcomes
are consistent with the data they collected."

For reporting on goal achievement, all of our outcomes are compiled for
programs and activities across the agency. To enable a uniform and
systematic organization of reporting information for the strategic
outcome goals, we have developed specially designed templates and
reporting guidelines for use by committees of external experts (COVs and
ACs). These templates and guidelines are reviewed and refined annually.
Options for rating NSF are limited to either successful or not
successful.

Types and Sources of Performance Data and Information

Most of the data that underlie achievement assessments for strategic
outcome goals originate outside the agency and are submitted to us
through the Project Reporting System, which includes annual and final
project reports for all awards. Through this system, performance
information/data (compiled by our staff) such as the following are
available to program staff, third party evaluators, and other external
committees:

-   Information on People -- student, teacher and faculty participants
    in NSF activities; demographics of participants; descriptions of
    student involvement; education and outreach activities under grants;
    demographics of science and engineering students and workforce;
    numbers and quality of educational models, products and practices
    used/developed; number and quality of teachers trained; and student
    outcomes including enrollments in mathematics and science courses,
    retention, achievement, and science and mathematics degrees
    received;

-   Information on Ideas -- published and disseminated results,
    including journal publications, books, software, audio or video
    products created; contributions within and across disciplines;
    organizations of participants and collaborators (including
    collaborations with industry); contributions to other disciplines,
    infrastructure, and beyond science and engineering; use beyond the
    research group of specific products, instruments, and equipment
    resulting from NSF awards; and role of NSF-sponsored activities in
    stimulating innovation and policy development; and

-   Information on Tools -- published and disseminated results; new
    tools and technologies, multidisciplinary databases; software,
    newly-developed instrumentation, and other inventions; data,
    samples, specimens, germ lines, and related products of awards
    placed in shared repositories; facilities construction and upgrade
    costs and schedules; and operating efficiency of shared-use
    facilities.

Most of the data supporting management goals can be found in NSF's
central systems. These central systems include the Enterprise
Information System (EIS); FastLane, with its Performance Reporting
System and its Facilities Reporting System; the Online Document System
(ODS); the Proposal, PI, and Reviewer System (PARS); the Awards System;
the Electronic Jacket; and the Financial Accounting System (FAS). These
systems are subject to regular checks for accuracy and reliability.

The Division of Human Resources Management (HRM/OIRM) maintains
information related to staff recruitment and staff training, under the
guidance of the Chief Information Officer. OEOP databases are also
available for reporting purposes.

The qualitative aspects associated with the goals on implementation of
both merit review criteria are addressed in reports of external
committees (COVs and ACs) and/or staff analyses.

###### Data / Information Limitations

For outcome goals, the collection of qualitative data during assessment
may be influenced by factors such as a lack of long-term
data/information to assess the impact of outcomes, the potential for
self-reporting bias, the unpredictable nature of discoveries, and the
timing of research and education activities. For the quantitative
management goals, the assessment may be influenced by factors such as
accuracy of data entry into central computer systems, lack of experience
in using new reporting systems or modules, or individual
non-responsiveness (e.g., self-reporting of diversity information;
workplace surveys).

Finally, external expert assessments (presented in COV and AC reports)
may lack sufficient justification for ratings or may provide incomplete
information. To address this issue NSF is continuing to modify its
reporting templates and improve guidance to committees and staff in
order to improve the completeness and consistency of the reports. This
will aid in compiling qualitative information. Additionally, we have
focused on clarifying language in goal and indicator statements.

# IX. Transition From FY 2000 to FY 2001

The NSF FY 2001 Performance Plan was based on our updated GPRA
*Strategic Plan FY 2001 -- 2006*, finalized in September 2000, and upon
newly developed Strategic Outcomes included therein. The chart below
clarifies the linkage between our new goals for FY 2001 and those
described in earlier NSF GPRA documents. The new Strategic Outcome Goal
areas of developing People, enabling Ideas, and providing Tools serve as
the linkage between NSF's mission and annual performance goals. The
goals in the FY 2001 Performance Plan took into account lessons learned
in FY 1999 and FY 2000, recommendations from the NSF Strategic Planning
Integration Group, and input from the research community, auditors,
Congressional groups, and stakeholders. Additional discussion of annual
performance goals and indicators pertaining to these Outcome areas may
be found in the NSF FY 2001 Performance Plan.

**[annual performance goals for nsf strategic outcomes]{.smallcaps}**

-   FY 2000 Outcome Goal: A diverse, globally oriented workforce of
    scientists and engineers.

```{=html}
<!-- -->
```
-   FY 2000 Outcome Goal: Improved achievement in mathematics and
    science skills needed by all Americans.

> These goals are incorporated under the FY 2001 Strategic Outcome:
> People-A diverse, internationally competitive and globally engaged
> workforce of scientists, engineers, and well-prepared citizens.

-   FY 2000 Outcome Goal: Discoveries at and across the frontier of
    science and engineering.

-   FY 2000 Outcome Goal: Connections between discoveries and their use
    in service to society.

> These goals are incorporated under the FY 2001 Strategic Outcome:
> Ideas - Discovery across the frontier of science and engineering,
> connected to learning, innovation and service to society.

-   FY 2000 Outcome Goal: Timely and relevant information on the
    national and international science and engineering enterprise.

> This goal addresses the concerns of Science Resources Studies (SRS)
> customers regarding the accuracy of and the ability to obtain needed
> information on science and engineering personnel and resources. The
> goal was adjusted to determine what data are needed to better reflect
> the 21^st^ century science and technology enterprise -- to develop,
> assess, and begin implementation of design options for recasting SRS
> science and engineering resources data collections. This goal is
> incorporated under the FY 2001 Strategic Outcome: Tools - Broadly
> accessible state-of-the-art and shared research and education tools.

######## [annual performance goals for nsf management]{.smallcaps}

-   FY 2000 Performance Goal: By the end of FY 2000, all staff will
    receive an orientation to FastLane, and at least 80% of program and
    program support staff will receive practice in using its key
    modules.

> NSF offers a comprehensive training program and strongly encourages
> all employees to keep current with technology improvements as well as
> government regulations. This goal was not continued in FY 2001.

-   FY 2000 Performance Goal: NSF will complete all activities needed to
    address the Year 2000 problem for its information systems according
    to plan, on schedule and within budget.

> OMB guidelines and milestones for assessment, renovation, validation
> and implementation were followed and achieved. External validation of
> NSF's systems compliance with Y2K guidance was accomplished. This goal
> is no longer relevant.

-   FY 2000 Performance Goal: During FY 2000, at least 85% of all
    project reports will be submitted through the new electronic Project
    Reporting System.

> Since the system was fully utilized the goal was not continued in FY
> 2001.

-   FY 2000 Performance Goal: By the end of FY 2000, NSF will have the
    technological capability to take competitive proposals submitted
    electronically through the entire proposal and award/declination
    process without generating paper within NSF.

This goal has been modified to focus on the review process. NSF will
conduct 10 pilot paperless projects that manage the competitive review
process in an electronic environment.

######## [changes in annual performance goals for nsf's investment process]{.smallcaps}

-   FY 2000 Performance Goal: At least 90% of NSF funds will be
    allocated to projects reviewed by appropriate peers external to NSF
    and selected through a merit-based competitive process.

> This performance goal was revised from that stated in FY 1999 and
> early 2000 in order to be consistent with the government-wide
> definition of merit-reviewed scientific research as specified by OMB
> in FY 2000:
>
> NSF exceeded the original goal of 90% for FY 1999 and FY 2000 by
> achieving results of 95% for both years. Nevertheless, NSF has
> calculated a new baseline, goals, and results based on OMB's revised
> merit review definition issued in FY 2000.

-   FY 2000 Performance Goal: NSF's performance in implementation of the
    new merit review criteria is successful when reviewers address the
    elements of both generic review criteria appropriate to the proposal
    at hand and when program officers take the information provided into
    account in their decisions on awards, as judged by external
    independent experts.

> This goal was separated into its component parts and directed toward
> reviewers and program officers to better measure the performance of
> each.

-   FY 2000 Performance Goal: Identify possible reasons for customer
    dissatisfaction with NSF's merit review system and with NSF's
    complaint system.

> The information is being utilized in staff training and in developing
> goals. This goal was not continued in FY 2001.

-   FY 2000 Performance Goal: Identify best practices and training
    necessary for NSF staff to conduct merit review and answer questions
    about the review criteria and process. Identify best practices and
    training necessary for NSF staff to answer questions from the
    community and to deal with complaints in a forthright manner.

Customer service continues to be of the highest priority for NSF. NSF
continues to address these concerns, particularly those involving the
merit review process and handling of customer complaints and will
concentrate on improving its Merit Review Process (see FY 2001
performance goals V-4 and V-5.) This goal was not continued in FY 2001.

-   FY 2000 Performance Goal: Improve NSF's overall American Customer
    Satisfaction Index (ACSI) compared to the FY 1999 index of 57 (on a
    scale of 0 to 100.)

> Customer service continues to be a high priority for NSF. The results
> of these surveys were used to identify issues of importance to
> respondents, which enables NSF to design meaningful goals. This goal
> was not continued in FY 2001.

-   FY 2000 Performance Goal: Develop a plan and system to request that
    Principal Investigators address the integration of research and
    education in their proposals, and develop a system to verify that
    PIs have done so.

> The goal is incorporated into the Implementation of Merit Review
> Criteria Goals for FY 2001 (see FY 2001 performance goals V-2 and
> V-3). Each program announcement, NSF's Guide to Programs and the Grant
> Proposal Guide explain the review criteria.
>
> The plan was developed and is being utilized. Issuance of Important
> Notice #125 reminded PIs of the importance of addressing this topic.
> This goal was not continued in FY 2001.

-   FY 2000 Performance Goal: Develop and implement a system/mechanism
    to request and track reviewer comments tied to merit review
    criterion #2, \"what are the broader impacts of the proposed
    activity?\".

> The system was developed and is being utilized. The goal above is
> incorporated into the Implementation of Merit Review Criteria Goals
> for FY 2001 (see FY 2001 performance goals V-2 and V-3). Each program
> announcement, NSF's Guide to Programs and the Grant Proposal Guide
> explain the review criteria (see Implementation of Merit Review
> Criteria.) This goal was not continued in FY 2001.

-   FY 2000 Performance Goal: In FY 2000, NSF will identify mechanisms
    to increase the number of women and underrepresented minorities in
    the proposal applicant pool, and will identify mechanisms to retain
    that pool.

> This goal is incorporated under the FY 2001 Strategic Outcome:
> People---A diverse, internationally competitive and globally engaged
> workforce of scientists, engineers, and well-prepared citizens, and
> also under the Implementation of Merit Review Criteria Goals (reviewer
> and Program Officer goals) for FY 2001.
>
> This goal was not continued as stated. In FY 2001, NSF focused its
> "broadening participation" performance area initially on
> diversification of the reviewer pool.

**X. Transition From FY 2001 to FY 2002**

The following goals, which were presented in the FY 2001 Performance
Plan, have been modified or removed from the FY 2002 Revised Final
Performance Plan. The significance and rationale for changes or
exclusion are discussed below.

**[Annual Performance Goals for NSF Strategic Outcomes]{.smallcaps}**

[FY 2001 Performance Goal III-1a:]{.underline} People Strategic Outcome
\-- Development of "a diverse, internationally competitive and
globally-engaged workforce of scientists, engineers, and well-prepared
citizens."

[FY 2001 Performance Indicators:]{.underline}

NSF is successful when, *in the aggregate*, results reported in the
period demonstrate significant achievement in one or more of the
following indicators:

-   Improved mathematics, science, and technology skills for U.S.
    students at the K-12 level, and for citizens of all ages, so that
    they can be competitive in a technological society.

-   A science and technology and instructional workforce that reflects
    America's diversity.

-   Globally engaged science and engineering professionals who are among
    the best in the world.

-   A public that is provided access to the benefits of science and
    engineering research and education.

> [FY 2002 Performance Indicators:]{.underline}
>
> NSF's performance is successful when, *in the aggregate*, results
> reported in the period demonstrate significant achievement in the
> majority (4 of 7) of the following indicators:

-   Development of well-prepared scientists, engineers or educators
    whose participation in NSF activities provides them with the
    capability to explore frontiers and challenges of the future;

-   Improved science and mathematics performance for U.S. K-12 students
    involved in NSF activities;

-   Professional development of the SMET instructional workforce
    involved in NSF activities;

-   Contributions to development of a diverse workforce through
    participation of underrepresented groups (women, underrepresented
    minorities, persons with disabilities) in NSF activities;

-   Participation of NSF-supported scientists and engineers in
    international studies, collaborations, or partnerships;

-   Enhancement of undergraduate curricular, laboratory or instructional
    infrastructure; and

```{=html}
<!-- -->
```
-   Awardee communication with the public in order to provide
    information about the process and benefits of NSF-supported science
    and engineering activities.

> [Explanation of change:]{.underline} The set of performance indicators
> related to the People Goal has been expanded and modified to
> appropriately reflect the breadth of NSF activities. The criterion for
> successful performance was raised from significant achievement in at
> least one indicator to successful achievement in a majority of
> indicators.

[FY 2001 Performance Goal III-1b:]{.underline} After three years of
support, over 80% of schools participating in systemic initiative
programs will (1) implement a standards-based curriculum in science and
mathematics; (2) further professional development of the instructional
workforce; and (3) improve student achievement on a selected battery of
tests.

[FY 2002 Performance Goal III-1b:]{.underline} After three years of NSF
support, over 80% of schools participating in systemic initiative
programs will (1) implement a standards-based curriculum in science and
mathematics with at least one-third of their teachers; (2) provide
professional development for at least one-third of their teachers; and
(3) improve student achievement on a selected battery of math and
science tests at one or more of three educational levels (elementary,
middle and high school).

[Explanation of change]{.underline}: The revised wording of the goal
clarifies the threshold for success.

[FY 2001 Performance Goal III-1c:]{.underline} Through systemic
initiatives and related teacher enhancement programs, NSF will provide
intensive professional development experiences for at least 65,000
pre-college teachers.

[FY 2002 Performance Goal:]{.underline} Not included.

[Explanation of change:]{.underline} This performance goal is not part
of our FY 2002 Performance Plan. For FY 2002 NSF has reapportioned a
substantial amount of the funds for the Systemic Initiatives to support
the new Presidential Math and Science Partnership (MSP) activity. No new
competitions or awards are anticipated under the Systemic programs. A
goal related to the MSP has been included in the FY 2003 GPRA
Performance Plan.

[FY 2001 Performance Goal III-2:]{.underline} Ideas Strategic Outcome
\-- Enabling discovery across the frontier of science and engineering,
connected to learning, innovation and service to society.

[FY 2001 Performance Indicators:]{.underline}

> NSF is successful when, *in the aggregate*, results reported in the
> period demonstrate significant achievement in one or more of the
> following indicators:

-   A robust and growing fundamental knowledge base that enhances
    progress in all science and engineering areas including the science
    of learning;

-   Discoveries that advance the frontiers of science, engineering, and
    technology;

-   Partnerships connecting discovery to innovation, learning, and
    societal advancement;

-   Research and education processes that are synergistic.

> [FY 2002 Performance Indicators:]{.underline}
>
> NSF's performance is successful when, *in the aggregate*, results
> reported in the period demonstrate significant achievement in the
> majority (4 of 6) of the following indicators:

-   Discoveries that advance the frontiers of science, engineering, or
    technology;

-   Discoveries that contribute to the fundamental knowledge base;

-   Leadership in fostering newly developing or emerging areas;

-   Connections between discoveries and their use in service to society;

-   Connections between discovery and learning or innovation; and

-   Partnerships that enable the flow of ideas among the academic,
    public or private sectors.

[Explanation of change:]{.underline} The set of performance indicators
related to the Ideas Goal has been revised to better enable qualitative
assessment and reflect the breadth of NSF activities. The criterion for
successful performance was raised from significant achievement in at
least one indicator to successful achievement in a majority of
indicators.

[FY 2001 Performance Goal III-3:]{.underline} Tools Strategic Outcome
\-- Providing "broadly accessible, state-of-the-art and shared research
and education tools."

[FY 2001 Performance Indicators:]{.underline}

> NSF's performance is successful when, *in the aggregate*, results
> reported in the period demonstrate significant achievement for one or
> more of the following indicators:

-   Shared use platforms, facilities, instruments, and databases that
    enable discovery and enhance the productivity and effectiveness of
    the science and engineering workforce;

-   Networking and connectivity that take full advantage of the Internet
    and make SMET information available to all citizens;

-   Information and policy analyses that contribute to the effective use
    of science and engineering resources.

> [FY 2002 Performance Indicators:]{.underline}
>
> NSF's performance is successful when, *in the aggregate*, results
> reported in the period demonstrate significant achievement in the
> majority (4 of 6) of the following indicators:

-   Provision of facilities, databases or other infrastructure that
    enable discoveries or enhance productivity by NSF research or
    education communities;

-   Provision of broadly accessible facilities, databases or other
    infrastructure that are widely shared by NSF research or education
    communities;

-   Partnerships, e.g., with other federal agencies, national
    laboratories or other nations, to support and enable development of
    large facilities and infrastructure projects;

-   Use of the Internet to make SMET information available to the NSF
    research or education communities;

-   Development, management, or utilization of very large data sets and
    information-bases; and

-   Development of information and policy analyses that contribute to
    the effective use of science and engineering resources.

[Explanation of change:]{.underline} The set of performance indicators
related to the Tools Goal has been expanded and modified to
appropriately reflect the breadth of NSF activities. The criterion for
successful performance was raised from significant achievement in at
least one indicator to successful achievement in a majority of
indicators.

**[Annual Performance Goals for NSF Management]{.smallcaps}**

[FY 2001 Performance Goal IV-1:]{.underline} Ninety-five percent of full
proposals will be received electronically through FastLane.

[FY 2002 Performance Goal:]{.underline} Not included.

[Explanation of Change:]{.underline} This goal will not be continued in
FY 2002. Electronic submission of proposals via FastLane is now standard
operating procedure at NSF.

[FY 2001 Performance Goal IV-3:]{.underline} By the end of FY 2001, NSF
will increase usage of a broad-range of video-conferencing/long distance
communications technology by 100% over the FY 1999 level.

[FY 2002 Performance Goal:]{.underline} Not included.

[Explanation of change:]{.underline} By the end of FY 2001,
videoconferencing was viewed as a functioning, rather than experimental,
technology. Because videoconferencing is an established practice for us,
it will not be continued as a goal in the future. We will, however,
continue to emphasize this technology for current and emerging business
applications.

[FY 2001 Performance Goal V-7:]{.underline} NSF will award 30% of its
research grants to new investigators.

[FY 2002 Performance Goal:]{.underline} Not included.

[Explanation of change:]{.underline} This openness goal is not included
in the FY 2002 performance plan because we wish to fully consider
whether this particular goal provides a good measure of openness in the
system.

[FY 2001 Performance Goal V-8:]{.underline} NSF will begin to request
voluntary demographic data electronically from all reviewers to
determine participation levels of underrepresented groups in the NSF
reviewer pool.

[FY 2002 Performance Goal IV-6:]{.underline} Establish a baseline for
participation of members of underrepresented groups in NSF proposal
review activities.

[Explanation of change:]{.underline} The FY 2002 goal is a continuation
of the FY 2001 goal. To enable the development of robust baselines, we
will continue to gather the appropriate voluntary data from reviewers.

[FY 2001 Performance Goal V-9b:]{.underline} Ninety percent of
facilities will meet all major annual schedule milestones by the end of
the reporting period.

[FY 2002 Performance Goal IV-9b:]{.underline} Ninety percent of
facilities will meet all major annual schedule milestones.

[Explanation of change:]{.underline} This goal was adjusted based on
actual performance reporting experience in FY 2001 and feedback from
facilities managers, NSF program officers and PricewaterhouseCoopers.

# XI. other features

# Information on use of non-federal parties

This GPRA performance report was written and prepared solely by NSF
staff.

Non-Federal external sources of information we used in preparing this
report include:

-   Reports from awardees demonstrating results.

-   Reports prepared by evaluators -- Committees of Visitors (COV) and
    Advisory Committees -- in assessing our programs for progress in
    achieving Outcome Goals.

-   Reports prepared by a consulting firm to assess the procedures we
    use to collect, process, maintain, and report performance goals and
    measures.

-   Reports from facilities managers on construction/upgrade costs and
    schedules and on operational reliability.

Specific examples:

Highlights or sources of examples shown as results may be provided by
Principal Investigators who received support from NSF.

We use external committees to assess the progress of our programs toward
qualitative goal achievement. External evaluators provide us with
reports of programs, and provide feedback to us on a report template we
prepare. Examples are COV and Advisory Committee reports that provide an
independent external assessment of NSF's performance.

We engaged an independent third-party (PricewaterhouseCoopers LLP) to
conduct a review of data and information used in performance reporting.
PwC reviewed NSF's performance data and information pertaining to
selected outcome goals, management goals, and investment process goals.
This additional independent review helped to eliminate potential
reporting bias that can develop in self-assessments. It also provides
assurance of the credibility of performance reporting information and
results.

# Budget Information: 

# 

# NSF obligated \$4.5 billion in FY 2001. Administrative support for the Foundation was approximately 5% of the total NSF budget. 

## **Classified Appendices not Available to the Public** 

None to report.

## **Analysis of Tax Expenditures** 

None to report.

## **Waivers of Administrative Requirements**

None to report.

**MANAGEMENT CHALLENGES**

![](media/image2.png)

# XII. - Management Challenges And Reforms

Federal agency management challenges are discussed in the President's
Management Agenda (PMA). For NSF, they are also identified internally by
NSF staff and by OMB, GAO, and the NSF Office of the Inspector General
(OIG).

The **President's Management Agenda** lists five government-wide
initiatives. The first four of these initiatives (Strategic Management
of Human Capital, Competitive Sourcing, Improved Financial Performance,
and Expanded Electronic Government) are discussed in NSF's FY 2003
Performance Plan. NSF's implementation of the remaining initiative,
Budget and Performance Integration, is currently under discussion within
NSF and between NSF and OMB. We have contracted with
PricewaterhouseCoopers (PwC) to provide formal recommendations to
improve our approach on integrating the budget, performance and cost of
performance, within the intent of the Government Performance and Results
Act (GPRA), Statement of Federal Financial Accounting Standard (SFFAS)
4, and Managerial Cost Accounting Concepts and Standards for the Federal
Government.

The **OIG** issues addressed below are those included in a November 2000
statement by the Inspector General on NSF's management and performance
challenges. This statement was released on January 4, 2001 and is
contained in the NSF FY 2000 Accountability Report. In many instances,
the management and performance challenges contained in the PMA, OMB,
GAO, and the OIG documents are very similar.

For FY 2001, the NSF OIG identified 10 areas for NSF to monitor:

**1. FASTLANE**

**NSF OIG COMMENT:** "In the FY 2001 budget, OMB identifies streamlining
and simplifying grants management as one of the most important
management challenges facing the federal government. At NSF, the
development and implementation of FastLane, which began in 1994, has
moved the agency closer to the goal of establishing a widely accessible
paperless proposal and award process. In many respects the
implementation has been successful and NSF serves as a leader within
government in electronic innovation. The increase in the use of FastLane
by those seeking grants each year has been encouraging and has
undoubtedly helped contribute to the increase in productivity NSF has
achieved in recent years. However, problems remain, as reflected by the
inability of the help desk to cope with the high volume of incoming
questions and problems. Because FastLane serves as the primary interface
between NSF and its grantees and is critical to many of NSF\'s
administrative plans and goals, we believe that management must continue
to monitor its progress and assure that the system is as user-friendly
and reliable as possible."

**FROM THE PRESIDENT'S MANAGEMENT AGENDA:** An expanded
electronic-government is one of the government-wide initiatives
presented in *the President's Management Agenda for 2002*. That document
states that "the administration's goal is to champion citizen-centered
electronic government."

**FOCUSED NSF ACTIVITIES IN THIS AREA:** NSF's FastLane system uses the
Internet to allow its customers, the grantee community, to exchange
information with NSF. It permits users to prepare and submit proposals,
proposal reviews and project reports, determine the status of funding
actions, submit post-award requests, interactively participate in panel
evaluations of proposals, initiate cash requests, view reviews and award
letters, and perform other basic interactions. Over 200,000 scientists,
engineers, educators, technology experts and academic administrators use
FastLane, with over 99 percent of proposals submitted electronically in
FY 2001. In addition, the public can access titles, authors, funding
amounts and abstracts of NSF awards.

A past challenge for FastLane was to make the system more user-friendly
and reliable. In January 2001, FastLane implemented a conversion process
to allow Word, WordPerfect, TeX and other documents to be uploaded and
converted in real-time to PDF files. This significant, and technically
challenging, change to the system was greeted by FastLane's user
community with more positive responses than any other user-oriented
change in the system. In March 2001, FastLane included a detailed
manual, available through the web, for electronic preparation and
submission of proposals.

The implementation in FY 2000 of a toll-free phone number to the
FastLane Help Desk made it easier for NSF's user-community to obtain
assistance -- while at the same time increasing significantly the call
volume. In June and July 2001, the FastLane Help Desk was able to handle
peak loads without, for the first time, supplementing the Help Desk with
staff from program offices. This improvement is attributed to better
Help Desk practices, increase in the operating hours of the Fast Lane
Help Desk and the number of trained Help Desk staff, improved on-line
documentation, implementation of a word processor conversion tool, work
on the web interface to make the system more user-friendly, and
spreading out proposal deadlines.

# 2. GPRA DATA QUALITY

**NSF OIG COMMENT:** "GPRA seeks to improve the effectiveness,
efficiency and accountability of federal programs by requiring agencies
to set goals for performance and report on annual performance compared
with the goals. In addition, it requires agencies to \"describe the
means to be used to verify and validate measured values\" of performance
in their performance plans. A recent GAO study, [Managing for Results:
Opportunities for Continued Improvements in Agencies Performance Plans
(GAO/GGD/AIMD-99-215)]{.underline}, said that a key weakness of NSF\'s
FY 2000 Performance Plan is that it \"provides limited confidence in the
validation and verification of data.\" Meanwhile, the agency has
contracted with several firms to assist in validating the performance
data it reports. However, if uncertainty persists about data validity,
decision-makers will be reluctant to rely on the information, and its
usefulness will be diminished."

**FOCUSED NSF ACTIVITIES IN THIS AREA:** For FY 2000 and FY 2001 GPRA
reporting, NSF engaged an external party, PricewaterhouseCoopers LLP
(PwC), to provide an independent verification and validation (V&V) of
selected GPRA goals. The V&V focused on reliability of data, on
processes to collect, process, maintain, and report the data, and on
program reports prepared by external experts. PwC mapped NSF procedures
against GAO guidance for polices and procedures that underlie GPRA
performance reporting.

For FY 2000 PwC reviewed a limited set of our goals and concluded that
NSF "relies on sound business processes, system and application
controls, and manual checks of system queries to confirm the accuracy of
reported data. We believe that these processes are valid and
verifiable."

For FY 2001 PwC reviewed the goals it had reviewed in FY 2000 and
additional goals (see Appendix IV.). In their report they state: "We
commend NSF for undertaking this second year effort to confirm the
reliability of its data and the processes to collect, process, maintain,
and report this data. From our FY 2001 review, we conclude that NSF has
made a concerted effort to ensure that it reports accurately to the
federal government and has effective systems, policies, and procedures
to ensure data quality. We have noted some areas for improvement,
particularly in the area of data collection for the goals related to
facilities management. However, overall NSF relies on sound business
practices, system and application controls, and manual checks of system
queries to report performance. Further, our efforts to re-calculate the
Foundation's results based on these systems, processes and data were
successful."

FY 2001 progress on NSF's data quality program included completion of an
extensive analysis of the existing data dictionary designed in Access
and recommendations for improving the functionality of the data
dictionary.

**3. MERIT REVIEW**

**NSF OIG COMMENT:** "Because of its importance to the success of NSF\'s
mission, the merit review system remains on our list of management
challenges. Operating a viable, credible, efficient merit review system
is one of four critical factors identified by the agency in managing for
excellence. NSF must continue to ensure that: reviewers correctly apply
NSF\'s review criteria; due consideration is given to ideas, individuals
and institutions that have not received past support; and that the
process is fairly and effectively administered.

In particular, we believe that the agency has opportunities to improve
in two areas. We believe that NSF should enhance its effort to expand
the peer review community with regard to race, gender, geography, and
type of school, providing the chance to participate to all who are
qualified. In our view, the selection of peer reviewers is an
opportunity for NSF to reach out to underrepresented segments of the
scientific community and educate them about the process of obtaining
federal support for their research. This will help to generate proposals
from those who may have worthy research ideas but are unfamiliar with,
or intimidated by, the system. Secondly, we are concerned about the
agency\'s ability to maintain the confidentiality of proposals in an
electronic environment. As more proposal review functions migrate to the
internet, NSF must be able to ensure that the intellectual property
contained in a proposal is secure."

**FOCUSED NSF ACTIVITIES IN THIS AREA:** NSF considers its merit review
process the keystone for award selection. The agency evaluates proposals
using two criteria -- intellectual merit of the proposed activity and
broader impacts of the proposed activity. NSF focuses its management
activities on a wide variety of issues related to merit review --
including use of both merit review criteria by reviewers and program
officers, broadening participation, and enhancing customer service.

In FY 2001 NSF established an internal task force to examine strategies
to improve both proposer and reviewer attention to the broader impacts
criterion. The group assessed the characteristics and quality of
reviewer responses to this criterion and found that, based on a sample
of FY 2001 reviews, approximately 69% of reviews provided evaluative
comments in response to the broader impacts criterion. The group also
developed examples of broader impacts that may be useful to proposers in
developing proposals and reviewers in evaluating proposals. In FY 2002,
NSF will continue to develop and apply recommendations that focus on
strategies that stress the importance of using both criteria. It will
also make available examples of broader impacts.

With respect to increasing the diversity of the peer review pool, this
was addressed as part of our Investment Process Goal V-8. FY 2001 marks
the first time we have focused attention on reviewer pool data. To
establish the baselines for this goal, we have begun to gather the
appropriate voluntary data from the reviewers, which will be added to
the reviewer pool database. A baseline for FY 2002 will be derived from
this data.

We will address the challenge of proposal security below in our response
to the challenge of data security.

**4. COST SHARING**

**NSF OIG COMMENT:** "In accordance with Congressional requirements, NSF
requires that each grantee share in the cost of NSF research projects
resulting from unsolicited proposals. In addition to this statutory
requirement, NSF can require additional cost sharing when it believes
there is tangible benefit to the award recipient, such as infrastructure
development or the potential for program income. When cost sharing is
provided for in the approved award budget, it is presumed that the funds
are necessary to accomplish the objectives of the award. The commitment
becomes a condition of the award and subject to audit to the same extent
as the costs borne by NSF. Therefore, if promised cost sharing is not
realized, then either the awardee has not met its programmatic
objectives, or the project costs less than originally projected. In
either case, NSF should have at least a portion of its funds returned to
it.

We have been finding significant problems with awardees who are failing
to meet their cost sharing requirements. In the past semi-annual period,
we found several awardees with significant problems in this regard,
discussed in more detail in our September 2000 Semiannual Report. We are
continuing to focus our efforts in this area and are currently
conducting a broad review of cost sharing at numerous institutions.
Because of the importance of these research efforts to the scientific
and engineering community, and the detrimental impact a shortfall can
have on a project, we consider improvements in administering cost
sharing to be among the most important priorities for NSF management."

**FOCUSED NSF ACTIVITIES IN THIS AREA:** In June 1999 an "Important
Notice" was sent to Presidents of Universities and Colleges and Heads of
National Science Foundation Grantee Organizations. This notice
transmitted the "National Science Foundation Policy Statement on Cost
Sharing" as approved by the National Science Board. In addition to
providing a definition of cost sharing, the policy states that: (1)
NSF-required cost sharing is considered an eligibility rather than
review criterion; (2) NSF cost sharing requirements beyond the statutory
requirement (1%) will be clearly stated in the program announcement,
solicitation or other mechanism which generates proposals; (3) for
unsolicited research and education projects, only statutory cost sharing
will be required; and, (4) any negotiation regarding cost sharing will
occur within NSF-stated parameters. NSF has a long-standing policy
requiring cost- sharing certification when cost sharing exceeds
\$500,000.

This "Important Notice" was also distributed to NSF staff. During the
past year, NSF has held several training sessions on cost-sharing for
NSF staff and has also conducted sessions on cost-sharing for NSF
customers at regional conferences, seminars and workshops.

The Foundation recently conducted an analysis of grantee audits that
contain findings related to cost-sharing. It showed that while some
grantees have often provided cost-sharing, they may not have had
financial and accounting systems able to document their activities. NSF
is now conducting more pre-award reviews of grantee financial and
accounting systems to assess their capability to report on cost-sharing.
Post-award reviews are also conducted to assure compliance with agreed
upon cost-sharing requirements. In this analysis, over half of the audit
activity (both in number of audit reports and dollar amount of findings
cited) reported by the NSF OIG in its Semiannual Reports to Congress
since 1997, when the NSF OIG first reported separate statistics on
cost-sharing, was for grantee organizations that are "non-traditional"
(e.g., public school systems). NSF is currently developing an
appropriate strategy for reviewing cost-sharing by these types of
grantee organizations and providing outreach and instruction as
necessary.

# 5. AWARD ADMINISTRATION

**NSF OIG COMMENT:** "NSF\'s mission is to fund research and education
in science and engineering by issuing different types of awards
(primarily grants, contracts, and cooperative agreements) thereby
strengthening U.S. science and engineering. Assessing scientific
progress and ensuring effective financial/administrative management are
critical elements in managing NSF\'s grant programs. Program officers in
each of NSF\'s seven science Directorates are responsible for monitoring
the scientific progress of NSF\'s grants while the Division of Grants
and Agreements (DGA) and the Division of Contracts, Policy, and
Oversight (CPO) oversee grantees\' financial management of NSF awards.

At any one point in time NSF is administering as many as 30,000 ongoing
awards. NSF relies on a total staff of 1,150 employees to carry out this
oversight responsibility. This is in addition to their responsibility of
soliciting and awarding approximately 10,000 grants and cooperative
agreements annually amounting to over \$3.5 billion. Given this sizeable
workload, NSF is challenged to adequately monitor its awards for
scientific accomplishments and compliance with the award agreement and
federal regulations. For the most part, NSF relies on interim reports
from grantees to monitor progress, but is unable to test the reliability
of these reports. NSF also needs to establish a more coordinated
oversight effort between its program officers and its grant and contract
officers to ensure better sharing of information and more effective
action to address compliance issues."

**FOCUSED NSF ACTIVITIES IN THIS AREA:** NSF program portfolios have
become more diverse and more complex; however, there has not been a
concomitant increase in the staffing complement to provide additional
program and administrative oversight functions. The Draft NSF
Administrative and Management Strategic Plan currently under development
presents a framework for Award Management and Oversight that focuses on
a collaborative, multi-functional award management and oversight process
that is informed by risk management strategies and verifies that
projects are in compliance.

On-site post-award monitoring is one of the most effective methods of
ensuring awardee compliance with award agreements and federal
regulations. However, it is least often employed because of staff
resource constraints, including the availability of travel funds. In
order to leverage NSF administrative oversight capability, NSF will
develop and initiate in FY 2002 a risk assessment/risk management plan
for awards. A GPRA management goal covering these activities has been
added to the FY 2002 Revised Final Performance Plan for NSF.

**6. MANAGEMENT OF LARGE INFRASTRUCTURE PROJECTS**

**NSF OIG COMMENT:** "NSF is increasing its investments in large
infrastructure projects such as astronomy centers, research equipment,
supercomputing databases, and earthquake simulators. The agency spends
approximately \$1 billion a year on these research facilities and
equipment projects, with each of these projects costing several hundred
million dollars. Projects of this scale and complexity are becoming more
common for NSF, which historically has administered awards averaging
less than \$100,000 each. Successful management of these projects and
programs requires a more disciplined project management approach.
Management of these projects is particularly challenging for NSF because
of its limited number of staff. Although NSF recently issued guidelines
for managing these larger projects, the guidelines are interim and have
not been fully tested for adequacy."

**FROM OMB:** OMB has noted that NSF has several multi-year, large
facility projects awaiting approval for funding. Although the agency has
done well in keeping past projects on schedule and within budget, OMB
believes that NSF's capability to manage proposed projects needs to be
enhanced given the magnitude and costs of future projects. NSF was asked
to develop and submit a plan to OMB that documents its costing,
approval, and oversight of major facility projects.

**FOCUSED NSF ACTIVITIES IN THIS AREA:** In order to mitigate the risks
attendant to large facilities management, and to comply with the
President's mandate in *A Blueprint for New Beginnings: A Responsible
Budget for America's Priorities* (February 2001), NSF developed a Large
Facility Projects Management & Oversight Plan. The plan was submitted to
OMB in September 2001.

This new facilities plan has four major foci:

-   Enhance organizational and staff capabilities and improve
    coordination, collaboration, and shared learning among NSF staff and
    external partners;

-   Implement comprehensive guidelines and procedures for all aspects of
    facilities planning, management and oversight;

-   Improve the process for reviewing and approving Large Facility
    Projects; and

```{=html}
<!-- -->
```
-   Practice coordinated and proactive oversight of all facility
    projects.

Further development and implementation of the plan is continuing.

In FY 2002 the agency is also initiating development of a risk
assessment protocol focused towards on-site monitoring activities.

**7. MANAGEMENT OF U.S. ANTARCTIC PROGRAM**

**NSF OIG COMMENT:** "NSF plays a leadership role among federal agencies
involved in supporting research and logistics in the Antarctic through
its Office of Polar Programs (OPP). Charged with managing all U.S.
activities in the Antarctic as a single program, OPP not only funds
research, but also is responsible for operating the infrastructure and
logistics necessary to conduct scientific experiments in the harsh polar
environment. In this role, it faces a number of unique challenges such
as transporting and housing scientists and support staff, assuring their
safety and health, protecting the near pristine polar surroundings,
ensuring U.S. compliance with the international Antarctic Treaty, and
promoting the national interest in maintaining an active and influential
presence in Antarctica.

While OPP operates like other NSF directorates in making awards for
polar research, its responsibilities do not end there. In providing
science, operations, and logistics support to the research projects it
funds, it is significantly different than other NSF units. OPP staff
must not only know the science, but must also be able to manage
contractors engaged in delivering a broad range of services to the
American scientific community located in a difficult and dangerous
environment. Our audit work has focused on reviewing these activities
because of their many inherent risks. From our perspective, NSF\'s polar
programs involve not only a large expenditure of money, but also the
safety of scientists and workers, environmental concerns, and the
prestige of the U.S. government. The successful operation of the United
States Antarctic Program requires certain management and administrative
skills that are responsive to the special needs of Antarctic scientific
research."

**FOCUSED NSF ACTIVITIES IN THIS AREA:** Our staff provides special
expertise in:

-   Coordinating Department of Defense, NASA, USGS and DOE activities;

```{=html}
<!-- -->
```
-   Overseeing environmental, health, safety, and medical activities;

-   Overseeing construction and maintenance of all infrastructure at
    three U.S. stations in Antarctica (roads, fire stations, clinics,
    power stations, heating, communications, ground stations, air
    traffic control, ground vehicles, food services, sewage treatment,
    water supplies, etc.);

-   Coordinating support of scientists in Antarctica, construction of
    specialized science instrumentation, etc.;

-   Budgeting for the above activities; and

-   Selecting science projects for deployment on the basis of merit
    review and ability to meet logistics requirements.

**8. WORK FORCE PLANNING AND TRAINING**

**NSF OIG COMMENT:** "Although NSF has had healthy increases in its
program responsibilities and budgets in recent years, salaries and
expenses have remained relatively flat. NSF received an increase of 13.6
percent in its FY 2001 budget; however an increase of only 6 percent was
obtained for salaries and expenses. While we commend the agency for
successfully controlling its administrative overhead, the small
increases allocated for administration and management over the past few
years raise questions about whether NSF can successfully manage future
growth without adding more staff. Concerns about the adequacy of
staffing come at a time when the government as a whole is facing
succession planning and recruiting problems. In addition, NSF\'s
reliance on the Intergovernmental Personnel Act (IPA) personnel, who
serve on a term basis, poses a challenge to the agency to make certain
that personnel are adequately trained to administer grants. We are
planning audit work in this area to ensure that the agency has a
reasonable strategy for managing its human capital."

**THE PRESIDENT'S MANAGEMENT AGENDA** (2002) includes strategic
management of human capital as a government-wide initiative.

**GAO** *(GAO-01-236, April 2001)* has identified shortcomings of many
agencies involving key elements of modern strategic human capital
management, including (1) strategic planning and organizational
alignment; (2) leadership continuity and succession planning; and (3)
acquiring and developing staff whose size, skills, and deployment meet
agency needs.

**FOCUSED NSF ACTIVITIES IN THIS AREA:** NSF's flexible and motivated
workforce currently includes approximately 600 permanent and visiting
scientists and engineers (about 65% of whom are permanent government
employees), 450 administrative personnel (who provide business
operations support), and approximately 350 program support personnel.

NSF has a steadfast commitment to empower a workforce of teams and
individuals who are continuously expanding their capabilities to shape
the agency's future. To sustain its high-performing workforce, NSF is
exploring ways to recruit and retain excellent employees. New
initiatives include an updated telecommuting program, strategic
recruiting techniques that also seek to increase representation of
underrepresented groups in the NSF science and engineering workforce, a
renewed focus on continuous learning and an increased emphasis on
leadership and succession planning.

NSF's draft Administration and Management Strategic Plan (submitted to
OMB in March 2002) will examine organizational alignment and the
workforce size, skill mix, and deployment necessary to ensure mission
accomplishment.

**9. FOSTERING A DIVERSE SCIENTIFIC WORKFORCE**

**NSF OIG COMMENT:** "NSF is committed to increasing the diversity of
the nation\'s science and engineering workforce by embedding diversity
concerns in all of its programs. In its strategic plan, NSF says it aims
at new strategies for improving diversity and broadening participation
in NSF-funded activities. NSF\'s most recent performance plan promises
that the agency will begin implementing new strategies to increase
diversity. NSF executives and managers frequently stress the importance
of diversity in presentations to internal and external audiences.
Because diversity programs are difficult to implement in a society
challenged by economic, legal, and cultural constraints, NSF faces
numerous challenges and should clearly define its diversity strategies
and develop concrete steps (beyond giving general encouragement to its
program managers) for attaining its goals in this area."

**FOCUSED NSF ACTIVITIES IN THIS AREA:** NSF recognizes that a diverse
workforce -- one that includes members of underrepresented groups and
reflects institutional and geographic differences -- broadens the agency
outlook and talent base and enables it to better serve its research and
education communities and ultimately all citizens.

The FY 2003 NSF Performance Plan includes two goals related to the
agency's science and engineering (S&E) staff. This S&E group includes
program officers, division directors, the majority of staff assigned
under the provisions of the Intergovernmental Personnel Act, limited
term Visiting Scientists appointments, and others in management and
scientific positions. In aggregate, this group is the one most
intensively involved with the agency's external customers, the award
community. It is also the group involved within the Foundation in
development of new programs, in merit review and review analysis, and in
making recommendations to fund or decline proposals. It is thus
particularly important that these staff be diverse, with an ability "to
identify best practices that are appropriate to a diverse \[research and
education\] community" (Committee on Equal Opportunities in Science and
Engineering, 2000[^36]).

**10. SECURITY AND CONTROLS**

**NSF OIG COMMENT:** "Data Security: Electronic information and
automated systems are essential to NSF\'s operations. Next year NSF will
depend on its automated computer systems to manage over \$4 billion in
funds, receive and process over 35,000 grant proposals, handle over \$3
billion in cash transactions to NSF awardees, generate its agency wide
financial statements, and support a government wide website for federal
financial management initiatives and activities. Therefore, it is
imperative that NSF\'s systems are developed and operated with
appropriate security controls to reduce the ever increasing risk of
unauthorized access. NSF must be able to protect the availability,
integrity, and confidentiality of its computer based information.
Improvement is most needed in the areas of access controls and change
controls. Access controls limit or detect inappropriate access to
computer resources, while change controls prevent unauthorized
modifications to programs from being implemented. The audit of NSF\'s
financial statements has identified several internal control weaknesses
related to security of NSF\'s automated systems, although none were
material or rose to the level of a reportable condition."

**GAO** (01-758) noted that recent audits continue to show that federal
computer systems are riddled with weaknesses that make them highly
vulnerable to computer-based attacks and place a broad range of critical
operations and assets at risk of fraud, misuse, and disruption.

**FOCUSED NSF ACTIVITIES IN THIS AREA:** The NSF Information Technology
Security (ITS) Program is focused on assuring that NSF infrastructure
and critical assets are appropriately protected while maintaining an
open and collaborative environment for science and engineering research
and education. NSF's approach is based on a fundamental philosophy of
risk management where ITS risks are assessed, understood, and mitigated
appropriately.

An agency-wide ITS program had been implemented that encompasses all
aspects of information security, including policy and procedures, risk
assessments and security plans, managed intrusion detection services,
vulnerability assessments, and technical and management security
controls. The NSF Chief Information Officer provides overall leadership
for the ITS Program, and assures that policies, procedures, and
activities are coordinated with NSF program management and research and
education initiatives.

In FY 2001, NSF placed significant priority on ITS and initiatives to
assure adequate protection of resources. In December 2000, NSF appointed
an ADP Security Officer to coordinate ITS program plans and initiatives
with the NSF Chief Information Officer. The majority of NSF's
significant assets are managed within the Division of Information
Systems in the Office of Information and Resource Management. This
organization is responsible for managing the NSF Computer Center and
providing telecommunications, e-mail, and agency-wide applications and
services.

**APPENDICES**

![](media/image3.png)

**APPENDIX I. - TABLE OF EXTERNAL EVALUATIONS**

The Table below provides information on program assessments and
evaluations other than Committee of Visitor and Advisory Committee
assessments - with one exception -- the CAREER program. The CAREER
program is an agency-wide activity, and the assessment was contracted to
an external private vendor.

The table lists other types of evaluations, not used in GPRA performance
assessment, that were completed in FY 2001. These reports, studies, and
evaluations are frequently used in setting new priorities in a field or
in documenting progress in a particular area. The reader is encouraged
to review the reports for additional information on findings and
recommendations that are beyond the scope of this report.

Reports (other than COV and AC reports) produced by NSF are available
online at
[http://www.nsf.gov/pubs/start.htm](http://www.nsf.gov/pubs/start.htm )
using the NSF's online document system and the publication number
indicated.

Information on obtaining reports produced by the National Research
Council or National Academy of Sciences can be found online by searching
www.nap.edu or from the National Academy Press, 2101 Constitution
Avenue, N.W., Lockbox 285, Washington, D.C. 20055 (1.800.642.6242).

+-------------+--------------------------------------------------------+
| ###  {#se   |                                                        |
| ction-29 .B |                                                        |
| ox-Heading} |                                                        |
|             |                                                        |
| #           |                                                        |
| ## **Evalua |                                                        |
| tions Compl |                                                        |
| eted in FY  |                                                        |
| 2001** {#ev |                                                        |
| aluations-c |                                                        |
| ompleted-in |                                                        |
| -fy-2001 .B |                                                        |
| ox-Heading} |                                                        |
|             |                                                        |
| ###  {#se   |                                                        |
| ction-30 .B |                                                        |
| ox-Heading} |                                                        |
+-------------+--------------------------------------------------------+
| **BIO**     |                                                        |
+-------------+--------------------------------------------------------+
| *           | **Findings:** Representatives from both the Evolution  |
| **Evolution | of Developmental Mechanisms (EvoDevo) and Tree of Life |
| of          | (ToL) communities agreed that research progress in     |
| Development | these areas was constrained by the same major needs    |
| and Tree of | and that the elimination of these constraints would    |
| Life        | lead to remarkably fruitful and exciting interactions  |
| Workshop    | between evolutionary developmental biology and         |
| Report***   | evolutionary systematics. Communities long separated   |
|             | technically and conceptually have now converged on a   |
|             | common set of evolutionary questions. The participants |
|             | felt that after decades of separation they must        |
|             | exploit the powerful synergism presented by this       |
|             | convergence.                                           |
|             |                                                        |
|             | **Recommendations:** 1) Hold a competition to create   |
|             | about 100 arrayed bacterial artificial chromosome      |
|             | (BAC) and cDNA libraries of representative organisms   |
|             | on the Tree of Life ("First 100").                     |
|             |                                                        |
|             | 2\) Provide grants for functional analysis tool        |
|             | development production of BAC or sequenced libraries   |
|             | of additional organisms that address fundamental       |
|             | evolution of development questions.                    |
|             |                                                        |
|             | 3\) Provide support for sequencing about 24 major      |
|             | developmental gene families in the First 100           |
|             | organisms.                                             |
|             |                                                        |
|             | 4\) Encourage collaboration among phylogeneticists and |
|             | evolution of development researchers to develop a      |
|             | robust informatics infrastructure.                     |
|             |                                                        |
|             | **Availability:** NSF (                                |
|             | [http://www.nsf.gov/pubs/2001/bio012/)]{.underline}    |
+-------------+--------------------------------------------------------+
| ***Research | **Findings:** A major new coordinated research effort  |
| Needs in    | is necessary. The integration of expertise and of data |
| Phyloi      | from a variety of sources will be essential for        |
| nformatics" | resolving the most vexing phylogenetic problems, and   |
| and         | coordinated research groups will be the most efficient |
| "Developing | means to achieve this objective. Specifically, the     |
| the         | workshop participants envision the funding of Tree of  |
| Technology  | Life "networks" and "hubs," and of a                   |
| and         | "phyloinformatics" facility focused on synthesis and   |
| Inf         | outreach.                                              |
| rastructure |                                                        |
| Needed for  | **Recommendations:** The NSF should establish, as soon |
| Assembly of | as possible, a new program focused on "Assembling the  |
| the Tree of | Tree of Life" (ATOL). The concrete benefits to science |
| Life***     | and society stemming from ATOL, and the feasibility of |
|             | accomplishing its major mission, justifies the         |
|             | development of a major new initiative and the          |
|             | investment necessary to build and maintain such a      |
|             | program. Specifically, the ATOL program should support |
|             | the development of the following new structures:       |
|             |                                                        |
|             | \- [Tree of Life Networks.]{.underline} TOLNets are    |
|             | the essential mechanisms for coordinating individual   |
|             | investigators from diverse fields of knowledge who are |
|             | working on reconstructing the phylogeny of Life        |
|             |                                                        |
|             | \- [Tree of Life Hubs]{.underline}*.* TOLHubs, with a  |
|             | concentration of expertise and specialized facilities, |
|             | would serve the ATOL effort as focal points for        |
|             | obtaining and synthesizing phylogenetic data. They     |
|             | would facilitate interactions among TOLNets and        |
|             | function as ATOL training centers.                     |
|             |                                                        |
|             | \- [Phyloinformatics and Coordination Infrastructure   |
|             | (PICI)]{.underline}*.* Centralization of the           |
|             | informatics program would avoid duplication of effort  |
|             | and facilitate integration among databases.            |
|             | Intellectual synergy would be promoted by co-locating  |
|             | research scientists, visiting scholars, and support    |
|             | staff in one place. Investment in such a center would  |
|             | establish a global resource and encourage cooperation  |
|             | with ongoing biodiversity and bioinformatics           |
|             | initiatives.                                           |
|             |                                                        |
|             | Throughout the development of this program it will be  |
|             | critical to support training relevant to all aspects   |
|             | of ATOL, and the development of new methods for        |
|             | gathering, analyzing, and synthesizing phylogenetic    |
|             | data. Furthermore, every effort must be made to foster |
|             | cross-disciplinary efforts, international              |
|             | collaboration, and linkage to other relevant programs. |
|             |                                                        |
|             | **Availability:**                                      |
|             | [                                                      |
|             | http://www.research.amnh.org/biodiversity]{.underline} |
+-------------+--------------------------------------------------------+
| ***The      | **Findings:** There are major areas of research as yet |
| Microbe     | untouched that would increase our understanding of the |
| Project: A  | broader microbial world, its diversity, and its        |
| Report from | potential applications. A coordinated interagency and  |
| the         | international effort is needed to seize the            |
| Interagency | opportunities offered by genome-enabled microbial      |
| Working     | science. In recognition of this need, the Microbe      |
| Group on    | Project Interagency Working Group was convened in      |
| Microbial   | August 2000, and charged with developing a coordinated |
| G           | interagency action plan or microbial genomics          |
| enomics.*** | activities. The Microbe Project has three broad goals: |
|             | to build needed infrastructure, promote research and   |
|             | develop human resources and an informed public.        |
|             |                                                        |
|             | **Recommendations:**                                   |
|             |                                                        |
|             | \- Microbial genome sequencing should be expanded to   |
|             | include scientifically important but as yet            |
|             | understudied microbes.                                 |
|             |                                                        |
|             | \- Individual agencies should continue, or as          |
|             | necessary, increase support for research on technique  |
|             | and tool development.                                  |
|             |                                                        |
|             | \- The Federal government should initiate a deliberate |
|             | planning effort to address the issue of providing      |
|             | sustained support for and access to microbial genomic  |
|             | resources.                                             |
|             |                                                        |
|             | \- Develop standardized bioinformatics tools for the   |
|             | analysis of microbial genomes.                         |
|             |                                                        |
|             | \- Database issues (including standardized annotation, |
|             | interoperability and long term support) must be        |
|             | resolved through an interagency effort with planning   |
|             | activities to begin immediately.                       |
|             |                                                        |
|             | \- Each agency, as its mission directs, should         |
|             | encourage and support genome--enabled microbial        |
|             | research objectives, as described in this report.      |
|             |                                                        |
|             | \- Individual and interagency activities initiated as  |
|             | part of the Microbe Project should contain elements    |
|             | that encourage training and /or educational            |
|             | activities, and include efforts to enhance the         |
|             | diversity of participants in all aspects of each       |
|             | activity.                                              |
|             |                                                        |
|             | \- Interagency coordination of the development and     |
|             | distribution of training materials should be           |
|             | encouraged.                                            |
|             |                                                        |
|             | \- Continue coordination cross agencies of all Microbe |
|             | Project activities, in part through the development of |
|             | an interagency Microbe Project web site.               |
|             |                                                        |
|             | Availability**:**                                      |
|             | [www.ostp.gov/html/microbial/st                        |
|             | art.htm](http://www.ostp.gov/html/microbial/start.htm) |
+-------------+--------------------------------------------------------+

+-------------+--------------------------------------------------------+
| **CISE**    |                                                        |
+-------------+--------------------------------------------------------+
| ***White    | **Scope:** Program Analysis.                           |
| Paper on an |                                                        |
| NSF ANIR    | **Findings:** This group was commissioned to make      |
| Middleware  | recommendations for an NSF middleware program.         |
| In          | Recommended that NSF support middleware research and a |
| itiative*** | complementary middleware infrastructure program.       |
|             |                                                        |
|             | **Availability:** Division Director, ANIR/NSF.         |
+-------------+--------------------------------------------------------+
| ***Report   | **Scope:** Assessed the value to the research          |
| of Review   | community of 3 HPIIS awards (Transpac -- Asia-Pacific; |
| Committee   | Euro-Link; and Mirnet -- Russia) that connect US       |
| of NSF's    | researchers to researchers in other countries.         |
| High        | Determine continuing need for HPIIS program.           |
| Performance |                                                        |
| In          | **Findings:** Transpac and Euro-Link are well run and  |
| ternational | effective. Mirnet is making an excellent start. The    |
| Internet    | report recommended a classification of usage types and |
| Services    | metrics for usage. Continued support and recognition   |
| (HPIIS)     | of needs at application level (as illustrated by the   |
| Project***  | ITR GryPhyN project) were recommended.                 |
|             |                                                        |
|             | **Availability:** Division Director, ANIR/NSF.         |
+-------------+--------------------------------------------------------+
| ***Report   | **Scope:** Studied the feasibility of online voting at |
| of the      | the request of the White House (memorandum, December   |
| National    | 17, 1999).                                             |
| Workshop on |                                                        |
| Internet    | **Findings:** Poll Site Internet voting offers         |
| Voting:     | benefits and could be fielded within the next several  |
| Issues and  | election cycles. Remote voting and Internet voting     |
| Research    | registration pose significant integrity issues. It is  |
| Agenda***   | appropriate for the NSF to address technical and       |
|             | social science research in this area.                  |
|             |                                                        |
|             | **Availability:** Internet Policy Institute            |
|             | ([[http://www.internetpolic                            |
|             | y.org/]{.underline}](http://www.internetpolicy.org/)). |
+-------------+--------------------------------------------------------+
| ***Making   | **Scope:** Identifies research areas that need         |
| IT Better:  | increased effort for the Nation to enjoy full benefits |
| Expanding   | of the Information Technology (IT) systems.            |
| Information |                                                        |
| Technology  | **Findings:** Report recommends that NSF and DARPA     |
| Research to | establish programs for research on large scale IT      |
| Meet        | systems; boosted funding for basic IT research         |
| Society's   | commensurate with growth of research challenges;       |
| Needs***    | increased support for interdisciplinary research on    |
|             | social applications of IT.                             |
|             |                                                        |
|             | **Availability:** Computer Science and                 |
|             | Telecommunications Board, National Research Council    |
|             | National Academy Press.                                |
|             | ([[h                                                   |
|             | ttp://www.nap.edu/]{.underline}](http://www.nap.edu/)) |
+-------------+--------------------------------------------------------+
| ***The      | **Scope:** A study of the Internet and key challenges  |
| Internet's  | that shape its maturation.                             |
| Coming of   |                                                        |
| Age***      | **Findings:** Recommended continued support for        |
|             | research on scaling challenges; partnerships for       |
|             | research to be conducted in realistic operational      |
|             | settings, and research on the economics of             |
|             | interconnection.                                       |
|             |                                                        |
|             | **Availability**: Computer Science and                 |
|             | Telecommunications Board, National Research Council,   |
|             | National Academy Press.                                |
|             | [[www.nap.edu]{.underline}](../www.nap.edu)            |
+-------------+--------------------------------------------------------+
| ***Report   | **Scope:** Examined state of research on digital       |
| to the      | libraries (DL).                                        |
| President,  |                                                        |
| Digital     | **Findings:** Recommendations to NSF and other         |
| Libraries:  | agencies. Expand research in DL including organizing   |
| Universal   | content, scalability of systems, archival storage,     |
| Access to   | intellectual property, privacy and security, and human |
| Human       | use. Create several large-scale DL testbeds. Make      |
| K           | Federal content persistently available on the          |
| nowledge*** | Internet. Play a leadership role in policy for         |
|             | intellectual property rights.                          |
|             |                                                        |
|             | **Availability:** President's Information Technology   |
|             | Advisory Committee, Panel on Digital Libraries         |
|             | National Coordinating Office, Arlington VA.            |
|             | ([www.ccic.gov](../www.ccic.gov))                      |
+-------------+--------------------------------------------------------+
| ***Report   | **Scope:** Examined the use of IT in the health care   |
| to the      | sector.                                                |
| President,  |                                                        |
| T           | **Findings:** Recommendations were made in several     |
| ransforming | areas focused on NIH and DHHS. Relevant to NSF were    |
| Health Care | recommendations to work with NIH, DARPA and DOE to     |
| Through     | design and deploy a scalable computing and information |
| Information | infrastructure supporting biomedical research. Several |
| Te          | IT research areas were identified.                     |
| chnology*** |                                                        |
|             | **Availability:** President's Information Technology   |
|             | Advisory Committee, Panel on Transforming Health Care  |
|             | National Coordinating Office, Arlington VA.            |
|             | ([www.ccic.gov](../www.ccic.gov))                      |
+-------------+--------------------------------------------------------+

+-------------+--------------------------------------------------------+
| ***Report   | **Scope:** Computing based on growing vulnerability in |
| to the      | the development of software for high end computing.    |
| President,  | Group assessed the open source model to address this   |
| Developing  | need.                                                  |
| Open Source |                                                        |
| Software to | **Findings:** The Federal government should encourage  |
| Advance     | open source software with efforts on technical         |
| High End    | assessment, management plans, policy studies, etc.     |
| C           | These recommendations are particularly pertinent to    |
| omputing*** | NSF's PACI and Terascale Facilities.                   |
|             |                                                        |
|             | **Availability:** President's Information Technology   |
|             | Advisory Committee, Panel on Open Source Software for  |
|             | High End Computing National Coordinating Office,       |
|             | Arlington VA. ([www.ccic.gov](../www.ccic.gov))        |
+-------------+--------------------------------------------------------+

+-------------+--------------------------------------------------------+
| ## **EHR**  |                                                        |
+-------------+--------------------------------------------------------+
| #           | **Scope:** To assess the impact of Fellowships on      |
| # ***The Gr | successful applicants.                                 |
| aduate Rese |                                                        |
| arch Fellow | [*The Graduate Research Fellowships (GRF)              |
| ships (GRF) | Program*-Abt/WestEd (REC9912174)]{.underline}.         |
|  Program*** |                                                        |
|             | **Findings:** Findings indicated an overall positive   |
|             | effect of the fellowships. Fellows complete the Ph.D.  |
|             | at a higher rate than non-Fellows, and the percentage  |
|             | of female Fellows completing the Ph.D. has become      |
|             | essentially the same as for men. Findings from the     |
|             | evaluation that relate to policy considerations        |
|             | include:                                               |
|             |                                                        |
|             | -   Highly qualified students are funded and the award |
|             |     is highly prestigious.                             |
|             |                                                        |
|             | -   Fellows consider the major advantages of the       |
|             |     fellowship to be its prestige and the flexibility  |
|             |     it allows in choosing a research area, structuring |
|             |     a graduate program, and selecting the educational  |
|             |     institution and mentor.                            |
|             |                                                        |
|             | -   About two thirds of the NSF fellows complete their |
|             |     degrees within nine years, with comparable         |
|             |     completion rates for female and male fellows.      |
|             |     Minority Graduate Fellowship recipients take       |
|             |     longer, but the gap is narrowing. Recipients of    |
|             |     the add-on Women in Engineering awards tend to     |
|             |     complete their doctorates at a faster but lower    |
|             |     rate than their male GRF counterparts in           |
|             |     engineering.                                       |
|             |                                                        |
|             | -   A large fraction of NSF fellows earn their         |
|             |     baccalaureates from a small number of prestigious  |
|             |     institutions (40% from 18 institutions in 2001).   |
|             |     This year\'s applicants, on the other hand, came   |
|             |     from 699 domestic and 69 foreign institutions.     |
|             |                                                        |
|             | ```{=html}                                             |
|             | <!-- -->                                               |
|             | ```                                                    |
|             | -   NSF fellows tend to enroll and complete doctorates |
|             |     in a small number of highly ranked institutions,   |
|             |     more so in some disciplines than others.           |
|             |     Minority-serving institutions (MSIs) were the      |
|             |     baccalaureate origins of a large fraction of       |
|             |     applicants and awardees for the (discontinued      |
|             |     in 1998) Minority Graduate Fellowship Program. The |
|             |     elimination of the separate minority competition   |
|             |     has resulted in a dramatic decrease in             |
|             |     applications, awards, and success rates of         |
|             |     applicants from MSIs.                              |
|             |                                                        |
|             | **Availability:** Available from EHR Directorate, NSF  |
+-------------+--------------------------------------------------------+

+-------------+--------------------------------------------------------+
| ##          | **Scope:** An evaluative study of 22 of the USI        |
|             | districts funded between 1994-1999.                    |
| ## ***Acad  |                                                        |
| emic Excell | **Findings:** NSF's investments in urban education     |
| ence for Ur | have led to dramatic improvements in student           |
| ban Student | achievement in science and mathematics in most of      |
| s - Their A | these funded sites. ESR's urban program has been a     |
| ccomplishme | catalyst for large-scale systemic change directed      |
| nts in Scie | towards improving the science and mathematics          |
| nce and Mat | achievement of all students. Greatest gains were in    |
| hematics*** | districts that had participated in the USI program for |
|             | the longest period of time. USI students made gains in |
|             | science and mathematics achievement, while reducing    |
|             | achievement gaps among racial/ethnic groups. Students  |
|             | substantially improved their enrollment rates in       |
|             | advanced science and mathematics courses.              |
|             | Underrepresented minority students made even greater   |
|             | gains than their peers during the same period,         |
|             | resulting in reduced enrollment disparities.           |
|             | Implementation of a standards-based curriculum and     |
|             | instruction, aligned assessment practices, and         |
|             | appropriate professional development are key to an     |
|             | increase in student achievement. The convergence of    |
|             | resources, a strong leadership structure, and          |
|             | effective partners were also critical to the           |
|             | improvement in student performance. The study          |
|             | concluded that the infrastructure developed by these   |
|             | districts would likely sustain the achievement gains.  |
|             | The study also concluded that it takes 7-10 years to   |
|             | bring about substantial improvement in systemic reform |
|             | that may lead to the gains cited in the report (Kim,   |
|             | 2001).                                                 |
|             |                                                        |
|             | **Availability:** The executive summary and full       |
|             | report can be downloaded at                            |
|             | [www.systemic.com/usi](http://www.systemic.com/usi)    |
|             | and                                                    |
|             | [www.siurbanstudy.org/newspub                          |
|             | lication](http://www.siurbanstudy.org/newspublication) |
+-------------+--------------------------------------------------------+
| ***Insti    | **Scope:** The evaluators examined the differences     |
| tution-wide | between institutions receiving awards and institutions |
| Reform      | that had received high ratings in the IR competition,  |
| Initiative  | but were declined.                                     |
| (IR)***     |                                                        |
|             | **Findings:** The purpose of this three-year           |
|             | initiative was to encourage broader reform in          |
|             | undergraduate institutions by providing further        |
|             | support for reform-related activities already          |
|             | underway. The study showed that IR support had a       |
|             | substantial effect on students, faculty, and           |
|             | curriculum, with 58,000 students and nearly 1700       |
|             | faculty participating, and more than 1200 courses      |
|             | developed or revised. Results indicate that the IR     |
|             | awards brought about change in some institutions,      |
|             | particularly 2-year colleges, but differences were     |
|             | small when participants were compared to a similarly   |
|             | motivated set of institutions.                         |
|             |                                                        |
|             | **Availability:** Available from EHR Directorate, NSF  |
+-------------+--------------------------------------------------------+

+-------------+--------------------------------------------------------+
| ***Un       | **Scope:** To determine the effectiveness of faculty   |
| dergraduate | training on classroom practices. [Undergraduate        |
| Faculty     | Faculty Enhancement (UFE) Program-SRI                  |
| Enhancement | (9412964).]{.underline}                                |
| (UFE)       |                                                        |
| Program***  | **Findings:** The UFE Program provided opportunities   |
|             | for the education of undergraduate faculty through     |
|             | workshops and other activities. Evaluators examined    |
|             | the impact of these. They found that the UFE program   |
|             | had led to 5,000 new courses, 7,300 major course       |
|             | revisions, and 8,600 moderate course revisions.        |
|             | Approximately 1,200 programs of study were developed   |
|             | or redesigned. In addition at least 2,700 other        |
|             | faculty had developed a new course or lab as a result  |
|             | of contact with colleagues who were colleagues who     |
|             | were UFE participants. The evaluators also estimated   |
|             | that, by 1999, more that 1,850,000 students (one in 22 |
|             | students nationally) had completed courses that were   |
|             | developed or had major revisions as a result of UFE.   |
|             | "Faculty reported that students in their revised or    |
|             | modified courses performed better along a number of    |
|             | dimensions than comparable students in traditional     |
|             | courses. Faculty also cited improvements in students'  |
|             | abilities to solve problems, think critically,         |
|             | communicate, collaborate, use technology, and          |
|             | understand the scientific method."                     |
|             |                                                        |
|             | **Availability:** SRI. Available from EHR Directorate, |
|             | NSF                                                    |
+-------------+--------------------------------------------------------+

+-------------+--------------------------------------------------------+
| **ENG**     |                                                        |
+-------------+--------------------------------------------------------+
| ***Envi     | **Scope:** The report includes global benchmarking of  |
| ronmentally | the current technologies, systems, and policies in     |
| Benign      | manufacturing, with suggested recommendations for      |
| Manuf       | future research needs. Use of metals and polymers in   |
| acturing*** | the automotive and the electronics sectors were the    |
|             | primary topical areas.                                 |
|             |                                                        |
|             | **Findings:** The study found that better tools, data, |
|             | metrics, and technologies were needed on specific      |
|             | materials and industrial sectors. It called for        |
|             | development of high-performance business practices     |
|             | such as supply chain management, goals alignment, and  |
|             | assessment tracking.                                   |
|             |                                                        |
|             | **Availability:** World Technology Evaluation Center   |
|             | (WTEC) of the International Technology Research        |
|             | Institute at Loyola College of Maryland.               |
|             | <http://itri.loyola.edu/ebm/ebm.pdf>                   |
+-------------+--------------------------------------------------------+
| ***Outcomes | **Scope:** Focused on research cooperation between     |
| and Impacts | industries and universities.                           |
| of the      |                                                        |
| Sta         | **Findings:** Found that the program has been a modest |
| te/Industry | success as measured against its goals and objectives   |
| -University | and compared with the outcomes and impacts of the      |
| Cooperative | I/UCRC program that served as its model.               |
| Research    |                                                        |
| Centers     | **Availability:** Available from NSF (NSF 01-110)      |
| (S/IUCRC)   |                                                        |
| Program***  |                                                        |
+-------------+--------------------------------------------------------+
| ***Trends   | **Scope:** Focused on research cooperation between     |
| in          | industries and universities                            |
| Industrial  |                                                        |
| Support of  | **Findings:** The ongoing study found that Engineering |
| Unive       | Research Centers (ERCs) working to extend an           |
| rsity-Based | established area of interest in industry are           |
| Cooperative | attractive to large, research-intensive firms that     |
| Research*** | have a long-term interest in the results of the        |
|             | center's work. In contrast, centers working in areas   |
|             | that are out in front of existing product lines or     |
|             | corporate interests are much more likely be of         |
|             | interest to small firms and firms that do little or no |
|             | research and have few or no financial resources to     |
|             | support center work.                                   |
|             |                                                        |
|             | []{.mark}                                              |
|             |                                                        |
|             | **Availability:** Available from Engineering           |
|             | Directorate, NSF                                       |
+-------------+--------------------------------------------------------+
| ***Impact   | **Scope:** Examines the outcomes and impacts of ERC    |
| of          | membership on firms that are members of mature         |
| Interaction | second-generation ERCs (centers in the classes of      |
| with        | 1994--96) and identifies changes in firms'             |
| Engineering | interactions with ERCs due to changes in the program   |
| Research    | and in industry compared with first-generation ERCs    |
| Centers on  | (classes of 1985--90).                                 |
| Industry:   |                                                        |
| Repeat      |                                                        |
| Study***    |                                                        |
+-------------+--------------------------------------------------------+

+-------------+--------------------------------------------------------+
| **GEO**     |                                                        |
+-------------+--------------------------------------------------------+
| ***Ocean    | **Scope:** The Decadal Committee was charged to        |
| Sciences at | consider existing reports, additional sources of       |
| the New     | information, and community input in developing a       |
| Mil         | report summarizing the directions for ocean science    |
| lennium.*** | over the next decade.                                  |
|             |                                                        |
|             | **Findings:** Numerous findings were made relating to  |
|             | the scientific opportunities in ocean sciences in the  |
|             | coming decade. The committee recommended: the          |
|             | development of a multi-agency fleet replacement plan;  |
|             | a vigorous effort in technology development,           |
|             | implementation and support in all areas of ocean       |
|             | science; improvement of databases and ready access to  |
|             | these databases by the scientific community; continued |
|             | emphasis on the development of models that link        |
|             | different parts of the ocean system; and a vigorous    |
|             | exploration of a new class of controlled perturbation  |
|             | experiments.                                           |
|             |                                                        |
|             | **Availability:** National Academy of Sciences         |
|             | [www.nas.edu](../www.nas.edu)                          |
+-------------+--------------------------------------------------------+
| ***Initial  | **Scope:** To examine the scientific significance,     |
| Science     | technical feasibility, and potential societal benefits |
| Plan (ISP)  | of the ISP.                                            |
| for the     |                                                        |
| Integrated  | **Findings:** The Committee reaffirmed that the        |
| Ocean       | scientific significance, technical feasibility, and    |
| Drilling    | potential societal benefits of the ISP make it of      |
| Program***  | exceptional importance and timeliness. The Committee   |
|             | concluded that the benefits of the program described   |
|             | in the ISP far outweigh the costs and the technical    |
|             | uncertainties. The Committee gave its unreserved       |
|             | support to the priorities of the program as described  |
|             | in the ISP. A number of specific recommendations on    |
|             | scientific and technological objectives, facilities,   |
|             | organizational and implementation options, and         |
|             | resource requirements were included.                   |
|             |                                                        |
|             | **Availability:** National Academy of Sciences,        |
|             | [www.nas.edu](../www.nas.edu)                          |
+-------------+--------------------------------------------------------+
| ***Basic    | **Scope:** To undertake a major study of research in   |
| Research    | the Earth Sciences.                                    |
| Op          |                                                        |
| portunities | **Findings:** The Board found that the Division of     |
| in Earth    | Earth Sciences (EAR) has done an excellent job in      |
| Science***  | maintaining the balance among core programs supporting |
|             | investigator-driven disciplinary research,             |
|             | problem-focused programs of multidisciplinary          |
|             | research, and equipment-oriented programs for new      |
|             | instrumentation and facilities. The committee offers   |
|             | recommendations that address the evolving science      |
|             | requirements in all three of these programmatic areas  |
|             | and primarily pertain to new mechanisms that will      |
|             | allow EAR to exploit research opportunities identified |
|             | by the committee. The Board also strongly endorsed the |
|             | four observational components of the *EarthScope*      |
|             | Initiative.                                            |
|             |                                                        |
|             | **Availability:** National Research Council/National   |
|             | Academy Press, 2000, [www.nas.edu](../www.nas.edu)     |
+-------------+--------------------------------------------------------+

+-------------+--------------------------------------------------------+
| **MPS**     |                                                        |
+-------------+--------------------------------------------------------+
| ***Physics  | **Scope:** The report surveys the field of physics     |
| in a New    | broadly, identifies priorities, and formulates         |
| Era***      | recommendations. The overview assesses the state of    |
|             | physics in four broad categories -- quantum            |
|             | manipulation and new materials, complex systems,       |
|             | structure and evolution of the universe, and           |
|             | fundamental laws and symmetries -- emphasizing the     |
|             | unity of the field and the strong commonality that     |
|             | links the different areas, while highlighting new and  |
|             | emerging ones.                                         |
|             |                                                        |
|             | **Findings:** Six high-priority opportunities          |
|             | identified, nine recommendations are made: support of  |
|             | physics by the federal government; physics education;  |
|             | role of basic physics research in national security;   |
|             | increasingly important role of partnerships among      |
|             | universities, industry, and national labs; the         |
|             | stewardship of federal science agencies; and the       |
|             | rapidly changing role of information technology in     |
|             | physics research and education.                        |
|             |                                                        |
|             | **Availability:** National Research Council,           |
|             | <http://www.nap.edu/catalog/10118.html>                |
+-------------+--------------------------------------------------------+
| ***An       | **Scope:** An assessment of the scientific quality of  |
| Assessment  | the Department of Energy (DOE) Office of Fusion Energy |
| of the      | Sciences Program.                                      |
| Department  |                                                        |
| of Energy's | **Findings:** Although this report was generated at    |
| Office of   | the request of the DOE's Office of Science, NSF is     |
| Fusion      | often referred to within the document. In particular,  |
| Energy      | the report recommends that NSF play a greater role in  |
| Sciences    | extending the reach of fusion science and in           |
| Program.*** | sponsoring general plasma science.                     |
|             |                                                        |
|             | **Availability:** National Research Council,           |
|             | <http://books.nap.edu/catalog/9986.html>               |
+-------------+--------------------------------------------------------+
| *           | **Scope:** To assess the organizational effectiveness  |
| **Committee | of Federal support of astronomical sciences and,       |
| on          | specifically, the pros and cons of transferring NSF\'s |
| O           | astronomy responsibilities to NASA.                    |
| rganization |                                                        |
| and         | **Findings:** NSF's astronomy and astrophysics         |
| Management  | responsibilities should not be transferred to NASA.    |
| of Research | The Federal government should develop a single         |
| in          | integrated strategy for astronomy and astrophysics     |
| Astronomy   | research that includes supporting facilities both on   |
| and         | the ground and in space. An interagency planning board |
| Astr        | for astronomy and astrophysics should be formed that   |
| ophysics*** | would receive input from the community through a joint |
|             | advisory committee of outside experts. Additional      |
|             | recommendations address ways to improve the present    |
|             | overall management structure and strengthen NSF's      |
|             | ability to support astronomy.                          |
|             |                                                        |
|             | **Availability:** National Research Council            |
|             | (<http://www.nas.edu/>[)]{.underline} with the         |
|             | prepublication copy available at                       |
|             | <ht                                                    |
|             | tp://books.nap.edu/html/integrated_program/comraa.pdf> |
+-------------+--------------------------------------------------------+
| ***         | **Scope:** Define research opportunities in the field  |
| Proceedings | of solid-state chemistry and materials; identify the   |
| of the      | most important multidisciplinary areas for involvement |
| Workshop on | by the solid-state chemistry and materials community;  |
| the Present | determine novel roles for the Solid State Chemistry    |
| Status and  | and Materials community that will advance educational  |
| Future      | and training opportunities for future scientists,      |
| D           | engineers, and technicians; develop new approaches     |
| evelopments | that allow for the more effective and efficient        |
| of Solid    | conduct of research and educational activities.        |
| State       |                                                        |
| Chemistry   | **Findings:** Numerous recommendations are listed for  |
| and         | various sub-fields in this discipline.                 |
| M           |                                                        |
| aterials*** | **Availability:** NSF web site                         |
|             | <http://www.nsf.gov/mps/dmr/ssc.pdf>                   |
+-------------+--------------------------------------------------------+
| *           | **Scope:** The workshop explored research              |
| **US-Africa | opportunities directed towards expanding materials     |
| Materials   | research and education for the purpose of contributing |
| Workshop*** | to the development of new technologies as well as      |
|             | promoting collaboration among U.S. and African         |
|             | universities and industries.                           |
|             |                                                        |
|             | **Findings:** Need to establish one or more            |
|             | organizations to ensure the continuation of            |
|             | conversations that began at the workshop. Technical    |
|             | recommendations from a number of working groups are    |
|             | included in the report.                                |
|             |                                                        |
|             | **Availability:** <http://iumrs.org/docs/africa.pdf>   |
+-------------+--------------------------------------------------------+
| *National   | **Scope:** The goal of this workshop was to explore    |
| Science     | recent advances in research on force transduction in   |
| Foundation\ | biology at all length scales, and to seek possible     |
| Force       | overlap or synergies between these different areas. An |
| T           | additional goal was to explore the potential           |
| ransduction | interdisciplinary interactions that will lead to       |
| in Biology  | significant advances in this area. Also, the workshop  |
| Workshop\   | was to identify important new directions for research  |
| *           | and to make recommendations about potential funding    |
|             | opportunities.                                         |
|             |                                                        |
|             | **Findings:** Perhaps the most important conclusion of |
|             | the workshop was that research in force transduction   |
|             | in biology has important problems that span many       |
|             | length scales and many disciplines. However, the       |
|             | interdisciplinary nature of the research, the          |
|             | quantitative nature of the important problems and the  |
|             | key relationship between the materials properties and  |
|             | the important issues all make this an area that the    |
|             | NSF can play a significant role in fostering progress. |
|             |                                                        |
|             | **Availability:**                                      |
|             | <http://hurkle.deas.harvard.edu/nsf/workshop.html>     |
+-------------+--------------------------------------------------------+

+-------------+--------------------------------------------------------+
| SBE         |                                                        |
+-------------+--------------------------------------------------------+
| ***The      | **Focus:** The aim was to: (1) survey current studies  |
| Societal    | on the societal implications of nanotechnology         |
| I           | (educational, technological, economic, medical,        |
| mplications | environmental, ethical, legal, etc.); (2) identify     |
| of          | investigative and assessment methods for future        |
| Nanoscience | studies of societal implications; (3) propose a vision |
| and         | for accomplishing nanotechnology's promise while       |
| Nanote      | minimizing undesirable consequences.                   |
| chnology*** |                                                        |
|             | **Availability:** The report has been published both   |
|             | on the web and in book form                            |
|             | (<http                                                 |
|             | ://itri.loyola.edu/nano/NSET.Societal.Implications/>). |
+-------------+--------------------------------------------------------+

# APPENDIX II. -- SCHEDULE OF PROGRAM EVALUATIONS

The following table provides information on the scheduling of meetings
for Committees of Visitors (COVs) for our programs. The table lists the
fiscal year of the most recent COV meeting for the program and the
fiscal year for the next COV review of the program. We have highlighted
the COV meetings that were held in FY 2001 in bold font.

**Committee of Visitors Meetings**

**By Directorate**

**(COV meetings held during FY 2001 are highlighted in bold font)**

+------------------------------------------------------+-------+-------+
| ##### **DIRECTORATE**                                | F     | F     |
|                                                      | iscal | iscal |
|                                                      | Year  | Year  |
|                                                      | of    | of    |
+------------------------------------------------------+-------+-------+
| *Division*                                           | Most  | Next  |
+------------------------------------------------------+-------+-------+
| Program                                              | R     | COV   |
|                                                      | ecent |       |
|                                                      | COV   |       |
+------------------------------------------------------+-------+-------+
| ##### **BIOLOGICAL SCIENCES**                        |       |       |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| *Biological Infrastructure*                          |       |       |
+------------------------------------------------------+-------+-------+
| Instrument Related Activities                        | 2000  | 2002  |
+------------------------------------------------------+-------+-------+
| Research Resources                                   | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Training                                             | 2000  |       |
+------------------------------------------------------+-------+-------+
| **Plant Genome**                                     | **2   |       |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| *Environmental Biology*                              |       |       |
+------------------------------------------------------+-------+-------+
| Ecological Studies                                   | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
| **Long Term Research**                               | **2   |       |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| Systematic and Population Biology                    | 2000  | 2004  |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| #  *Integrative Biology and Neuroscience*            | **2   | 2005  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| Neuroscience                                         | 1999  | 2003  |
+------------------------------------------------------+-------+-------+
| Developmental Mechanisms                             | 2000  | 2004  |
+------------------------------------------------------+-------+-------+
| Physiology and Ethnology                             | 1997  | 2002  |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| *Molecular and Cellular Biosciences*                 |       | 2002  |
+------------------------------------------------------+-------+-------+
| Biomolecular Structure and Function                  | 2000  |       |
+------------------------------------------------------+-------+-------+
| Biomolecular Processes                               | 2000  |       |
+------------------------------------------------------+-------+-------+
| **Cell Biology**                                     | **2   |       |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| Genetics                                             | 1999  |       |
+------------------------------------------------------+-------+-------+

+------------------------------------------------------+-------+-------+
| ##### **DIRECTORATE**                                | F     | F     |
|                                                      | iscal | iscal |
|                                                      | Year  | Year  |
|                                                      | of    | of    |
+------------------------------------------------------+-------+-------+
| *Division*                                           | Most  | Next  |
+------------------------------------------------------+-------+-------+
| Program                                              | R     | COV   |
|                                                      | ecent |       |
|                                                      | COV   |       |
+------------------------------------------------------+-------+-------+
| **COMPUTER AND INFORMATION SCIENCE AND ENGINEERING** |       |       |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| *Advanced Computational Infrastructure and Research* |       |       |
+------------------------------------------------------+-------+-------+
| **Advanced Computational Research**                  | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| #####  PACI                                          | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| *Computer-Communications Research*                   |       |       |
+------------------------------------------------------+-------+-------+
| Communications                                       | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Computer Systems Architecture                        | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Design Automation                                    | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Hybrid and Embedded Systems (new in '02)             | N/A   | 2003  |
+------------------------------------------------------+-------+-------+
| Numeric, Symbolic and Geometric Computation          | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Operating Systems and Compilers                      | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Signal Processing Systems                            | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Software Engineering and Languages                   | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| #####  Theory of Computing                           | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Trusted Computing (new in '02)                       | N/A   | 2003  |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| *Information and Intelligent Systems*                |       |       |
+------------------------------------------------------+-------+-------+
| Computation and Social Systems                       | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
| Human Computer Interaction                           | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
| Knowledge and Cognitive Systems                      | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
| Robotics and Human Augmentation                      | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
| Information and Data Management                      | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| *Advanced Networking Infrastructure and Research*    |       |       |
+------------------------------------------------------+-------+-------+
| Networking Research                                  | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Special Projects in Networking Research              | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Advanced Networking Infrastructure                   | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| *Information Technology Research (ITR) (new in '00)* | N/A   | 2003  |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| ***Experimental and Integrative Activities***        | **2   |       |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| -Instrumentation Infrastructure Cluster              |       |       |
+------------------------------------------------------+-------+-------+
| **Research Infrastructure**                          | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| #####  Research Resources (new in '02 )              | N/A   | 2004  |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| -Multidisciplinary Research Cluster                  |       |       |
+------------------------------------------------------+-------+-------+
| Biological Information Technology and Systems (new   | N/A   | 2004  |
| in '02)                                              |       |       |
+------------------------------------------------------+-------+-------+
| Quantum and Biologically Inspired Computing (new in  | N/A   | 2004  |
| '02)                                                 |       |       |
+------------------------------------------------------+-------+-------+
| **Digital Government**                               | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| #####  **Next Generation Software**                  | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| -Education Workforce Cluster                         |       |       |
+------------------------------------------------------+-------+-------+
| Information Technology Workforce (new in '02)        | N/A   | 2004  |
+------------------------------------------------------+-------+-------+
| #####  **Minority Institutions Infrastructure**      | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| **CISE Educational Innovation**                      | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| **\*\*CISE Postdoctoral Research Associates**        | **2   |       |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| -EIA Special Projects Cluster                        |       |       |
+------------------------------------------------------+-------+-------+
| Special Projects (new in '02)                        | N/A   | 2004  |
+------------------------------------------------------+-------+-------+
| **\*\*NSF-CONACyT Collaborative Research**           | **2   |       |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| **\*\*NSF-CNPq Collaborative Research**              | **2   |       |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| \*\*EIA monitored, managed/reviewed by Division in   |       |       |
| Partnership with Engineering                         |       |       |
+------------------------------------------------------+-------+-------+

+------------------------------------------------------+-------+-------+
| ##### **DIRECTORATE**                                | F     | F     |
|                                                      | iscal | iscal |
|                                                      |       | Year  |
|                                                      | Year  |       |
+------------------------------------------------------+-------+-------+
| *Division*                                           | of    | of    |
|                                                      | Most  | Next  |
+------------------------------------------------------+-------+-------+
| Program                                              | R     | COV   |
|                                                      | ecent |       |
|                                                      |       |       |
|                                                      | COV   |       |
+------------------------------------------------------+-------+-------+
| ##### **EDUCATION AND HUMAN RESOURCES**              |       |       |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| *Educational Systemic Reform*                        |       |       |
+------------------------------------------------------+-------+-------+
| **Statewide Systemic Initiatives**                   | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| **Urban Systemic Initiatives**                       | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| **Rural Systemic Initiatives**                       | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| #####                                                |       |       |
+------------------------------------------------------+-------+-------+
| #####  *Office of Innovation Partnerships*           |       |       |
+------------------------------------------------------+-------+-------+
| #                                                    | N/A   | 2004  |
| ####  Innovation Partnership Activities (new in '01) |       |       |
+------------------------------------------------------+-------+-------+
| #####  EPSCoR                                        | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| *Elementary, Secondary and Informal Education*       |       |       |
+------------------------------------------------------+-------+-------+
| **Informal Science Education**                       | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| Teacher Enhancement                                  | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Instructional Materials Development                  | 1997  | 2002  |
+------------------------------------------------------+-------+-------+
| Centers for Learning and Teaching (new in '01)       | N/A   | 2004  |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| *Undergraduate Education*                            |       |       |
+------------------------------------------------------+-------+-------+
| Teacher Preparation                                  | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Advanced Technological Education                     | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| NSF Computer, Science, Engineering and Mathematics   | N/A   | 2002  |
|                                                      |       |       |
| Scholarships (new in '01)                            |       |       |
+------------------------------------------------------+-------+-------+
| Distinguished Teaching Scholars (new in '02)         | N/A   | 2004  |
+------------------------------------------------------+-------+-------+
| Scholarship for Service (new in '01)                 | N/A   | 2004  |
+------------------------------------------------------+-------+-------+
| National SMETE Digital Library (new in '01)          | N/A   | 2002  |
+------------------------------------------------------+-------+-------+
| Course, Curriculum, and Laboratory Improvement       | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Undergraduate Assessment (new in '02)                | N/A   | 2004  |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| *Graduate Education*                                 |       |       |
+------------------------------------------------------+-------+-------+
| Graduate Research Fellowships                        | 1999  | 2003  |
+------------------------------------------------------+-------+-------+
| **NATO Postdoctorate Fellowships**                   | **2   | 2005  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| IGERT (new in '97)                                   |       | 2002  |
+------------------------------------------------------+-------+-------+
| GK-12 Fellows (new in '99)                           | N/A   | 2002  |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| *Human Resource Development*                         |       |       |
+------------------------------------------------------+-------+-------+
| **The Louis Stokes Alliances for Minority            | **2   | 2004  |
| Participation**                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| **Centers for Research Excellence In Science and     | **2   | 2004  |
| Technology**                                         | 001** |       |
|                                                      |       |       |
| **(CREST)**                                          |       |       |
+------------------------------------------------------+-------+-------+
| Programs for Gender Equity (PGE)                     | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Programs for Persons with Disabilities (PPD)         | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| **Alliances for Graduate Education and the           | **2   | 2004  |
| Professoriate (AGEP)**                               | 001** |       |
+------------------------------------------------------+-------+-------+
| Tribal Colleges Program (TCP) (new in '01)           | N/A   | 2004  |
+------------------------------------------------------+-------+-------+
| **Historically Black Colleges and Universities       | **2   | 2004  |
| (HBCU)**                                             | 001** |       |
+------------------------------------------------------+-------+-------+

+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| *Research, Evaluation & Communications*              |       |       |
+------------------------------------------------------+-------+-------+
| #####  REPP/ROLE (new in '96)                        |       | 2002  |
+------------------------------------------------------+-------+-------+
| Evaluation                                           | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Education Research Initiative (ERI) (new in '01)     | N/A   | 2002  |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| *Other*                                              |       |       |
+------------------------------------------------------+-------+-------+
| H-IB VISA K-12                                       |       | 2004  |
+------------------------------------------------------+-------+-------+
| Math and Science Partnership (MSP) (new in '02)      | N/A   | 2005  |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+

+------------------------------------------------------+-------+-------+
| **DIRECTORATE**                                      | F     | F     |
|                                                      | iscal | iscal |
|                                                      | Year  | Year  |
|                                                      | of    | of    |
+------------------------------------------------------+-------+-------+
| *Division*                                           | Most  | Next  |
+------------------------------------------------------+-------+-------+
| Program                                              | R     | COV   |
|                                                      | ecent |       |
|                                                      | COV   |       |
+------------------------------------------------------+-------+-------+
| **ENGINEERING**                                      |       |       |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| *Bioengineering and Environmental Systems*           |       | 2002  |
+------------------------------------------------------+-------+-------+
| #####  Biochemical Engineering                       | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
| Biotechnology                                        | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
| Biomedical Engineering                               | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
| Research to Aid the Disabled                         | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
| Environmental Engineering                            | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
| Environmental Technology                             | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| ***Civil and Mechanical Systems***                   | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| **Dynamic System Modeling, Sensing and Control**     | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| **Geotechnical and GeoHazard Systems**               | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| **Infrastructure and Information Systems**           | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| #####  **Solid Mechanics and Materials Engineering** | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| #####  **Structural Systems and Engineering**        | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| **Network for Earthquake Engineering Simulation**    | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| *Chemical and Transport Systems*                     |       | 2003  |
+------------------------------------------------------+-------+-------+
| Chemical Reaction Processes                          | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Interfacial, Transport and Separation Processes      | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Fluid and Particle Processes                         | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Thermal Systems                                      | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| *Design, Manufacture and Industrial Innovation*      |       |       |
+------------------------------------------------------+-------+-------+
| -Engineering Decision Systems Programs (new in '02)  | N/A   | 2003  |
+------------------------------------------------------+-------+-------+
| Engineering Design                                   | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Manufacturing Enterprise Systems (new in '02)        | N/A   | 2003  |
+------------------------------------------------------+-------+-------+
| Service Enterprise Systems (new in '02)              | N/A   | 2003  |
+------------------------------------------------------+-------+-------+
| Operations Research                                  | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| -Manufacturing Processes and Equipment Systems       | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Materials Processing and Manufacturing               | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Manufacturing Machines and Equipment                 | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Nanomanufacturing (new in '02)                       | N/A   | 2003  |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| -Industrial Innovation Programs Cluster              |       |       |
+------------------------------------------------------+-------+-------+
| **Small Business Innovation Research (SBIR)**        | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| Innovation and Organizational Change                 | 2000  |       |
+------------------------------------------------------+-------+-------+
| **Grant Opportunities for Academic Liaison with      | 2000  | 2003  |
| Industry**                                           |       |       |
+------------------------------------------------------+-------+-------+
| #####  **Small Business Technology Transfer**        | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| *Electrical and Communications Systems*              |       |       |
+------------------------------------------------------+-------+-------+
| Electronics, Photonics and Device Technologies       | 2000  | 2002  |
+------------------------------------------------------+-------+-------+
| Control, Networks, and Computational Intelligence    | 2000  | 2002  |
+------------------------------------------------------+-------+-------+
| Integrative Systems (new in '02)                     | N/A   | 2002  |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| ***Engineering, Education and Centers***             | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| **Engineering Education**                            | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| **Engineering Research Centers**                     | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| **Earthquake Engineering Research Centers**          | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| **Human Resource Development**                       | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| **State/Industry/University Cooperative Research     | **2   | 2004  |
| Centers**                                            | 001** |       |
+------------------------------------------------------+-------+-------+
| **Industry/Univ. Cooperative Research Centers**      | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+

+------------------------------------------------------+-------+-------+
| **DIRECTORATE**                                      | F     | F     |
|                                                      | iscal | iscal |
|                                                      | Year  | Year  |
|                                                      | of    | of    |
+------------------------------------------------------+-------+-------+
| *Division*                                           | Most  | Next  |
+------------------------------------------------------+-------+-------+
| Program                                              | R     | COV   |
|                                                      | ecent |       |
|                                                      |       |       |
|                                                      | COV   |       |
+------------------------------------------------------+-------+-------+
| ##### **GEOSCIENCES**                                |       |       |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| *Atmospheric Sciences*                               |       |       |
+------------------------------------------------------+-------+-------+
| *-*Lower Atmospheric Research Cluster                |       |       |
+------------------------------------------------------+-------+-------+
| **Atmospheric Chemistry**                            | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| **Climate Dynamics**                                 | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| **Meoscale Dynamic Meteorology**                     | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| **Large-scale Dynamic Meteorology**                  | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| **Physical Meteorology**                             | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| #####  **Paleoclimate**                              | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| *-*Upper Atmospheric Research Cluster                |       |       |
+------------------------------------------------------+-------+-------+
| Magnetospheric Physics                               | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
| Aeronomy                                             | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
| Upper Atmospheric Research Facilities                | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
| Solar Terrestrial Research                           | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| -Centers and Facilities Cluster                      |       |       |
+------------------------------------------------------+-------+-------+
| Lower Atmospheric Observing Facilities               | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| UNIDATA                                              | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| NCAR/UCAR                                            | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| *Earth Sciences*                                     |       |       |
+------------------------------------------------------+-------+-------+
| Instrumentation and Facilities                       | 1997  | 2004  |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| *-*Research Support Cluster                          | 1998  |       |
+------------------------------------------------------+-------+-------+
| Tectonics                                            | 1998  | 2002  |
+------------------------------------------------------+-------+-------+
| Geology and Paleontology                             | 1998  | 2002  |
+------------------------------------------------------+-------+-------+
| Hydrological Sciences                                | 1998  | 2002  |
+------------------------------------------------------+-------+-------+
| Petrology and Geochemistry                           | 1998  | 2002  |
+------------------------------------------------------+-------+-------+
| Geophysics                                           | 1998  | 2002  |
+------------------------------------------------------+-------+-------+
| Continental Dynamics                                 | 1998  | 2002  |
+------------------------------------------------------+-------+-------+

  ------------------------------------------------------ -------- --------
                                                                  

  *Ocean Sciences*                                                

  *-*Integrative Programs Cluster                        1997     2002

  Oceanographic Technical Services                       1994     2002

  Ship Operations                                        1994     2002

  Oceanographic Instrumentation                          1994     2002

  Ship Acquisitions and Upgrades (new in '02)            N/A      2002

  Shipboard Scientific Support Equipment (new in '02)    N/A      2002

  Oceanographic Tech and Interdisciplinary Coordination  1998     2002

                                                                  

  *-*Marine Geosciences Cluster                                   

  Marine Geology and Geophysics                          1998     2003

  Ocean Drilling                                         1994     2003

                                                                  

  -Ocean Cluster                                                  

  Chemical Oceanography                                  1998     2003

  Physical Oceanography                                  1998     2003

  Biological Oceanography                                1998     2003
  ------------------------------------------------------ -------- --------

+------------------------------------------------------+-------+-------+
| Program                                              | R     | COV   |
|                                                      | ecent |       |
|                                                      |       |       |
|                                                      | COV   |       |
+------------------------------------------------------+-------+-------+
| **MATHEMATICAL AND PHYSICAL SCIENCES**               |       |       |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| *Astronomical Sciences*                              | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
| Planetary Astronomy                                  | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
| Stellar Astronomy and Astrophysics                   | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
| Galactic Astronomy                                   | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
| Education, Human Resources and Special Programs      | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
| Advanced Technologies and Instrumentation            | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
| Electromagnetic Spectrum Management                  | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
| Extragalactic Astronomy and Cosmology                | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| *-*Facilities Cluster                                |       |       |
+------------------------------------------------------+-------+-------+
| Gemini 8-Meter Telescopes                            | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
| National Radio Astronomy Observatory (NRAO)          | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
| National Optical Astronomy Observatories (NOAO)      | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
| National Astronomy and Ionosphere Center (NAIC)      | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| ***Chemistry***                                      | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| **Office of Special Projects**                       | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| **Chemistry Research Instrumentation and Facilities  | **2   | 2004  |
| (CRIF)**                                             | 001** |       |
+------------------------------------------------------+-------+-------+
| **Organic Chemical Dynamics**                        | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| **Organic Synthesis**                                | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| #####  **Chemistry of Materials**                    | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| **Theoretical and Computational Chemistry**          | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| **Experimental Physical Chemistry**                  | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| **Inorganic, Bioinorganic and Organometallic         | **2   | 2004  |
| Chemistry**                                          | 001** |       |
+------------------------------------------------------+-------+-------+
| **Analytical and Surface Chemistry**                 | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+

+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| *Materials Research*                                 | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
| -Base Science Cluster                                |       |       |
+------------------------------------------------------+-------+-------+
| Condensed Matter Physics                             | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
| Solid-State Chemistry                                | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
| Polymers                                             | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| -Advanced Materials and Processing Cluster           |       |       |
+------------------------------------------------------+-------+-------+
| Metals                                               | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
| Ceramics                                             | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
| Electronic Materials                                 | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| -Materials Research and Technology Enabling Cluster  |       |       |
+------------------------------------------------------+-------+-------+
| Materials Theory                                     | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
| Instrumentation for Materials Research               | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
| National Facilities                                  | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
| Materials Research Science and Engineering Centers   | 1999  | 2002  |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| ***Mathematical Sciences***                          | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| **Applied Mathematics**                              | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| **Topology and Foundations**                         | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| **Computational Mathematics**                        | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| **Infrastructure**                                   | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| **Geometric Analysis**                               | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| #####  **Analysis**                                  | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| **Algebra, Number Theory, and Combinatories**        | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| **Statistics and Probability**                       | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| *Physics*                                            | 2000  |       |
+------------------------------------------------------+-------+-------+
| Atomic, Molecular, Optical and Plasma Physics        | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Elementary Particle Physics                          | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Theoretical Physics                                  | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Particle and Nuclear Astrophysics (new in '00)       | N/A   | 2003  |
+------------------------------------------------------+-------+-------+
| Nuclear Physics                                      | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Education and Interdisciplinary Research (new in     | N/A   | 2003  |
| '00)                                                 |       |       |
+------------------------------------------------------+-------+-------+
| Gravitational Physics                                | 2000  | 2003  |
+------------------------------------------------------+-------+-------+

+------------------------------------------------------+-------+-------+
| **DIRECTORATE**                                      | F     | F     |
|                                                      | iscal | iscal |
|                                                      |       | Year  |
|                                                      | Year  | of    |
+------------------------------------------------------+-------+-------+
| *Division*                                           | of    | Next  |
|                                                      | Most  |       |
+------------------------------------------------------+-------+-------+
| Program                                              | R     | COV   |
|                                                      | ecent |       |
|                                                      | COV   |       |
+------------------------------------------------------+-------+-------+
| ##### **SOCIAL, BEHAVIORAL, AND ECONOMIC SCIENCES**  |       |       |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| *Office of International Science and Engineering     | 1999  | 2002  |
| (INT)*                                               |       |       |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| *Science Resource Statistics (SRS) (new in '99)*     |       | 2004  |
+------------------------------------------------------+-------+-------+
| -NSF-wide Programs Cluster                           |       |       |
+------------------------------------------------------+-------+-------+
| **CAREER**                                           | **2   |       |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
| ADVANCE (new in '01)                                 |       |       |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| *Behavioral and Cognitive Sciences (BCS)*            |       | 2004  |
+------------------------------------------------------+-------+-------+
| Archeology and Archaeometry                          | 1999  | 2004  |
+------------------------------------------------------+-------+-------+
| Child Learning and Development                       | 1997  | 2004  |
+------------------------------------------------------+-------+-------+
| Cultural Anthropology                                | 1999  | 2004  |
+------------------------------------------------------+-------+-------+
| Linguistics                                          | 1999  | 2004  |
+------------------------------------------------------+-------+-------+
| Human Cognition and Perception                       | 1999  | 2004  |
+------------------------------------------------------+-------+-------+
| Social Psychology                                    | 1999  | 2004  |
+------------------------------------------------------+-------+-------+
| Physical Anthropology                                | 1999  | 2004  |
+------------------------------------------------------+-------+-------+
| Geography and Regional Sciences                      | 1999  | 2004  |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| *Social and Economic Sciences (SES)*                 |       | 2003  |
+------------------------------------------------------+-------+-------+
| Decision, Risk, and Management Sciences              | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Political Science                                    | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Law and Social Science                               | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Innovation and Organizational Change                 | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Methodology, Measurement and Statistics              | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Science and Technology Studies                       | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Societal Dimensions of Engineering, Science, and     | 2000  | 2003  |
| Technology                                           |       |       |
+------------------------------------------------------+-------+-------+
| Economics                                            | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Sociology                                            | 2000  | 2003  |
+------------------------------------------------------+-------+-------+

+------------------------------------------------------+-------+-------+
| ##### **DIRECTORATE**                                | F     | F     |
|                                                      | iscal | iscal |
|                                                      |       | Year  |
|                                                      | Year  | of    |
|                                                      | of    |       |
+------------------------------------------------------+-------+-------+
| *Division*                                           | Most  | Next  |
+------------------------------------------------------+-------+-------+
| Program                                              | R     | COV   |
|                                                      | ecent |       |
|                                                      | COV   |       |
+------------------------------------------------------+-------+-------+
| **OFFICE OF POLAR PROGRAMS**                         |       |       |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| ***Polar Research Support***                         | **2   | 2004  |
|                                                      | 001** |       |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| *Antarctic Sciences*                                 |       | 2003  |
+------------------------------------------------------+-------+-------+
| Antarctic Aeronomy and Astrophysics                  | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Antarctic Biology and Medicine                       | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Antarctic Geology and Geophysics                     | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Antarctic Glaciology                                 | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Antarctic Ocean and Climate Systems                  | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| *Arctic Sciences*                                    |       | 2003  |
+------------------------------------------------------+-------+-------+
| Arctic Research Opportunities                        | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Arctic Research and Policy                           | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Arctic System Sciences                               | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Arctic Natural Sciences                              | 2000  | 2003  |
+------------------------------------------------------+-------+-------+
| Arctic Social Sciences                               | 2000  | 2003  |
+------------------------------------------------------+-------+-------+

+------------------------------------------------------+-------+-------+
| ##### **DIRECTORATE**                                | F     | F     |
|                                                      | iscal | iscal |
|                                                      | Year  | Year  |
|                                                      | of    | of    |
+------------------------------------------------------+-------+-------+
| *Division*                                           | Most  | Next  |
+------------------------------------------------------+-------+-------+
| Program                                              | R     | COV   |
|                                                      | ecent |       |
|                                                      | COV   |       |
+------------------------------------------------------+-------+-------+
| ##### **OFFICE OF INTEGRATIVE ACTIVITIES**           |       |       |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| Major Research Instrumentation (MRI)                 | 2     |       |
|                                                      | 000\* |       |
+------------------------------------------------------+-------+-------+
| Science and Technology Centers (STC)                 | 1     | 2007  |
|                                                      | 996\* |       |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+
| \*External evaluations                               |       |       |
+------------------------------------------------------+-------+-------+
|                                                      |       |       |
+------------------------------------------------------+-------+-------+

######### APPENDIX III. -- TABLE OF ACRONYMS

# 

AC Advisory Committee

ACR Advanced Computational Research

ACSI American Customer Satisfaction Index

ADP Automated Data Processing

AGEP Alliances for Graduate Education and the Professoriate

AKRSI Alaska Rural Systemic Initiative

ALMA Atacama Large Millimeter Array

AMANDA Antarctic Muon and Neutrino Detection Array

AMRC Antarctic Meteorological

> Research Center

AMS American Mathematical Society

AO Arctic Oscillation

ARSI Appalachia Rural Systemic

> Initiative

AST Astronomical Sciences Division

ATE Advanced Technological Education

ATLAS A Toroidal LHC Apparatus

BAC Bacterial Artificial Chromosome

BBC British Broadcasting Corporation

BCS Division of Behavioral and Cognitive Sciences

BE Biocomplexity in the Environment

BIO Directorate for Biological Sciences

CAREER Faculty Early Career Development Program

CASES Cooperative Atmosphere-Surface Exchange Study

CAVE Cave Automatic Virtual Environment

CDA Cross-Directorate Activities Program

CDRC Child Development Research Collaborative

CERN European Organization for Nuclear Research

CHE Chemistry Division

CHESS Cornell High Energy Synchrotron Source

CHRNS Center for High Resolution Neutron Scattering

CIRE Collaboratives to Integrate Research and Education

CIS Center for Integrated Studies

CISE Directorate for Computer and Information Science and Engineering

CLAWPACK Conservation Law Package

CLT Centers for Learning and Teaching

CMS Compact Muon Spectrometer

CNRS/INRA Centre National de Recherche Scientifique/Institut National de
la Recherche Agronomique

CompTIA Computing Technology Industry Association

COV Committee of Visitors

CPMSA Comprehensive Partnerships for Mathematics and Science Achievement

CPO Division of Contracts, Policy and Oversight

CREST Centers for Research Excellence In Science and Technology

CRI Children's Research Initiative

CRIF Chemistry Research Instrumentation and Facilities

DARPA Defense Advanced Research Projects Agency

DDI Data Documentation Initiative

DGA Division of Grants and Agreements

DHHS Department of Health and Human Services

DL Digital Libraries

DMR Division of Materials Research

DMS Division of Mathematical Sciences

DNA Deoxyribonucleic Acid

DOD Department of Defense

DOE Department of Energy

DRSI Delta Rural Systemic Initiative

DYCOMS Dynamics and Chemistry of Marine Stratocumulus

EAR Division of Earth Sciences

EHR Directorate for Education and Human Resources

EIA Division of Experimental and Integrative Activities

EIS Enterprise Information System

ENG Directorate for Engineering

EPSCoR Experimental Program to Stimulate Competitive Research

ERC Engineering Research Center

ERI Education Research Initiative

ESR Educational System Reform

EST Expressed Sequence Tag

FAA Federal Aviation Administration

FACA Federal Advisory Committee Act

FAS Financial Accounting System

FEMA Federal Emergency Management Agency

FT-ICR Fourier-Transform-Ion Cyclotron Resonance

GABA Gamma Aminobutyric Acid

GAO General Accounting Office

GEO Directorate for Geosciences

GK-12 Graduate Teaching Fellows in K-12 Education

GPA Grade Point Average

GPRA Government Performance and Results Act

GPS Global Positioning System

GRF Graduate Research Fellowship

GSN Global Seismographic Network

GSS General Social Survey

GW Ground water

HBCU Historically Black Colleges and Universities

HDGC Human Dimensions of Global Change

HPIIS High Performance International Internet Services

HPNC High Performance Network Connections

HRM Division of Human Resources Management

IBN Division of Integrative Biology and Neuroscience

ICPSR Inter-University Consortium for Political and Social Research

IGERT Integrative Graduate Education and Research Traineeship

INT Office of International Science and Engineering

IP Internet Protocol

IPA Intergovernmental Personnel Act (appointee)

IPCC Intergovernmental Panel on Climate Change

IRIS Incorporated Research Institutions for Seismology

IRIS Industrial Research & Development Information System

ISE Informal Science Education

IT Information Technology

ITR Information Technology Research

ITS Information Technology Security

IUCRC Industry University Cooperative Research Center

KeLP Kernel Lattice Parallelism (KeLP)

KHEP K-12 Higher Education Partnerships

LAPACK Linear Algebra Package

LASER Leadership and Assistance for Science Education Reform Center

LHC Large Hadron Collider

LIGO Laser Interferometer Gravitational-wave Observatory

LSAMP Louis Stokes Alliances for Minority Participation

MEMS Microelectromechanical Systems

MPS Directorate for Mathematical and Physical Sciences

MRE Major Research Equipment (account)

MRI Major Research Instrumentation (program)

MRSEC Materials Research Science and Engineering Center

MS Master of Science or Mass Spectrometry

MSP Math and Science Partnerships

NAIC National Astronomy and Ionosphere Center

NAPA National Academy of Public Administration

NASA National Aeronautics and Space Administration

NATO North Atlantic Treaty Organization

NCAR National Center for Atmospheric Research

NCTM National Council of Teachers of Mathematics

NEES Network for Earthquake Engineering Simulation

NHGIS National Historical Geographic Information System

NHMFL National High Magnetic Field Laboratory

NIH National Institutes of Health

NNI National Nanotechnology Initiative

NNUN National Nanofabrication Users Network

NOAA National Oceanic and Atmospheric Administration

NOAO National Optical Astronomy Observatory

NRAO National Radio Astronomy Observatory

NRC National Research Council

NRL Naval Research Laboratory

NSB National Science Board

NSE Nanoscale Science and Engineering

NSEC Nanoscale Science and Engineering Centers

NVO National Virtual Observatory

ODS Online Document System

OEOP Office of Equal Opportunity Programs

OIG Office of Inspector General

OIRM Office of Information and Resource Management

OMB Office of Management and Budget

OPM Office of Personnel Management

OPP Office of Polar Programs

ORISE Oak Ridge Institute for Science and Education

OTS Organization for Tropical Studies

OWC Oklahoma Weather Center

PACI Partnerships for Advanced Computational Infrastructure

PARS Proposal, PI and Reviewer System

PDF Program Document Format

PFI Partnerships for Innovation

PGE Programs for Gender Equity

PHY Division of Physics

PI Principal Investigator

PICI Phyloinformatics and Coordination Infrastructure

PIMS Program Information Management System

PMA President's Management Agenda

PMET Physical Meteorology

PPD Programs for Persons with Disabilities

PSID Panel Study of Income Dynamics

PwC LLP PricewaterhouseCoopers LLP

REPP Research in Education Policy and Practice

RET Research Experiences for Teachers

REU Research Experiences for Undergraduates

ROLE Research on Learning and Education

RSI Rural Systemic Initiative

SAR Synthetic Aperture Radar

SBE Directorate for Social, Behavioral and Economic Sciences

SBIR Small Business Innovation Research

ScaLAPACK Scalable Linear Algebra Package

SES Division of Social and Economic Sciences

SFFAS Statement of Federal Financial Accounting Standard

SGER Small Grant for Exploratory Research

SMET Science, Mathematics, Engineering and Technology

SMETE Science, Mathematics, Engineering and Technology Education

SOARS Significant Opportunities in Atmospheric Research and Science

SPSM South Pole Station Modernization

SRC Synchrotron Radiation Center

SRI SRI International

SRS Division of Science Resources Statistics

SSI Statewide Systemic Initiative

STC Science and Technology Center

STEM Science, Technology, Engineering and Mathematics

TCS Terascale Computing System

TEA Teachers Experiencing Antarctica and the Arctic

UCAN Utah, Colorado, Arizona, New Mexico

UCAR University Corporation for Atmospheric Research

UFE Undergraduate Faculty Enhancement

UML Unified Modeling Language

UPR University of Puerto Rico

URM Underrepresented Minorities

USGS United States Geological Survey

USI Urban Systemic Initiative

USP Urban Systemic Program

VORTEX Verification of Origins of Rotation in Tornadoes Experiment

VR Virtual Reality

WAIS West Antarctic Ice Sheet

WDCP World Data Center for Paleoclimatology

WTC World Trade Center

WTEC World Technology Evaluation Center

# APPENDIX IV. PwC EXECUTIVE SUMMARY

# 

# Excerpt from the PRICEWATERHOUSECOOPERS LLP REPORT *"NATIONAL SCIENCE FOUNDATION FY 2001 GPRA Performance Measurement Validation and Verification Final Report January 2002"*

# 

# 1 Executive Summary

The National Science Foundation ("NSF" or "the Foundation"), as a
Federal agency, is subject to the performance reporting requirements of
the Government Performance and Results Act (GPRA). Accordingly, NSF
developed a series of performance measures to help the agency meet its
mission, goals, and objectives. The Foundation asked
PricewaterhouseCoopers (PwC) to assess whether the methods that NSF uses
to compile and report selected FY 2001 performance measures are
verifiable and produce valid results. This is the second consecutive
year that PwC has performed this function.

We commend NSF for undertaking this second year effort to confirm the
reliability of its data and the processes to collect, process, maintain,
and report this data. From our FY 2001 review, we conclude that NSF has
made a concerted effort to ensure that it reports accurately to the
federal government and has effective systems, policies, and procedures
to ensure data quality. We have noted some areas for improvement,
particularly in the area of data collection for the goals related to
facilities management. However, overall NSF relies on sound business
practices, system and application controls, and manual checks of system
queries to report performance. Further, our efforts to re-calculate the
Foundation's results based on these systems, processes and data were
successful.

The General Accounting Office (GAO) has directed federal agencies to
provide confidence that the policies and procedures that underlie GPRA
performance reporting are complete, accurate and consistent. To address
GAO's mandate and past concerns, NSF asked us to conduct an independent
verification and validation review of eighteen FY 2001 quantitative and
qualitative goals contained in the FY 2001 NSF GPRA Performance Plan.
GAO defines verification as a means to check or test performance data in
order to reduce the risk of using data that contains significant errors.
GAO defines validation as a way to test data to ensure that no error
creates significant bias. Significant error, including bias, would
affect conclusions about the extent to which NSF has achieved its
performance goals. These definitions and the GAO-specified criteria were
the guiding principles of our assessment.

Thirteen of the goals we assessed are undergoing review for the first
time, while the remaining six are being reviewed a second time. As part
of our review of the processes and results for these selected
performance goals, we:

-   Assessed the accuracy of NSF's performance measures

-   Described the reliability of the processes NSF uses to collect,
    process, maintain, and report data

-   Reviewed system controls to confirm that quality input results in
    quality output

-   Identified changes to processes and data for those goals undergoing
    review for the second time

This assessment is not an audit and, as such, was not conducted in
accordance with generally accepted government auditing standards.
Rather, we followed GAO's *Guide to Assessing Agency Annual Performance
Plans* (GAO/GCD-10.1.20) to guide our review. Our assessment was
intended neither to determine whether NSF's goals are appropriate nor to
conclude whether these goals are the appropriate way to gauge agency
success. Based on GAO guidance, we assessed whether NSF's processes to
collect, process, maintain, and report data for its goals meet the
following criteria:

-   Does the process provide for periodic review of collection,
    maintenance, and processing procedures to ensure they are
    consistently applied and continue to be adequate?

-   Does the process provide for periodic sampling and review of data to
    ensure their completeness, accuracy, and consistency?

-   Does the process rely on independent audits or other established
    procedures for verifying and validating financial information when
    performance measures require the use of financial information?

-   Does NSF address problems, in verification and validation
    procedures, known to GAO or the agency?

-   Does the agency recognize the potential impacts of data limitations
    should they exist?

For goals undergoing review for the first time, we documented the
processes NSF follows to collect, process, maintain, and report
performance data. We also identified relevant controls and commented on
their effectiveness.

## **Results**

From our review, we determined that NSF has reported on ten of the
quantitative goals and all five of the qualitative goals in a manner
such that any errors, should they exist, would not be significant enough
to change the reader's interpretation of the Foundation's success in
meeting the supporting performance goal. For these goals, NSF relies on
sound business processes, system and application controls, and manual
checks of system queries to report performance. We believe that these
processes are valid and verifiable. For the four goals related to
facilities management, we identified significant data limitations, which
impaired our ability to verify the processes. However, we believe that
NSF's reported outcomes are consistent with the data they collected. We
summarize our results in the following table:

  ------------------------------------------------------------------------------------------
  **FY 2001 Performance Goal**                         **Are                        
                                                       processes                    
                                                       verifiable                   
                                                       and are                      
                                                       results                      
                                                       valid?**                     
  ---------------------------------------------------- ------------ --------------- --------
  **Quantitative goals reviewed for the first time in  **Yes**      **Partially**   **No**
  FY 2001**                                                                         

  IV-2: In FY 2001, NSF will conduct ten pilot                                     
  paperless projects that manage the competitive                                    
  review process in an electronic environment.                                      

  IV-3: By the end of FY 2001, NSF will increase usage                             
  of a broad-range of video-conferencing/long-distance                              
  communications technology by 100% over the FY 1999                                
  level.                                                                            

  V-1: At least 85% of basic and applied research                                  
  funds will be allocated to projects, which undergo                                
  merit review.                                                                     

  V-6a: NSF will increase the average annualized award                             
  size for research grants to \$110,000.                                            

  V-6b: NSF will increase the average duration of                                  
  awards of research grants to at least three years.                                

  V-9a: For 90 percent of facilities, keep                                         
  construction and upgrades within annual expenditure                               
  plan, not to exceed 110 percent of estimates.                                     

  V-9b: Ninety percent of facilities will meet all                                 
  major annual schedule milestones by the end of the                                
  reporting period.                                                                 

  V-9c: For all construction and upgrade projects                                  
  initiated after 1996, when current planning                                       
  processes were put in place, keep total cost within                               
  110 percent of estimates made at the initiation of                                
  construction.                                                                     

  V-10: For 90 percent of facilities, keep operating                               
  time lost due to unscheduled downtime to less than                                
  10 percent of the total scheduled operating time.                                 

  III-1b: Over 80% of schools participating in                                     
  systematic initiative programs will 1) implement a                                
  standards-based curriculum in science and                                         
  mathematics, 2) further professional development of                               
  the instructional workforce, and 3) improve student                               
  achievement on a selected battery of tests, after                                 
  three years of NSF support.                                                       

  III-1c: Through systematic initiatives and related                               
  teacher enhancement programs, NSF will provide                                    
  intensive professional development experiences for                                
  at least 65,000 pre-college teachers.                                             

  **Update Reviews (Goals initially reviewed in FY                                  
  2000)**                                                                           

  IV-1: Ninety-five percent of full proposals will be                              
  received electronically through FastLane.                                         

  IV-4: NSF will show an increase over 1997 in the                                 
  total number of hires to S&E positions from                                       
  underrepresented groups.                                                          

  V-5: For 70 percent of proposals, be able to tell                                
  applicants whether their proposals have been                                      
  declined or recommended for funding within six                                    
  months of receipt.                                                                

  **Qualitative Goals**                                                             

  III-1: Development of "a diverse, internationally                                
  competitive and globally-engaged workforce of                                     
  scientists, engineers, and well-prepared citizens."                               

  III-2: Enabling "discovery across the frontier of                                
  science and engineering, connected to learning,                                   
  innovation, and service to society."                                              

  III-3: Providing "broadly accessible,                                            
  state-of-the-art and shared research and education                                
  tools."                                                                           

  V-2: NSF performance in implementation of the merit                              
  review criteria is successful when reviewers address                              
  the elements of both generic review criteria.                                     

  V-3: NSF performance in implementation of the merit                              
  review criteria is successful when program officers                               
  address the elements of both generic review criteria                              
  when making their award decisions.                                                
  ------------------------------------------------------------------------------------------

## 

## Our conclusions that the qualitative goals are valid and verifiable are based on our ability to confirm the ratings and interpretations contained in the Advisory Committee (AC) and Committee of Visitors (COV) reports. At the date of this report, we have been unable to review the final language that NSF will use in presenting the Foundation-wide results due to varying external reporting due dates for performance measurement information. However, we expect that the results that will be reported in upcoming months will coincide with the comments and conclusions reported in the AC and COV reports.

In addition, we concluded that there was insufficient information in
many of the COV reports on which to base an unequivocal determination of
success in achieving certain indicators. We recommend that NSF balance
their final performance report language reflecting the neutrality of
these reports and the inability to support the AC and COV report text
with clearly identifiable examples and awards.

## 

## **Recommendations**

For each goal under review, we provide recommendations for how NSF can
strengthen the processes it uses to collect, process, maintain, and
report GPRA information. Details for our recommendations can be found in
the report. We highlight our overarching recommendations below:

-   **For goals that compare actual performance to estimates, ensure
    that estimates are unchangeable.** For the goals related to
    facilities management, NSF designed the data collection system to
    allow principal investigators (PIs) to change the estimates, which
    are used to calculate the results. NSF allows these changes to
    account for management problems beyond the facilities control.
    However, the ability to change estimates and the fact that the
    system does not track these changes hinders the ability to compare
    actual costs, milestones and completion dates to original estimates.
    By making estimates unchangeable, NSF could create true
    project-specific baselines for these goals, which will provide NSF
    an accurate picture of project performance, compared to estimates.
    Should NSF choose to continue to allow estimates to be changed, we
    recommend that the system be enhanced to track estimate changes, as
    a management and monitoring tool for NSF and Program Officers (POs).

-   **Simplify the GPRA reporting process for facilities goals.** NSF
    should consider allowing POs, rather than PIs, to report on the
    progress of facilities projects. By allowing POs to report on
    project performance, NSF can simplify the reporting process, improve
    internal accountability, and lessen the reporting burden on PIs. POs
    could use annual project reports, schedule. or other reports already
    developed by the PI, to report progress on facilities projects for
    GPRA. This would eliminate the need for PIs to provide duplicative
    information and create budgets and schedules based on the federal
    fiscal year yet have little value for program management.

-   **Clarify language for goals to better reflect NSF's objectives and
    thresholds for success.** For some of the goals we reviewed, we
    believe that NSF can revise the language to be more specific and
    indicative of what NSF is trying to achieve. For example, NSF could
    clarify the language for goal III-1b to indicate that the goal only
    measures schools participating for three years or more in the
    systemic initiative program. Also, for construction and upgrade
    goals, NSF could revise the language to mention that it only
    measures construction and upgrade projects that have a total cost of
    at least \$5 million or funded out of the Major Research and
    Equipment Account. Clarifying the language of these goals will help
    NSF staff and external reviewers understand NSF's objectives and
    facilitate the process to collect, process, maintain, and report
    data.

-   **Further refine reporting templates and instructions for the
    qualitative measures.** NSF has made great strides to develop and
    improve the templates that are provided to the committees. However,
    we believe that this improvement can continue to evolve. A
    well-designed template will save committees valuable time, provide
    more verifiable support, reduce ambiguity, and provide more
    comprehensive evaluations. Committees could be encouraged to provide
    more than one example, if desired. A sample template for the
    "People" goals at a Division level is provided below.

-   (Sample not included in this excerpt).

About the

National Science Foundation

The National Science Foundation (NSF) funds research and education in
most fields of science and engineering. Awardees are wholly responsible
for conducting their project activities and preparing the results for
publication. Thus, the Foundation does not assume responsibility for
such findings or their interpretation.

NSF welcomes proposals from all qualified scientists, engineers and
educators. The Foundation strongly encourages women, minorities and
persons with disabilities to compete fully in its programs. In
accordance with Federal statutes, regulations and NSF policies, no
person on grounds of race, color, age, sex, national origin or
disability shall be excluded from participation in, be denied the
benefits of, or be subjected to discrimination under any program or
activity receiving financial assistance from NSF (unless otherwise
specified in the eligibility requirements for a particular program).

Facilitation Awards for Scientists and Engineers with Disabilities
(FASED) provide funding for special assistance or equipment to enable
persons with disabilities (investigators and other staff, including
student research assistants) to work on NSF-supported projects. See the
program announcement/ solicitation for further information.

The National Science Foundation has Telephonic Device for the Deaf (TDD)
and Federal Information Relay Service (FIRS) capabilities that enable
individuals with hearing impairments to communicate with the Foundation
about NSF programs, employment or general information. TDD may be
accessed at (703) 292-5090, FIRS at 1-800-877-8339.

The National Science Foundation is committed to making all of the
information we publish easy to understand. If you have a suggestion
about how to improve the clarity of this document or other NSF published
materials, please contact us at plainlanguage@nsf.gov.

[National Science Foundation]{.smallcaps}

[4201 Wilson Boulevard]{.smallcaps}

[Arlington, Virginia 22230]{.smallcaps}

![](media/image4.wmf)

[^1]: Source: NSF FY 2003 Budget Request to Congress, p. 43.

[^2]: George A. Akerlof, Eric A. Cornell , Wolfgang Ketterle, William S.
    Knowles, K. Barry Sharpless, A. Michael Spence, Joseph E. Stiglitz,
    Carl E. Wieman.

[^3]: <http://www.scientificamerican.com/explorations/2001/051401top50/#ScientificAmerican>

[^4]: The first five of these elements are provided with each goal
    discussed in our report. The last element is discussed in Section
    VIII.

[^5]: Information on program evaluations is given in Appendices I. and
    II. Quality of reported performance information is discussed in
    Section VIII. The other features are discussed in Section XI.

[^6]: In our report one will find the acronyms SMET (Science,
    Mathematics, Engineering, and Technology) and STEM (Science,
    Technology, Engineering and Mathematics). The two acronyms are
    equivalent. External experts used one or the other of these terms.

[^7]: "*Distributed Mentor Project: Comprehensive Participant Survey
    Analyses for 1994-1999*", The LEAD Center, University of
    Wisconsin-Madison, Preliminary Report, March 2000.

[^8]: "*Baccalaureate and Beyond; Longitudinal Study*", National Center
    for Educational Statistics, 1994.

[^9]: [http://www.nsf.gov/pubs/2002/nsf02072/nsf02072.pdf](../%20http://www.nsf.gov/pubs/2002/nsf02072/nsf02072.pdf),
    <http://www.nsf.gov/od/lpa/news/press/00/pr0084.htm>

[^10]: <http://www.nsf.gov/od/lpa/news/press/01/pr0188.htm#attachment>

[^11]: The auditing firm of PricewaterhouseCoopers LLP (PwC) reviewed
    the data collection, maintenance, processing, and reporting
    procedures used to calculate results for this goal. They concluded
    that the procedures related to this goal were sufficient and
    adequate and yielded valid results. We provide the Executive Summary
    of their entire report, as well as a table listing their conclusions
    as to whether the processes we used for selected goals were
    verifiable and the results valid, in Appendix IV.

[^12]: [http://www.lrsm.upenn.edu/lrsm/outr.html#CIRE](http://www.lrsm.upenn.edu/lrsm/outr.html#CIRE))

[^13]: <http://www.nsf.gov/mps/divisions/dmr/research/c_reusites.htm>

[^14]: Speech to the Boston College Conference on the New Economy, March
    6, 2000

[^15]: The auditing firm of PricewaterhouseCoopers LLP (PwC) reviewed
    the data collection, maintenance, processing, and reporting
    procedures used to calculate results for this goal. They concluded
    that the procedures related to this goal were sufficient and
    adequate and yielded valid results. We provide the Executive Summary
    of their entire report, as well as a table listing their conclusions
    as to whether the processes we used for selected goals were
    verifiable and the results valid, in Appendix IV.

[^16]: http://www.nsf.gov/pubs/2000/iin126/iin126.htm.

[^17]: The auditing firm of PricewaterhouseCoopers LLP (PwC) reviewed
    the data collection, maintenance, processing, and reporting
    procedures used to calculate results for this goal. They concluded
    that the procedures related to this goal were sufficient and
    adequate and yielded valid results. We provide the Executive Summary
    of their entire report, as well as a table listing their conclusions
    as to whether the processes we used for selected goals were
    verifiable and the results valid, in Appendix IV.

[^18]: The auditing firm of PricewaterhouseCoopers LLP (PwC) reviewed
    the data collection, maintenance, processing, and reporting
    procedures used to calculate results for this goal. They concluded
    that the procedures related to this goal were sufficient and
    adequate and yielded valid results. We provide the Executive Summary
    of their entire report, as well as a table listing their conclusions
    as to whether the processes we used for selected goals were
    verifiable and the results valid, in Appendix IV.

[^19]: In FY 1999, our goal was "In FY 1999, as all appointments for
    scientists and engineers are considered, the recruiting organization
    will demonstrate efforts to attract applications from groups that
    are underrepresented in the science and engineering staff as
    compared to their representation among Ph.D. holders in their
    fields."

[^20]: Includes 1 Female hired by OPP. FY 2001 is the first time OPP
    data is included.

[^21]: The auditing firm of PricewaterhouseCoopers LLP (PwC) reviewed
    the data collection, maintenance, processing, and reporting
    procedures used to calculate results for this goal. They concluded
    that the procedures related to this goal were sufficient and
    adequate and yielded valid results. We provide the Executive Summary
    of their entire report, as well as a table listing their conclusions
    as to whether the processes we used for selected goals were
    verifiable and the results valid, in Appendix IV.

[^22]: *"Merit-reviewed scientific research with competitive selection
    and external (peer) evaluation. Intramural and extramural research
    programs where funded activities are competitively awarded from a
    pool of qualified applicants following review by a set of external
    scientific or technical reviewers (often called peers) for merit.
    The review is conducted by appropriately qualified scientists,
    engineers, or other technically-qualified individuals who are apart
    from the people or groups making the award decisions, and serves to
    inform the program manager or other qualified individual who makes
    the award."*

[^23]: NSF's original definition included both merit-reviewed projects
    with competitive selection and external evaluation and projects with
    limited competitive selection as a percentage of all NSF funding.
    The revised OMB merit-review definition as of FY 2000 does not
    include funds for merit-reviewed scientific research with limited
    competitive selection (*e.g*., applicants that are limited to
    organizations that were created to largely serve Federal missions,
    such as Federally-Funded Research and Development Centers
    \[FFRDCs\]). It also does not include merit-reviewed scientific
    research with competitive selection and internal evaluation (for
    example, reviews conducted from within the agency program, without
    additional independent evaluation, such as NSF's Small Grants for
    Exploratory Research \[SGERs\]). The revised definition measures
    merit-reviewed research as a percentage of basic and applied
    research funds rather than as a percentage of all NSF funding.

[^24]: In FY 1999 and FY 2000 the goal required that both reviewers
    *and* program officers use both criteria in order for NSF to be
    successful. In FY 2001 this goal was separated into two distinct
    goals.

[^25]: The auditing firm of PricewaterhouseCoopers LLP (PwC) reviewed
    the data collection, maintenance, processing, and reporting
    procedures used to calculate results for this goal. They concluded
    that the procedures related to this goal were sufficient and
    adequate and yielded valid results. We provide the Executive Summary
    of their entire report, as well as a table listing their conclusions
    as to whether the processes we used for selected goals were
    verifiable and the results valid, in Appendix IV.

[^26]: In FY 1999 and FY 2000 the goal required that both reviewers
    *and* program officers use both criteria in order for NSF to be
    successful. In FY 2001 this goal was separated into two distinct
    goals.

[^27]: A number of continuing programs have standing or previously
    established deadline dates. Some of these programs reissue
    announcements within 90 days of a proposal due date. As long as that
    deadline date was previously announced, thereby providing the
    community with at least 90 days to prepare a proposal, the
    announcement is considered to be in compliance with this GPRA goal.

[^28]: The award size and duration performance goals are applicable only
    to competitive research grants (a subset of awards that focuses on
    awards to individual investigators and small groups).

[^29]: The auditing firm of PricewaterhouseCoopers LLP (PwC) reviewed
    the data collection, maintenance, processing, and reporting
    procedures used to calculate results for this goal. They concluded
    that the procedures related to this goal were sufficient and
    adequate and yielded valid results. We provide the Executive Summary
    of their entire report, as well as a table listing their conclusions
    as to whether the processes we used for selected goals were
    verifiable and the results valid, in Appendix IV.

[^30]: The award size and duration performance goals are applicable only
    to competitive research grants (a subset of awards that focuses on
    awards to individual investigators and small groups).

[^31]: The auditing firm of PricewaterhouseCoopers LLP (PwC) reviewed
    the data collection, maintenance, processing, and reporting
    procedures used in this goal. They concluded that the procedures
    related to this goal were sufficient and adequate and yielded valid
    results. We provide the Executive Summary of their entire report, as
    well as a table listing their conclusions as to whether the
    processes we used for selected goals we report were verifiable and
    the results valid, in Appendix IV.

[^32]: In their report of January 2002, the auditing firm of
    PricewaterhouseCoopers LLP (PwC) stated: "For the four goals related
    to facilities management, we identified significant data
    limitations, which impaired our ability to verify the processes.
    However, we believe that NSF's reported outcomes are consistent with
    the data they collected." We are in the process of refining the data
    collection procedures for FY 2002.

[^33]: In their report of January 2002, the auditing firm of
    PricewaterhouseCoopers LLP (PwC) stated: "For the four goals related
    to facilities management, we identified significant data
    limitations, which impaired our ability to verify the processes.
    However, we believe that NSF's reported outcomes are consistent with
    the data they collected." We are in the process of refining the data
    collection procedures for FY 2002.

[^34]: In their report of January 2002, the auditing firm of
    PricewaterhouseCoopers LLP (PwC) stated: "For the four goals related
    to facilities management, we identified significant data
    limitations, which impaired our ability to verify the processes.
    However, we believe that NSF's reported outcomes are consistent with
    the data they collected." We are in the process of refining the data
    collection procedures for FY 2002.

[^35]: In their report of January 2002, the auditing firm of
    PricewaterhouseCoopers LLP (PwC) stated: "For the four goals related
    to facilities management, we identified significant data
    limitations, which impaired our ability to verify the processes.
    However, we believe that NSF's reported outcomes are consistent with
    the data they collected." We are in the process of refining the data
    collection procedures for FY 2002.

[^36]: Enhancing the Diversity of the Science and Engineering Workforce
    to Sustain America's Leadership in the 21^st^ Century, Committee on
    Equal Opportunities in Science and Engineering, 2000 Biennial Report
    to the United States Congress.