The Honorable Arden L. Bement, Jr.

Director

National Science Foundation

4201 Wilson Boulevard

Arlington, VA 22230

Dear Dr. Bement:

We are pleased to submit the report of the Advisory Committee for GPRA
Performance Assessment (AC/GPA) for 2005. It was the unanimous judgment
of the Committee that NSF has demonstrated significant achievement for
all indicators in the Ideas and Tools goals and also for the merit
review indicator of the Organizational Excellence outcome goal. The
Committee concluded that NSF demonstrated significant achievement for
indicators P1-P4 of the People goal, but has not done so with respect to
P5. The Advisory Committee on Business and Operations concluded that NSF
demonstrated significant achievement for the other indicators of the
Organizational Excellence goal.

The Committee also concluded that the four outcome goals are mutually
reinforcing and synergistic. They represent an integrated framework that
combines research and education in a positive way and also provides the
organizational infrastructure to advance the national scientific,
technological, engineering, and mathematics enterprise. Thus, all four
goals should always be considered as an integrated whole when assessing
NSF's performance.

The Committee appreciates the continued improvement in the AC/GPA
process this year. Not only were we again able to do much of our work in
advance of the meeting, but NSF also provided periodic updates on the
progress of each subgroup in its analyses of indicators. This Committee,
with NSF's invaluable assistance, has, we believe, modeled the very
organizational excellence behavior that we seek from NSF. We also note
that this committee is, perhaps, unique within the Federal government.
We commend you and your predecessors for both foresight and commitment
to an assessment process that is both thorough and impactful.

This report represents the collective work of a large group of
individuals, the members of the Committee, all of whom worked with a
level of commitment and diligence that we have rarely encountered. Each
of them made significant contributions to the report and collectively we
believe they have demonstrated that advisory committees can themselves
demonstrate organizational excellence and become "learning committees."
NSF is indeed fortunate to have such people in its "corner" and it was
an honor and a privilege for us to lead this effort. In addition, many
members of the NSF staff, especially Marilyn Suiter and Craig Robinson,
were instrumental in enabling our work and we are truly grateful for
their assistance.

We would be happy to talk with you or others about any aspect of this
report. We hope it will be helpful to NSF as it completes its
Performance and Accountability Report.

Sincerely,

# Norine E. Noonan Carolyn W. Meyers

> Norine E. Noonan, Ph.D. Carolyn W. Meyers, Ph.D.
>
> Dean Provost and Vice Chancellor for
>
> School of Science and Mathematics Academic Affairs
>
> College of Charleston North Carolina A&T University
>
> Chair, AC/GPA Vice-Chair, AC/GPA

#  

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# REPORT OF THE ADVISORY COMMITTEE FOR GPRA PERFORMANCE ASSESSMENT

# 

# 

# 

# 

# 

# 

# 

# 

# 

# Submitted: July 20, 2005

# Norine E. Noonan, Ph.D.

######### Chairman

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

**Table of Contents**

> **Page**

-   **List of Committee Members 5**

-   **NSF Strategic Outcome Goals 8**

-   **Overall Summary and Conclusions 11**

-   **Approach and Methodologies Used by the AC/GPA 13**

-   **Foundation Level Summary of FY 2005 Investments 15**

-   **Detailed Assessments of Outcome Goals:**

```{=html}
<!-- -->
```
-   **People 18**

-   **Ideas 26**

-   **Tools 36**

-   **Organizational Excellence 47**

```{=html}
<!-- -->
```
-   **Comments on the Process and the Committee's Work 57**

```{=html}
<!-- -->
```
-   **Appendices**

#  

National Science Foundation

**Advisory Committee for GPRA Performance Assessment (AC/GPA)**

\* Absent on June 17, 2005

\*\* Absent on June 16 and 17, 2005

**Dr. Norine E. Noonan (Chair)**

Dean, School of Sciences and Mathematics

College of Charleston

Charleston, South Carolina

**Dr. Carolyn W. Meyers (Vice-Chair)**

Provost and Vice Chancellor for Academic Affairs

North Carolina A&T State University

Greensboro, North Carolina

# Dawn Adrian Adams

President and Founder

Tapestry Institute

Harrison, Nebraska

**Lisa Alvarez-Cohen\***

Department of Civil & Environmental Engineering\
University of California, Berkeley

Berkeley, California

**Shenda Baker**

Professor

Department of Chemistry

Harvey Mudd College

Claremont, California

# Cecilia Conrad\*\*

Stedman-Sumner Professor of Economics

Department of Economics

Pomona College

Claremont, California

**Susan E. Cozzens**

Professor and Director, Technology Policy and Assessment Center

School of Public Policy

Georgia Institute of Technology

Atlanta, Georgia

# David J. Farber\*\*

Distinguished Career Professor of

Computer Science and Public Policy

Carnegie Mellon University

Pittsburgh, Pennsylvania

**Irwin Feller\*\***

Senior Visiting Scientist

American Association for the Advancement of Science

Professor Emeritus of Economics

Pennsylvania State University

State College, Pennsylvania

**Jack Fellows**

Vice President of Corporate Affairs

and Director, Office of Programs

University Corporation for Atmospheric Research

Boulder, Colorado

**Howard A. Gordon**

Associate Chair

Physics Department

Brookhaven National Laboratory

Upton, New York

**Robert C. Harriss**

Director, Environment and Societal Impacts Group

National Center for Atmospheric Research

Boulder, Colorado

# Lynn Jelinski

President

Sunshine Consultants, International

Green Cove Springs, Florida

# Jon Kettenring

Telcordia Technologies (retired)

Drew University

R.I.S.E. Hall of Sciences

Madison, New Jersey

**Sally K. Mason**

Provost

Purdue University

West Lafayette, Indiana

**Joseph O'Rourke**

Olin Professor and Chair,

Department of Computer Science

Smith College

Northampton, Massachusetts

**Marilyn Raphael**

Associate Professor

Department of Geography

University of California, Los Angeles

Los Angeles, California

# Gloria M. Rogers

Director, Research and Assessment

ABET

Baltimore, Maryland

# Eric Sheppard

Dean, School of Engineering

Hampton University

Hampton, Virginia

**Harry Shipman**

Annie Jump Cannon Professor of Astronomy

Department of Physics & Astronomy

University of Delaware

Newark, Delaware

# David Spencer

Chief Executive Officer

wTe Corporation

Bedford, Massachusetts

# Susan G. Stafford\*\*

Dean, College of Natural Resources

University of Minnesota, Twin Cities

St. Paul, Minnesota

# Richard Thompson

Dean of Mathematics and Sciences

The College of Saint Rose

Albany, New York

# Timothy W. Tong

Dean, School of Engineering and Applied Science

The George Washington University

Washington, District of Columbia

NSF Strategic Plan, FY 2003 -- 2008

Strategic Outcome Goals

# PEOPLE GOAL

A DIVERSE, COMPETITIVE, AND GLOBALLY-ENGAGED U.S. WORKFORCE OF
SCIENTISTS, ENGINEERS, TECHNOLOGISTS AND WELL-PREPARED CITIZENS

# Goal Indicators

P1: Promote greater diversity in the science and engineering workforce
through increased participation of underrepresented groups and
institutions in all NSF programs and activities.

P2: Support programs that attract and prepare U.S. students to be highly
qualified members of the global S&E workforce, including providing
opportunities for international study, collaborations and partnerships.

P3: Develop the Nation's capability to provide K-12 and higher education
faculty with opportunities for continuous learning and career
development in science, technology, engineering and mathematics.

P4: Promote public understanding and appreciation of science,
technology, engineering, and mathematics, and build bridges between
formal and informal science education.

P5: Support innovative research on learning, teaching and mentoring that
provides a scientific basis for improving science, technology,
engineering and mathematics education at all levels.

**IDEAS GOAL**

DISCOVERY ACROSS THE FRONTIER OF SCIENCE AND ENGINEERING, CONNECTED TO
LEARNING, INNOVATION, AND SERVICE TO SOCIETY

**Goal Indicators**

I1: Enable people who work at the forefront of discovery to make
important and significant contributions to science and engineering
knowledge.

I2: Encourage collaborative research and education efforts -- across
organizations, disciplines, sectors and international boundaries.

I3: Foster connections between discoveries and their use in the service
of society.

I4: Increase opportunities for underrepresented individuals and
institutions to conduct high quality, competitive research and education
activities.

I5: Provide leadership in identifying and developing new research and
education opportunities within and across S&E fields.

I6: Accelerate progress in selected S&E areas of high priority by
creating new integrative and cross-disciplinary knowledge and tools, and
by providing people with new skills and perspectives.

# TOOLS GOAL

BROADLY ACCESSIBLE, STATE-OF-THE-ART S&E FACILITIES, TOOLS AND OTHER
INFRASTRUCTURE THAT ENABLE DISCOVERY, LEARNING AND INNOVATION

# Goal Indicators

T1: Expand opportunities for U.S. researchers, educators, and students
at all levels to access state-of-the-art S&E facilities, tools,
databases, and other infrastructure.

T2: Provide leadership in the development, construction, and operation
of major, next-generation facilities and other large research and
education platforms.

T3: Develop and deploy an advanced cyberinfrastructure to enable all
fields of science and engineering to fully utilize state-of-the-art
computation.

T4: Provide for the collection and analysis of the scientific and
technical resources of the U.S. and other nations to inform policy
formulation and resource allocation.

T5: Support research that advances instrument technology and leads to
the development of next-generation research and education tools.

# ORGANIZATIONAL EXCELLENCE GOAL 

[AN AGILE, INNOVATIVE ORGANIZATION THAT FULFILLS ITS MISSION THROUGH
LEADERSHIP IN STATE-OF-THE-ART BUSINESS PRACTICES]{.smallcaps}

Excellence in managing NSF's activities is an objective on par with the
Foundation's mission-oriented outcome goals. It is critical to
achievement of all NSF goals. In addition, this goal addresses the
President's Management Agenda and focuses on management challenges and
reforms identified by OMB or the General Accountability Office, in NSF's
annual review of financial and administrative systems as required by the
Federal Managers' Financial Integrity Act, or by the NSF Office of
Inspector General.

## *Investment Categories: The following long-term investment categories directly link to NSF programs and budget resources.* 

-   **Human Capital:** Investments that produce a diverse, agile,
    results-oriented cadre of NSF knowledge workers committed to
    enabling the agency's mission and to constantly expanding their
    abilities to shape the agency's future.

-   **Business Processes**: Investments that produce effective,
    efficient, strategically aligned business processes that integrate
    and capitalize on the agency's human capital and technology
    resources.

-   **Technologies and Tools**: Investments that produce flexible,
    reliable, state-of-the-art business tools and technologies designed
    to support the agency's mission, business processes, and customers.

**Objectives:** Excellence in managing the agency's activities underpins
all of NSF's goals. The following objectives are especially critical to
NSF\'s goal achievement.

-   **Operate a credible, efficient merit review system.** NSF's merit
    review process is the keystone for award selection, through which
    NSF achieves its goals. All proposals for research and education
    projects are evaluated using two criteria: the intellectual merit of
    the proposed activity and its broader impacts. Specifically
    addressed in these criteria are the creativity and originality of
    the idea, the development of human resources, and the potential
    impact on the research and education infrastructure. Ensuring a
    credible, efficient system requires constant attention and openness
    to change.

-   **Utilize and sustain broad access to new and emerging technologies
    for business application.** NSF has moved aggressively to adopt new
    technologies in our business processes. NSF must sustain and further
    develop exemplary mechanisms to streamline business interactions,
    enhance organizational productivity, ensure accessibility to a
    broadened group of participants, and maintain financial integrity
    and internal controls.

-   **Develop a diverse, capable, motivated staff that operates with
    efficiency and integrity.** NSF is dependent on the capability and
    integrity of its staff. Innovative methods of recruitment,
    development, and retention and employee recognition are needed to
    meet future challenges.

-   **Develop and use performance assessment tools and measures to
    provide an environment of continuous improvement in NSF's
    intellectual investments as well as its management effectiveness.**
    An organization that is dependent on public funds must be
    accountable to the public. The development and use of effective
    indicators of agency performance \-- measuring NSF\'s ability to
    meet mission-oriented goals, its competent use of resources in the
    investment process, and its efficiency and effectiveness as a
    reliable partner to others \-- are needed to better explain the
    agency\'s role to the public.

# Overall Summary and Conclusions

The Advisory Committee for GPRA Performance Assessment (AC/GPA) met on
June 16-17, 2005 at the National Science Foundation (NSF) in Arlington,
Virginia, to consider the activities and achievements of NSF relative to
its Government Performance and Results Act (GPRA) performance goals for
FY 2005. The charge to the Committee asked that it provide:

-   An assessment of results for indicators associated with the
    strategic outcome goals of People, Ideas, Tools, and with the merit
    review indicator for the Organizational Excellence goal. (The other
    three indicators for this goal were assessed by the Advisory
    Committee on Business and Operations -- see below under Approach and
    Methodologies Used by the AC/GPA).

-   Comments on the quality and relevance of award portfolios.

-   Comments on transformative/bold/innovative-high risk research and
    education.

The Committee reviewed voluminous materials from NSF's award portfolio
both prior to and during the meeting. In addition, the Committee had
electronic access to supporting documentation for all indicators
including a large database of accomplishments (which NSF terms
"nuggets"), annual and final project reports and an extensive set of
reports from various Committees of Visitors (COVs). The Committee also
received input on the Organizational Excellence (OE) goal from the
Advisory Committee for Business and Operations (AC/B&O). The AC/GPA
reviewed materials supporting an assessment of the merit review
indicator of the OE goal. The group conducted extensive discussions on
the indicators for NSF's four strategic outcome goals. ***[It was the
unanimous judgment of the Committee that NSF has demonstrated
significant achievement for all indicators in the Ideas and Tools goals
and also for the merit review indicator of the Organizational Excellence
outcome goal. The Committee concluded that NSF demonstrated significant
achievement for indicators P1-P4 of the People goal, but has not done so
with respect to P5. The Advisory Committee on Business and Operations
concluded that NSF demonstrated significant achievement for the other
indicators of the Organizational Excellence goal.]{.underline}***

The Committee reiterates that all of the strategic outcome goals are
mutually reinforcing and synergistic. They represent an integrated
framework that combines research and education and also provides the
organizational structure to advance the national scientific,
technological, engineering, and mathematics enterprise. The extensive
documentation that the Committee reviewed also underscores this
interdependence. The Committee also notes that there is great vitality
at the junction of disciplines (often multiple disciplines) and that
these junctions offer great opportunities for pushing the frontiers of
knowledge. However, these advances are inevitably built on a base of
disciplinary strength, funded through the "core" of discipline-based NSF
programs \-- even though, over time, the discoveries at the
intersections redefine what a "discipline" is. This is the natural
evolution of science and engineering research. Moreover, breakthrough
discoveries are only possible when enabled by talented people and
capable tools -- all supported by innovative, fair and effective
business processes to make funding decisions. These decisions must then
periodically be assessed for their quality and relevance. Thus, the four
goals should always be considered as an integrated whole when assessing
NSF's overall performance.

NSF's portfolio of accomplishments for the PEOPLE outcome goal
encompasses a wide variety of activities that are intellectually strong
and encompass multiple approaches, paradigms and methods. The portfolio
contains important examples of programs that are designed to enable
students, educators and researchers from a variety of backgrounds and
experiences to explore the challenges of science, technology,
engineering, and mathematics (STEM) related fields. NSF accomplishments
in the IDEAS outcome goal have both advanced the frontiers of discovery
and nurtured productive collaborations across disciplines, each of which
hold great promise for addressing important current and future societal
concerns. NSF accomplishments in the TOOLS outcome goal have expanded
access to and availability of data and materials, and have enabled the
capacity for discovery by scientists, engineers and educators. NSF's
accomplishments in the ORGANIZATIONAL EXCELLENCE goal demonstrate
innovation in business processes; in methods of recruitment,
development, retention, and recognition of its staff; attention to
continuous improvement in management effectiveness; and a strong
commitment to continued improvements in its merit review process. Taken
together, the strategic outcome goals demonstrate excellence, relevance
and leadership. The nation can be both proud and confident of its
investment in these activities.

This report is organized as follows:

-   A Foundation level summary of FY 2005 investments, including comment
    on the R&D investment criteria of quality and relevance, and
    comments on NSF's portfolio of transformative/bold/innovative-high
    risk research awards.

-   Information on the approach and methodologies used by the Committee.

-   Detailed assessment of the People outcome goal.

-   Detailed assessment of the Ideas outcome goal.

-   Detailed assessment of the Tools outcome goal.

-   Detailed assessment of the Merit Review Indicator of the
    Organizational Excellence outcome goal with a meta-assessment of the
    other three OE indicators.

-   Comments on the ACGPA process and the Committee's work.

The Committee would like to extend its deep gratitude to the NSF GPRA
staff, particularly Marilyn Suiter, Craig Robinson, Patricia Tsuchitani,
Eve Barak, Blane Dahl, Connie Della-Piana, Michael Sieverts, Jennie
Moehlmann and Kelli Savia (student intern) for their excellent support.
Our work (and this report) would simply not have been possible without
their dedication and careful attention to both the "big picture" and the
smallest details and their grace under pressure. We want to especially
thank Peggy Gartner and Teresa Rinehart, Betty Wong, and Tyler Higgins
for developing, refining, and improving the outstanding database for
accomplishments and the website. We would also like to thank Joan Miller
for her cheerful and competent administrative support before and during
the meeting. Lastly, we thank the NSF program staff for their thoughtful
reporting of accomplishments of their program portfolios, and NSF's
senior leadership for their commitment to this effort.

**APPROACH AND METHODOLOGIES USED BY THE AC/GPA**

The Advisory Committee for GPRA Performance Assessment (AC/GPA) is
comprised of 25 members representing the nation's scientific and
engineering research and education communities in the public and private
sectors. About half the AC/GPA membership is drawn from existing
directorate or office advisory committees and about half are "at-large"
members. The membership reflects a broad cross section of talent,
expertise, and experience. Its purpose is to provide expert advice and
recommendations to NSF regarding the Foundation's performance under the
Government Performance and Results Act (GPRA) of 1993. The findings and
recommendations of the Committee will provide valuable input to NSF's
annual GPRA Performance and Accountability Report.

The focus of the AC/GPA is on the activities and results associated with
the indicators and emphasis areas of NSF's four strategic outcome goals:
PEOPLE, IDEAS, TOOLS, and ORGANIZATIONAL EXCELLENCE (OE). The principal
work of the Committee was conducted as a "committee of the whole." Three
subgroups (PEOPLE, IDEAS, and TOOLS) composed of AC/GPA members provided
the detailed analysis of the results associated with each indicator. A
fourth subgroup for OE was comprised of the AC/GPA Vice Chairman and two
other committee members, one of whom chaired this subgroup. All of the
subgroups reported their analyses, findings and conclusions to the full
AC/GPA for its discussion. Within the subgroups, each AC/GPA member was
assigned specific indicators to review on the basis of a large volume of
accomplishments provided by the NSF staff. It should be noted that these
accomplishments and examples were provided in a "bottoms up" fashion by
the Directorates and were not subject to any selective process by NSF
GPRA staff prior to the Committee's review. Thus, although illustrative
of the range of NSF's activities, these accomplishments/examples did not
constitute a strictly statistically "representative" sample (i.e., every
program did not necessarily provide accomplishments/examples and the
total numbers were not weighted in any way). The Committee has made
recommendations to NSF in previous reports on this issue and NSF has
endeavored to assure that the largest 30 programs were represented in
the nugget database.

Materials were available to Committee members via a secure web site
where information was accessible and much of it was electronically
linked to the source documentation three months before the annual
meeting. Thus, the Committee members were able to do virtually all of
the indicator analysis well ahead of the meeting. This provided the
opportunity to enrich and enlarge the discussion at the meeting and to
focus on the evaluation of the entire portfolio. The subgroups (and the
full Committee) had a large amount of material available for its work,
including:

-   A large database (nearly 900) of accomplishments (also known as
    "nuggets").

-   Committee of Visitors reports.

-   A database containing the universe of annual and final project
    reports.

-   NSF's Strategic Plan.

-   NSF's Budget Requests to Congress.

-   Relevant National Science Board reports.

-   Special analyses conducted by NSF staff in response to specific
    questions raised previously (e.g., fate of resubmitted proposals).

The subgroups consolidated their respective preliminary analyses,
indicator-by-indicator, into a draft report for discussion of and
consideration by the full AC/GPA committee. Similarly, overall portfolio
assessments from each subgroup were shared with the entire AC/GPA for
discussion. Comments and amendments from any member of the full
Committee were then included in these outcome goal "chapters" and
subsequently in the final draft report. The final draft was then
distributed electronically to each committee member for review and
concurrence.

[Assessment of the Organizational Excellence (OE) Strategic Outcome
Goal]{.underline}

Organizational Excellence (OE) is a specific NSF strategic outcome goal.
We appreciate that NSF has included this goal at the urging of the
Advisory Committee for Business and Operations (AC/B&O) since it is an
important enabling goal for the outcome goals of People, Ideas, and
Tools.

The AC/GPA recommended in its FY2003 report that NSF should consider an
approach that involved a significant component of "self study." We
envisioned that this would involve a greater number of NSF staff, would
be based on NSF's strategic goals and indicators, would be data driven
and would provide key information at multiple levels of detail. NSF
adopted this approach for the Organizational Excellence goal. Early on,
it was determined that the AC/B&O would provide an assessment of three
of the indicators for the OE goal, Human Capital, Technology-Enabled
Business Processes, and Performance Assessment. The AC/GPA would conduct
an assessment of the Merit Review indicator.

Following a discussion with the AC/B&O in late March, an assessment of
the three indicators was prepared by NSF staff and shared with the
AC/B&O for review and comment. Subsequently, the AC/B&O held a
conference call to discuss the draft assessment, the draft was revised
based on AC/B&O comments, and a letter was transmitted to NSF with the
results of the AC/B&O deliberations. The AC/B&O supported NSF's
determination that the agency had demonstrated significant achievement
for the three indicators it considered. The AC/B&O also made a number of
comments to improve the approach, methodology and analysis for the
assessment of performance in subsequent years. The letter and the
revised assessment are found in an appendix to this report (see Appendix
1). The OE subgroup of the AC/GPA reviewed the letter and the assessment
and performed its own review of the merit review indicator. The results
of this analysis were presented to the full AC/GPA for its
consideration.

**FOUNDATION-LEVEL SUMMARY OF NSF FY 2005 INVESTMENTS**

**The Committee was asked to provide Foundation-level comments on:**

-   **The R&D criteria of quality and relevance of the award portfolio
    as reflected by the accomplishments reported in FY 2005.**

-   **Transformative/bold/innovative-high risk research and education in
    NSF's portfolio of accomplishments reported in FY 2005.**

As noted previously, the Committee relied on numerous and varied sources
of information to do its work. In addition, because members of the
Committee, both individually and collectively, possess deep familiarity
with various aspects of NSF's portfolio, the Committee could complement
these data sources with its own expertise and experience in crafting
this independent assessment.

[Quality and Relevance]{.underline}

The Committee concluded that the quality of the NSF portfolio was high
in the three outcome goals of People, Ideas, and Tools and that the
Organizational Excellence goal demonstrated quality and innovativeness
in its activities. The diversity of projects in the research portfolio
is remarkable, representing a spectrum of approaches, methods, ideas,
and award types. This diversity enables NSF to support a wide variety of
performers including individuals, teams of all sizes, and large centers
as well as facilities and other infrastructure (defined broadly).

NSF continues to make important contributions toward the achievement of
key national goals. It also provides important service to its
constituents in the scientific community as well as serving the broader
needs of science, engineering and education as human endeavors. In
addition, NSF is recognized as a high-performing organization. Its focus
on organizational excellence as a strategic outcome goal is a necessary
complement to the other goals and will enable NSF to continue to use the
nation's investments wisely and efficiently in support of science,
engineering, mathematics, and education.

The Committee wants to reiterate again that the synergy of the four
outcome goals is a major source of their power. Discoveries at the
frontiers of knowledge are both supportive of and dependent on progress
in effectively linking education and research, the development of new
instrumentation, facilities, and other tools, and the education and
training of a highly qualified cadre of individuals motivated and
excited by science, engineering, and mathematics. Organizational
excellence in people, processes, and assessment enables all three. The
Committee felt that it was important to continue to make this point, as
it has done previously.

The Committee concluded that the high quality, relevance, and
performance of the NSF portfolio are principally due to NSF's use of a
rigorous process of competitive merit review in making awards. NSF has
continued to make progress in implementing its two principal review
criteria -- intellectual merit and broader impacts with over 90 percent
of all reviews now addressing both criteria. NSF also continues to
provide a heightened focus on the use of both criteria by proposers,
reviewers, and program officers. The Committee notes that this will
continue to be a "work in progress," that is, an ongoing effort that
will require constant vigilance by the NSF program staff and further
education for the proposing and reviewing community as to the importance
of addressing both criteria adequately. Competitive merit review is a
key process for ensuring the quality and relevance of research and in
maintaining US leadership in many areas of science and technology. NSF
and its external stakeholders, both within and outside the Federal
government, should work together to resist the corrosive influence of
forces that are inimical to merit review. The National Science Board
should use its influence to advocate for expanded competitive merit
review across the Federal government's research portfolio.

[Transformative/Bold/Innovative-High Risk Research and
Education]{.underline}

With regard to transformative/bold/innovative-high risk research and
education, the Committee saw evidence of accomplishment. NSF itself has
sought to clarify the definition of such research using an "operational"
approach. NSF asked its program staff to identify projects they believed
reflected transformative/bold/innovative-high risk research and
education. The agency then attempted to organize the 150 nuggets so
identified into a definitive framework with guidance or rubric. The
Committee compared this rubric against the proposals and also reviewed
comments in the Committees of Visitors reports on this topic. Based on
that analysis, the Committee concluded that there is still work to be
done in defining what constitutes transformative research. A complete
discussion of this issue is found in the Organizational Excellence
section of this report. The Committee appreciates the work of the
National Science Board on this issue over the past year and looks
forward to its efforts to initiate a dialogue with the research and
education community.

No matter how much time is spent to carefully and thoughtfully craft a
rubric to define transformative research, there is still no empirical
way to determine what fraction of the portfolio should be the farthest
out on the frontier. This difficulty is complicated by the fact that
researchers (particularly academic researchers) don't typically think of
their research in terms of its "riskiness" in the sense we are using
that word here.

Clearly, the nation benefits and the research enterprise advances when
transformative research is part of the equation. However, when COVs were
asked to comment on this issue, their responses raised the very issues
that we know to be the toughest to address, namely, how do you know this
research when you see It?; how much should be funded in a constrained
environment?; and, how should the very necessary flexibility of NSF
program staff be balanced against what might appear to be a rather
conservative merit review process in making investment decisions in
favor of such research?

This AC/GPA process looks retrospectively at a year, or two or three, of
research progress (as evidenced through the accomplishments). The
determination about whether an investment in a proposal has yielded
results that could fundamentally transform our understanding of the
physical or natural world may take decades. All of NSF's stakeholders,
internal and external, would do well to keep that in mind.

Lastly, the Committee notes that NSF will shortly begin the revision of
its Strategic Plan. This is the continuation of an evolutionary process
that began in the mid-1990's following the enactment of GPRA. As NSF
reflects on its progress toward its current strategic outcome goals and
whether these (and their associated indicators) merit revision, we urge
NSF to give some considerable thought to the concept of "significant
achievement." As a committee, we have been charged to make
determinations of whether NSF has met this threshold of accomplishment.
But "significant achievement" has been, and continues to be, very much
in the eye of the beholders -- even though NSF has endeavored to provide
a wealth of data and information to inform our collective assessment.
Over time, and with more and more project accomplishments in the
database, the gradation between significant achievement and the lack of
it will likely become not only tougher, but also more nuanced. We have
no answer today to the question of whether there might be an objective
standard (or standards) by which "significant" in the context of
achievement might be measured. However, we also know that NSF supports
some of the best and most creative individuals working in the area of
research assessment and the agency should enlist these people and the
broader scientific community in thinking through this issue before the
next Strategic Plan is finalized.

PEOPLE Strategic Outcome Goal

The Committee found significant achievement for PEOPLE indicators 1, 2,
3 and 4. However, based on evidence provided, the Committee did not find
significant achievement for indicator P5.

[Quality and relevance:]{.underline} Based on the review of COV reports
and project accomplishments (nuggets), the overall quality of projects
was determined to be high and relevant to the People strategic outcome
goal. Delivery methods and application of research findings were also
found to contribute to the high quality of projects reviewed. Many of
the projects reviewed have high relevance to the development of a strong
workforce and public understanding of science.

[Transformative/Bold/High risk-Innovative projects:]{.underline}
Projects contributing to the People goal were found across NSF as
evidenced by the breadth of nuggets selected to illustrate significant
achievement. Overall, the Committee found ambitious projects that we
would consider "bold.\" One general observation was that high risk or
bold projects seemed to be less likely to be funded under the PEOPLE
strategic goal.

**Other Comments:**

[Reduced funding:]{.underline} The Committee members reviewing this
strategic goal expressed serious concern about the significant decrease
in funding for programs that focus on the People Goal. Funding levels
for this goal have declined from \$1,146,880,000 in 2004 to the FY 2006
Request of \$978,770,000. In addition, the number of people involved in
or impacted by NSF activities has declined from an estimated 215,350 in
2004 to 168,280 in 2006. This trend should be monitored carefully by the
AC/GPA because it could have an adverse impact on NSF's ability to
demonstrate significant achievement in the future. The principal
organizational unit within NSF for meeting the PEOPLE outcome goal is
the Education and Human Resources (EHR) directorate. This directorate
has borne the brunt of the funding reductions noted above. This may have
long-term implications for meeting the objectives in the People goal.
The Committee recognizes that other directorates within NSF are making
major contributions to this goal. However, delegating yet more
responsibility for meeting these objectives to other parts of NSF
because of budgetary realignments may result in lack of experience and
expertise in K-12 education, particularly in programs that sustain
high-quality, high-commitment engagement of scientists and
mathematicians with students and teachers in classroom settings.

[Data Collection and Assessment]{.underline}: Effective assessment
should be, at its heart, data-driven. Thus, it is very important to
develop simple but effective metrics and to provide data that enable
both qualitative and quantitative analyses of progress toward the People
goal. This will be critical to establishing a context for future
evaluations by this Committee or others of NSF's level of achievement.
The Committee on Equal Opportunity in Science and Engineering (CEOSE)
recommended in its 2004 report, *\"Broadening Participation in
America\'s Science and Engineering Workforce*," that NSF should expand
its systematic and objective evaluation efforts by continuing to
"obtain, refine and disaggregate data and factors related to the
participation and advancement of persons from underrepresented groups in
STEM education and careers" ([Executive
Summary](http://www.nsf.gov/od/oia/activities/ceose/reports/ceose2004exec.pdf),
p. 7-8; CEOSE 04-02). We support that recommendation and urge NSF to
increase its focus on this issue and to strive to identify those data
elements (particularly those collected over a long period) that are the
most critical to assessing program impact.

[Broadening Participation:]{.underline} It is important for NSF to
emphasize that \"broadening the participation of underrepresented
groups\" is not an issue of simple demographics, but of increasing the
diversity of *paradigms, ideas, methods, and perceptions* brought to the
Foundation\'s programs. In particular, NSF must develop strategies to
ensure that activities aimed at broadening participation are carried out
with rigor and attention to high-quality research.

[P5 Designation:]{.underline} P5, "Support innovative research on
learning, teaching and mentoring that provides a scientific basis for
improving science, technology, engineering and mathematics education on
all levels," is a research goal that contributes to building a
workforce. NSF is encouraged to review whether or not it would be more
appropriate under the "Ideas" goal.

[**PEOPLE GOAL \-- Indicator P1**:]{.underline} Promote greater
diversity in the science and engineering workforce through increased
participation of underrepresented groups and institutions in all NSF
programs and activities.

Based on the accomplishments provided, NSF devotes a substantial amount
of resources to fund projects that contribute to the attainment of the
PEOPLE strategic outcome goal as articulated under Indicator P1.
Collectively, the projects demonstrate significant achievement toward
producing a workforce with strong representation of under-represented
groups and women in science and engineering. Within EHR, there are
numerous programs that are relevant and contribute directly toward this
indicator. In total, the estimation was that 134,050, 113,890, and
86,050 individuals were or will be involved in each of FY2004, 2005, and
2006 respectively. In addition, there are programs distributed across
all directorates that support this indicator. Examples include selected
projects in the Research Experiences for Undergraduates (REU)
([0244221](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0244221))
and CAREER programs. The portfolio of the funded activities is broad.
Projects such as the "Valle Imperial Project" and the "University of
California Alliance for Graduate Education" range from professional
development of K-6 teachers in one of the poorest counties in the
country
([9731274](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=9731274))
to graduating more minority doctoral recipients in STEM fields
([0450366](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0450366)),
and from increasing ethnic minority student participation at the college
level through projects in the REU and Model Institutions for Excellence
(MIE) programs ("Research Experiences for Undergraduates in
Environmental Sciences at Northern Arizona University,"
[0244221](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0244221)
and "University of Texas at El Paso's MIE-Supported Academic Center for
Engineers and Scientists,"
[9550502](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=9550502))
to reaching out to young women in middle and high schools through the
Research on Gender in Science and Engineering program
([0080386](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0080386)).
Due to adverse funding trends in most of the EHR programs, the level of
achievement is expected to decline unless funding in other programs is
increase sufficiently to compensate for the EHR reduction.

Some of the ongoing projects address the "pipeline" issue by focusing on
K-12 students. Consider the Valle Imperial Project in Science
([9731274](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=9731274))
conducted by the El Centro School District located in the Imperial
County in southeast California. It involves 14 other school districts in
the county and the Imperial Valley Campus of San Diego State University.
Most of the K-12 students are underrepresented minorities and from
low-income families. The project has increased the number of students
taking college prep STEM classes and led to tripling the percentage of
graduates eligible for enrollment in the University of California
system. The Techbridge project conducted by a collaborative partnership
based at the Chabot Space and Science Center
([0080386](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0080386)),
focuses entirely on encouraging more women to pursue science and
engineering in a girls-only environment. The approach taken involves
exposing the students to experiences and opportunities that are
otherwise not available to them. The curriculum developed has been found
to produce positive results and is available online
(<http://www.chabotspace.org/visit/programs/techbridge.asp>).

At the college level, there are many projects that aim to increase
participation of ethnic minorities in science and engineering. Examples
include the "REU: Environmental Science Summer Program at Northern
Arizona University"
([0244221](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0244221))
and the "University of Texas at El Paso's MIE-Supported Academic Center
for Engineers and Scientists" (ACES)
([9550502](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=9550502)).
Due to its location, Northern Arizona University is able to attract a
significant number of American Indians to participate in the program. In
2003 and 2004, there were 8 and 10 students, respectively, that attended
the program: among the 18 were 14 Native American and 2 Hispanic
students. Likewise, the University of Texas at El Paso serves an area
with a large Hispanic population. Two-thirds of the STEM students at the
university are participating in the MIE-ACES program, which has
contributed to a 9 percent increase in undergraduate STEM degrees.

Targeted at the post-baccalaureate level, the "University of California
Alliance for Graduate Education and the Professoriate (AGEP) Phase II"
([0450366](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0450366))
was initiated in 2004. It involves all 10 UC campuses. Impressive
results were achieved in Phase I of this project. There was an average
of 131 new minority graduate students enrolled in STEM during 1997-1999.
By 2003, the number had increased to 237, yielding an 80 percent
increase. Phase II of this AGEP will build upon prior success and has
the potential to pose a new model for recruiting, retaining, and
graduating STEM minority doctoral degree recipients and assisting with
postdoctoral placements.

**[PEOPLE GOAL \-- Indicator P2:]{.underline}** Support programs that
attract and prepare U.S. students to be highly qualified members of the
global S&E workforce, including providing opportunities for
international study, collaborations and partnerships.

The success of NSF in meeting the P2 indicator is largely due to the
activities of one foundation-wide program, OISE (Office of International
Science and Engineering, and its earlier incarnation INT). Of the 46
nuggets listed under "primary indicator" for P2, only 13 \-- less than
30percent \-- met both the stated criteria for selection, namely that
the program attract and prepare US students to science **and** that part
of preparing them to be highly qualified members of the global workforce
include providing opportunities for international collaboration. Six of
those were produced by OISE, with a range of other divisions
represented. Exemplary activities recruited and trained students in
science and offered them significant opportunities for international
collaborative learning. The collaborative elements of these
opportunities superseded standard international field practices of the
past, in which researchers collected specimens or data abroad, brought
them to the U.S., and published without consulting, conferring with, or
including colleagues from the host nations.

The Boulder School for Condensed Matter and Materials Physics
([0437903](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0437903))
brings together large numbers of graduate students (60 this year) from
around the world for summer coursework and lectures. Not only is the
student body international, so is the team of presenters brought to the
campus. This program meets students at a high level to forge new
partnerships, understandings, and research agendas at the frontier
juncture of optic, atomic, and condensed matter physics. Another highly
interdisciplinary program, PRIME (Preparing Undergraduates for the
Global Workforce in Cyberinfrastructure) of University of California,
San Diego (UCSD)
([0407508](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0407508)),
brings together a smaller number of students at an earlier career stage
and across a broader level of engagement. Nine students, 3 of them from
the US, studied and worked together on research while immersed in the
international environment generated by UCSD partners in the Cybermedia
Center of Osaka University (Japan), the National Center for
High-Performance Computing in Hsinchu (Taiwan), and the Department of
Computer Science at Monash University (Australia). Admission to both
these programs is competitive, and PRIME requires participants to return
to UCSD in the fall for at least one quarter in order to continue their
project work and share their experiences with potential new PRIME
students.

Graduate students in the University of Alaska, Fairbanks IGERT program,
\"Regional Resilience and Adaptation: Planning for Change,\"
([0114423](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0114423))
have done research and helped develop related international policy and
legislation with scientific bodies of other governments, namely the
Swedish Royal Academy of Agriculture and Forestry and the Alaska Native
Science Commission. Graduate students in another University of Alaska,
Fairbanks, program have participated in a U.S.-Russia International
Volcanological Field School at sites in Alaska and Kamchatka, developing
professional relationships with each another as they study the
relationships between the two major areas of volcanic activity
([0429155](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0429155)).
And in yet another variant of this indicator theme
([0096097](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0096097)),
graduate students at the University of Kentucky and MIT have been able
to carry out research in the Japanese university system known worldwide
for its leadership in carbon science, as part of a U.S.-Japan
collaborative research project that\'s paid off in numerous
publications, conferences, and advancements in carbon science.

The 2002 COV report for OISE (still designated as INT at that time)
stated: \"INT clearly enjoys a level of impact that goes far beyond its
very modest budget. It is exciting to imagine how much greater the
impact could be if INT had resources more commensurate with its level of
responsibility, particularly for project funding and travel for INT
personnel.\" Given the importance of the P2 indicator in achieving the
NSF\'s strategic People goals, we note with approval that OISE has been
given a role as a crosscutting "agent of change" within NSF. OISE's new
organizational position should enhance its success in stimulating
international activities across the Foundation.

# [PEOPLE GOAL \-- Indicator P3:]{.underline} Develop the Nation's capability to provide K-12 and higher education faculty with opportunities for continuous learning and career development in science, technology, engineering and mathematics.

NSF invests in developing the Nation's capability to provide K-12 and
higher education faculty with opportunities for continuous learning and
career development in science, technology, engineering and mathematics.
Development opportunities are funded over a wide range of programs, and
evidence from outcomes reported by projects funded in FY 2005
demonstrate significant achievement in taking a variety of approaches to
engage teachers and faculty in quality development experiences across
STEM disciplines. Research Experiences for Teachers (RET), the CAREER
awards, and the Teacher Preparation Continuum (TPC) are examples at the
program level that help achieve NSF\'s goals.

Some projects connect K-12 teachers with university STEM faculty members
through active research collaborations. For example, "The Alaska Lake
Ice and Snow Observatory Network (ALISON): A Statewide K-12 and
University Science Education and Research Partnership"
([0326631](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0326631))
at the University of Alaska, Fairbanks Campus, provides teachers at 17
schools around the state with a professional development experience and
with researcher mentors, as well as connecting them with other teachers
throughout the state of Alaska. This experience with science and
professional networks can help alleviate the feelings of isolation
common to teachers in rural Alaska, where teacher turnover is high and
student populations are largely Alaska Native. Another approach that
gives teachers opportunities to do STEM research is seen in "RET Site:
Research Experience for Teachers in Areas of Innovative and Novel
Technologies in Philadelphia" (RETAIN Technologies in Philadelphia)
([0227700](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0227700))
at Drexel University. Providing K-12 teachers with hands-on research and
education experiences demonstrated the power of experiential learning in
science and engineering. The project also helped participants bridge the
gap between technology and curriculum by providing workshops and
resources to support curriculum development. Finally, the project has
led to a number of other related projects throughout Philadelphia
schools.

Other projects are providing a foundation for professional development
opportunities. The National Science Teachers Association (NSTA) has a
conference grant
([0442722](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0442722))
sponsored by the Teacher Professional Continuum program, that is testing
a strategy that assembles experts supported by NSF to disseminate their
findings that address important questions in K-12 science and
mathematics education. The first prototype conference, "Linking Science
and Literacy in the Classroom," was offered at the NSTA Regional Meeting
in Seattle in November 2004. A total of 375 teachers, administrators and
professional development providers participated. Presenters included
leading scholars, researchers, and practitioners who described
NSF-funded work on the multiple aspects of literacy in K-8 science
classrooms. The 30 presenters were Principal Investigators (PIs) or
participants in TPC, Local Systemic Change, Teacher Enhancement,
Instructional Materials Development, or other related NSF programs that
have been researching this high profile topic. Another approach is the
Lesley/TERC Science Education Master project
([9911770](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=9911770)),
a national, on-line Master\'s program for K-8 STEM educators that merges
the expertise of scientists and educators, and is carrying out research
on the effectiveness of on-line learning. Its enrichment curricula
should be flexible to accommodate busy schedules and geographical
challenges and must be relevant to the classroom. A total of 380
teachers from 33 states and three countries have participated in one or
more courses since the program's inception in Summer 2000. The first
graduates were in Spring 2003; 47 teachers have graduated from the
program, and currently there are 114 M.Ed. candidates. Leadership is the
focus of a third example, the Fulcrum Institute for Education in Science
([0412456](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0412456))
at Tufts University, where teachers prepare for roles as school-based
intellectual leaders in their fields and catalysts for reforming the
mathematics and science programs in their schools. Their schools and
districts commit to providing the time and resources commensurate with
the positions of increased responsibility that the emerging
teacher-leaders are expected to assume upon completion of an Institute
program that deepens and updates their content knowledge, instructional
strategies and leadership skills.

The Southeast Center for Networking and Information Technology
Education,
([0071047](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0071047))
located at the Daytona Beach Community College, is an example of faculty
development in higher education. The center established a framework for
community colleges to collaborate in the delivery of advanced technology
faculty development workshops that helps colleges offer courses in the
key high demand IT curriculum areas. Based on data from the Florida
Community College System, the project\'s 105 faculty development
workshops supported instruction across 557 different course titles
within the system since the fall of 2000, benefiting 914 community
college faculty members, who in turn teach over 20,000 students annually
in the region.

[**PEOPLE GOAL \-- Indicator P4**:]{.underline} Promote public
understanding and appreciation of science, technology, engineering, and
mathematics, and build bridges between formal and informal science
education.

The range of accomplishments reported under Indicator P4 show that NSF
is investing in effective informal science education materials and
incorporating public outreach and dialog into many programs and
projects. The accomplishments indicate that NSF-supported activities are
reaching large numbers of people of all ages with insights from many
fields, including biology, the earth and atmospheric sciences,
engineering mathematics, and psychology. The portfolio of work shows a
willingness to push the envelope, and is highly multidisciplinary. NSF
has reached a level of significant achievement in this area.

One set of channels for informal science education is popular media:
television, radio, movies, and the Web. An example of using these routes
effectively is the "Magic School Bus," the most successful children's
science series in history, with more than 54 million books in print and
52 television episodes
([9153967](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=9153967)).
NSF supported the original development of the series, and more recently
funded development of associated bilingual traveling exhibits for
children aged 5 to 12
([9627162](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=9627162)).
The traveling exhibit, which has visited 36 cities in a six-year tour,
allows students to explore the dynamics of weather. Other examples of
the broad outreach of informal science education include the "Pulse of
the Planet" series (heard over 309 broadcast outlets worldwide)
([0337143](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0337143)),
TV411, for adult math education
([0104712](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0104712)),
*Under Antarctic Ice*, a program in the PBS Nature series
([0000373](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0000373)),
"Peep and the Big Wide World," a television series for 3 to 5 year olds
that was rated second in viewing audience in its time slot
([0104700](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0104700));
and web access to news from Antarctica
([0000373](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0000373)).
NSF is supporting evaluation of the effectiveness of its informal
science education work. For example, visitor impact evaluation of
learning outcomes from the Magic School Bus tour indicates that 80
percent of the children who tour the exhibit gain new knowledge about
weather dynamics or learn a new weather concept. Follow-up telephone
interviews indicated that the children stay interested in the weather
several months after their visit. Likewise, evaluation has shown that
children who watch *Peep* are much more likely to ask questions and
solve problems than those who do not.

Museums also provide an opportunity to engage the public. For example,
Martin Luther King Day at the Cleveland Museum of Natural History
([0133164](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0133164))
was advertised broadly to 54 municipal schools, plus youth groups,
church groups, and recreation facilities. NSF-supported polymer
researchers, with their graduate students, put together mini-lectures,
displays, demonstrations, and hands-on experiments for the more than
4,000 visitors. Attendance was up 50% from the previous year. Other
examples of informal science education through museums include "Go
Figure," an exhibit at the Minnesota Children's Museum to engage parents
and children in mathematics learning, particularly in underserved
communities
([9725857](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=9725857));
the CAREER program's courses involving undergraduate students in
independent historical research at science museums
([0134482](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0134482));
engaging the public in botanical gardens through studies of the vanilla
orchid
([0108100](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0108100));
and an exhibit on "Strange Matter," produced by materials scientists and
visited by tens of thousands at New Jersey's Liberty Science Center
([0213706](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0213706)).

Science education goes two ways, especially when it moves into
communities with special knowledge of the environment. An example is the
project "Fire-Mediated Changes in the Arctic System: Inter ... and Human
Activities"
([0328282](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0328282)).
The community of Huslia, Alaska, has been teaching university
researchers about how fire affects their community and researchers share
what they know about future changes in climate and fire regime. The
mutual learning workshops are turned into teaching materials, which are
shared with local schools after approval by the Huslia Tribal Council.
The elders view the project as one of few opportunities they have to
talk to students about traditional knowledge. Other community-based
mutual learning projects include the Community Collaborative Rain, Hail,
and Show Network on the Great Plains
([0229723](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0229723));
*Math in the Garden*, a set of activities that teach math to children
and adults in relation to gardening topics
([9909764](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=9909764));
and a project on well water quality on the Navajo Reservation
\[0348873\].

NSF programs also move into the classroom to spread interest and
confidence in science, and move students from classrooms into the
laboratory. An example is Project SERVE (Science Enrichment using
Retired Volunteer Educators)
([0412101](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0412101)),
which links senior citizens with young students. A Discovery Corps
Senior Fellowship supported the investigator to train senior citizens in
age- and pedagogically-appropriate general chemical principles. The
senior citizens then volunteer in elementary and middle school
classrooms as teacher's aides, tutors, mentors, and resource persons for
under-performing students. Other classroom enrichment projects include
EdGCM, a global climate model that is run on inexpensive desktop
computers
([0231400](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0231400));
glassblowing demonstrations for K-12 students at the University of Iowa
([9972466](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=9972466));
nanoscience made simple for junior-high school students in southeastern
Ohio
([0304314](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0304314);
femtosecond laser systems at Michigan State for middle school students
([0135581](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0135581));
safe racer competitions in Baltimore that involve elementary students in
engineering design
([9731748](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=9721748));
and demonstrations on nanostructured materials and interfaces for K-12
students in Wisconsin
([0079983](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0079983)).
NSF's outreach in informal science education is even becoming
international. The Fab Lab project
([0122419](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0122419))
goes into the field to allow participants to fabricate objects at micron
size and microsecond speed. This gives participants a hands-on
experience with manufacturing components for information technologies,
not just with using the technologies themselves. The exhibit has reached
many under-served communities, including in rural India, northern
Norway, Boston, and Costa Rica. In the past year, it has engaged the
public in Ghana, and is working on a collaborative exhibit in South
Africa. The worldwide public is also able to participate in LIGO, the
Laser Interferometer Gravitational wave Observatory, searching LIGO data
through Project Einstein@home
([0200852](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0200852))
and web access to real-time Mars exploration
([0104589](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0104589)).

[**PEOPLE GOAL \-- Indicator P5**:]{.underline} Support innovative
research on learning, teaching and mentoring that provides a scientific
basis for improving science, technology, engineering and mathematics
education on all levels.

As the following five projects show, activity in this indicator area is
found within current NSF-sponsored programs. However, these projects
were the only ones found within the set of nuggets proposed to satisfy
this indicator (56 in the Primary set, 88 in the Secondary). Thus, we
conclude that this does not constitute a body of work sufficient to
determine that NSF has met the "significant achievement" threshold with
respect to this important indicator. Though one relevant COV report
(ROLE) from 2002 suggested significant achievement in this area, the
paucity of current nuggets seems to contradict this. This lack of
relevant nuggets may be due to confusion on the part of program
directors as to what exactly this indicator means. The following
programs, significant in their own right in terms of quality, relevance,
and multi-disciplinarity, are involved in the study of individual
learning, group/collaborative learning, the assessment of learning, the
dissemination of the results of learning research, and the mentoring of
STEM faculty.

The work of Robert Sternberg of Yale University
([9979843](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=9979843))
is focused on the methods or "modalities" of individual learning through
triarchic instruction and assessment. Sternberg's work suggests that
individuals learn through a combination of three approaches: creative,
analytic, and practical thinking. By training elementary school teachers
in this "Triarchic" theory, Sternberg is helping them to recognize the
learning patterns of their students and to tailor their lessons to the
individual student' needs. Work is also being carried out to better
understand how STEM students learn in groups. Gerry Stahl of Drexel
University
([0325447](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0325447))
is studying how math students utilize the Internet to work together to
solve problems. By collecting and analyzing records of student
problem-solving chat groups, Stahl hopes to develop a theory for how
students best learn in such situations and to disseminate this
information to mathematics teachers world wide.

A fundamental problem in pedagogical research is that of assessment. It
is crucial to the scientific study of learning that new and innovative
teaching techniques be assessed. One NSF-funded project aims to improve
upon current assessment techniques. Tiffany Koszalka of Syracuse
University
([0335644](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0335644))
is leading an attempt to understand and assess how practitioners of a
field move from novice toward expert-level problem solving abilities. By
discerning the thinking and decision making methodologies followed by
experienced practitioners, the "Enhanced Evaluation of Learning in
Complex Domains" (DEEP) project hopes to improve the methods of
assessing the learning of novice and intermediate-level practitioners.
However, individual results from pedagogy research can only be useful to
the teaching community at large if they are efficiently disseminated.
This is the goal of the project, "Program Evaluation for the Math and
Science Partnership"
([0456995](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0456995)).
This partnership of related programs, known as the MSP Learning Network,
is developing a community of connected researchers, allowing them to
quickly and easily share their results. Through the building of
electronic communities and digital databases, the results of learning
and pedagogy research are being made available to K-12 teachers, college
faculty, and the technical/scientific community at large.

Lastly, it is important to the success of new pedagogical initiatives
that those involved in the teaching be actively mentored. The project,
"SOMAS: Support of Mentors and their Students in the Neurosciences," led
by Julio Ramirez of Davidson College
([0426266](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0426266))
has received funding to both allow junior STEM faculty to involve
undergraduate students in their research activities and to bring these
students together with mentors to help the mentors make the most of
their pedagogical opportunities. The SOMAS project aims to assist junior
faculty in integrating students into their scholarly activities thereby
improving the students' oral, written, and cognitive skills and making
them much more likely to succeed in their programs.

## IDEAS Strategic Outcome Goal

The Committee concluded that there has been [significant achievement in
all indicators]{.underline} of the IDEAS strategic outcome goal, which
is to foster "discovery across the frontier of science and engineering,
connected to learning, innovation, and service to society." The
Committee concluded that NSF had met the goal for each indicator in
making investments in discovery, collaborative research and education,
connections between discoveries and their use in society, increased
opportunities for underrepresented individuals and institutions,
developing new research and education opportunities, and creating new
integrative and cross-disciplinary knowledge and tools. It is worth
noting that our determination of "significant achievement" is, in large
part, a reflection of the fact that the ideas embodied in the projects
in this portfolio are themselves significant -- that is, of high quality
and relevance.

Whether we consider engineering, life sciences, physical sciences,
social sciences, or information technology, it is apparent that
NSF-sponsored research is having a significant impact on our nation and
world today and shows every indication of continuing this into the
future. The challenge for the Committee in this strategic outcome goal
was in selecting a relatively few nuggets from the vast array of very
fine projects from which to choose. For each of the six indicators, the
accomplishments were chosen to illustrate the breadth and depth of NSF's
portfolio with special emphasis placed on the important objective of
broadening participation.

IDEAS in themselves are the essence of the research and education
mission of NSF. Themes emerged in the arena of IDEAS many of which
involve enhanced interaction between scientists and engineers,
especially across broad areas within the life sciences. For example, the
potential of nanotechnology coupled with the biological sciences is
generating research projects that hold significant potential for
understanding and improving the human condition. Applications of
engineering principles and practices to the environment are now yielding
new ways in which we can temper the effects of natural forces such as
earthquakes. These themes illustrate the power that multidisciplinary
research can have on approaches to answer questions that could not
previously be addressed.

Perhaps one of the most powerful illustrations of the potency and
efficacy of NSF sponsorship comes from an analysis of funding for Nobel
Prize winners. In 2004, Kydland and Prescott won the Nobel Prize in
Economics. Both were beneficiaries of NSF support throughout their
careers, such as "Studies in Aggregate Analyses"
([0422539](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0422539)),
and winning a Nobel Prize is further validation of the quality and
relevance of NSF-sponsored research. Remarkably, within economics, the
NSF has sponsored research for 32 winners of Nobel Prizes.

To broaden participation, the Foundation has supported international
collaborations, often involving cross-cultural and crosscutting
experiences for investigators and students in particular. For example,
there are large and important societal benefits as well as scientific
benefits that have been gained from NSF support to send teams of
investigators to Africa to investigate ways to preserve and propagate
endangered wild animal species. NSF has also significantly increased
opportunities for underrepresented individuals to participate fully in
the research enterprise embodied in the IDEAS portfolio. Several themes
emerged, including projects to improve the access to STEM by disabled
persons; culturally-based learning projects that utilize the student\'s
life experience and culture as jumping-off points for hands-on learning;
CAREER awards that provide the groundwork for highly successful careers;
and the strong coupling between outstanding science and thoughtful
mentorship in NSF projects.

Thus, NSF's portfolio of accomplishments in the IDEAS strategic outcome
goal exhibits both exceptional quality and high relevance to important
national goals. In addition, the Committee found numerous examples of
"transformative/bold/innovative-high risk" research in the IDEAS
portfolio. A more in-depth discussion of this topic is found in the
section on the Organizational Excellence strategic outcome goal.

**[IDEAS GOAL \-- Indicator I]{.underline}:** Enable people who work at
the forefront of discovery to make important and significant
contributions to science and engineering knowledge.

The NSF was established as the "patron of pure science." Therefore,
researchers who work at the forefront of discovery are the best
candidates for NSF support and are the most likely to receive it. We
find that NSF support has been critical to enabling researchers be in
the vanguard of those at the frontier. There are numerous examples of
major results and below we summarize a few examples that give a sense of
the wide breadth and significance of NSF support.

The "Biomechanics and Hydrodynamics of Fish Locomotion" research focuses
on the analysis of the motion of fish fins and the resulting propulsion
and positioning accuracy using techniques from fluid engineering
([0316675](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0316675)).
One goal of this research is to apply this knowledge to man made
vehicles that at present have several limitations. Recent findings show
that 1) fish can extract energy from high-speed turbulent flows and thus
maintain position using minimal muscular energy; and 2) fish use several
fins simultaneously to generate discrete vortex rings allowing them to
achieve fine positional control.

After devastating earthquakes in Turkey and Taiwan, the NSF funded
several reconnaissance missions including the project, "Ground
Improvement Techniques Shown to Mitigate Earthquake Damage"
([0085281](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0085281)).
This work investigated the performance of sites that had been improved
prior to construction to reduce the liquefaction potential of these
sites. The study demonstrated that ground improvement was effective in
mitigating earthquake-caused damage and in particular was the first to
verify that closely spaced jet-grout columns worked well. Although these
techniques have been widely used, this work is first to give evidence of
the effectiveness in an actual earthquake. This work has immediate
application to the design and implementation of these techniques in the
U.S. and worldwide.

"How Does the Brain Overcome Obstacles to Successful Memory Performance?
Insights from Studies of Prefrontal Cortex and Interference Resolution"
has helped to increase our insight on neuroimaging of cognitive and
mnemonic control
([0401641](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0401641)).
When we try to remember a particular piece of information -- like the
location of our parked car -- there can be interference in the brain due
to the recalling of memories having been associated with previous
parking situations. This work performed several studies using functional
magnetic resonance imaging that established a correlation with activity
in the left ventrolateral prefrontal cortex and the interference of
memory. This research is important in trying to further understand and
hopefully improve memory performance.

There has been a long standing mystery in understanding the seismic data
that have been collected from the layer between the outer liquid core of
the Earth and the inner mantle at a distance of about 2,700 kilometers
below Earth's surface. This boundary is called the D'' layer. The "Inner
Earth Revealed" team supported by the NSF analyzed x-ray images of
perovskite taken at the high pressure and temperature expected in the
D'' layer and found a new type of structure that will explain the
previous data
([0135533](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0135533),
[0215587](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0215587),
and
[0230319](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0230319)).
This discovery will allow better understanding of the of Earth's
interior.

Researchers on "Nanotube Membrane Mimics the Functions of the Biological
Cell Wall" created a working synthetic membrane made of 8 to 12
nanometer gold nanotubes deposited on a polycarbonate template
([9987646](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=9987646)).
They verified that this membrane did function like cell membranes in
recognizing and allowing certain DNA segments to pass more easily than
others. This multidisciplinary project uses a chemical model to mimic a
biological cell membrane. Such membranes could be useful for DNA
separation and/or genomic research.

**[IDEAS GOAL \-- Indicator I2]{.underline}:** Encourage collaborative
research and education efforts across organizations, disciplines,
sectors and international boundaries.

There were many examples of projects that met many (or even all) of
these goals simultaneously. The project accomplishments that were chosen
represent the range of goals captured in this indicator. Additionally,
projects were chosen to represent a range of subjects and societal
needs.

One outstanding example is the project, "A Sex Pheromone Elicits
Distinct Behavior in Male African Elephants," which is multidisciplinary
in nature, involving the collaboration of Principal Investigators with
different training from three universities across the US
([0216862](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0216862)
and
[0217062](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0217062)).
In terms of education, this project serves not only the graduate
students who are about to become professionals in their fields but also
creates excellent opportunities for the succeeding group of students,
the undergraduates. The project is international in nature, involving
the cooperation of international organizations and governments and could
not be successful without it. Additionally, this research has the
potential for preservation of the African elephant, an endangered
species, and therefore maintaining current levels of biodiversity.

Another excellent example is a project, "U.S./Africa Materials
Institute"
([0231418](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0231418)),
in which chemists, materials scientists and biomedical researchers from
US universities and organizations join with their counterparts from
several African countries to conduct research on improving early cancer
detection. Successful treatment of cancer depends in part on its size at
detection. Current imaging techniques can resolve tumors a few
millimeters in size. So far, the team of scientists working on this
project is able to detect tumors that are a fraction of a millimeter.
This has untold benefit for the treatment of cancer. The
multidisciplinary, collaborative and international nature of the project
is clear. One of the interesting (unusual) aspects of this project is
that the education is not occurring at the university student level but
at the level of the research scientists. And it involves a transfer of
information from the African scientists to the US scientists and vice
versa. More often the transfer of information is from the US to the
lesser-developed region. This research provides opportunities that would
be otherwise difficult for the African scientists to access and has
beneficial effects on the field of health and medicine in the US and
Africa (and potentially the world).

The project, "Beetles and Their Yeast Endosymbionts From Basidiocarp
Habitats," is multidisciplinary and collaborative at the U.S. university
level but not at the international level
([0072741](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0072741)).
Although its scientific basis is sound and interesting, it was chosen as
an example of a project that has a very strong undergraduate student
component, a commitment to entraining minority students, and outreach to
elementary and secondary students. Undergraduate students participated
in science at field sites, where they identify, collect and preserve
biological specimens \-- an invaluable experience. The involvement of
undergraduates, minority students and students at earlier stages of
their education has important long-term benefits for the students in
particular and science in general.

Note: the tenses need rationalizing here -- they should all be in the
past ('participated') or in the present ('identify, collect,
preserve'\_). I would choose the past since we were looking at
accomplishments.

The "Puerto Rico Collaborative for Excellence in Teacher Preparation"
(PR-CETP) project is different from the mainstream. It does not focus
directly on scientific research; rather, it focuses on the training of
the teachers who deliver the scientific information to pre-university
students
([0331998](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0331998)).
It involves the cooperation of university and K-12 teachers. This effort
is notable because of its focus on improving education at the earlier
stages of the learning process. Teachers are better prepared which means
that students entering university would be better prepared. This bodes
well for the ultimate advancement of science.

The "Children's Research Initiative" (CRI) researches routine tools used
by wild Capuchin Monkeys
([0125486](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0125486)).
It meets each of the goals outlined above. It stands out from the rest
of the group because it is an excellent example of research led by a
female Principal Investigator and it has the potential for understanding
further the links between humans and other primates. This research
examines the use of tools by the wild capuchin monkeys and is an
opportunity to study the development of this behavior, which was once
thought to be peculiar to humans.

**[IDEAS GOAL \-- Indicator I3]{.underline}**: Foster connections
between discoveries and their use in the service of society.

One of the goals of the NSF is to build and foster connections between
research that leads to new discoveries and the societal benefits of
these discoveries. What is truly impressive about the breadth of
research sponsored by the NSF in this regard is that it is both broad
and deep, from large-scale studies that examine carbon cycling in our
oceans to improvement of cities at risk for massive earthquake damage.

The project accomplishments selected to illustrate the impacts of
NSF-sponsored research in this area include:

1.  A Long-Term Ecological Research (LTER) grant, "Plum Island Sound
    Comparative Ecosystem Study (Pisces) Effects of Changing land Cover,
    Climate and Sea Level on Estuarine Trophic Dynamics," that involves
    an investigation of the contribution of dissolved organic matter
    from living organisms to the overall carbon cycling within deep
    oceans
    ([9726921](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=9726921));

2.  "Intrusion Detection Techniques for Mobile Ad Hoc Networks," a
    project involving student participation at many levels, has led to
    advancements in wireless security technology that have the potential
    to be developed for use at very low cost
    ([0311024](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0311024));

3.  The project, "Earthquake Engineering Research Center" had direct
    applications in improving the ability of the critical infrastructure
    of the city of San Francisco to withstand significant earthquake
    activity
    ([8607591](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=8607591));

4.  "Organic Materials of Intermediate Dimensions for Optoelectronic
    Technologies" is a project that has led to the discovery of new
    optoelectric capabilities for building sensors for the detection of
    individual viruses or bacteria, a technology that may prove critical
    in the area of homeland security
    ([0097611](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0097611));
    and

5.  The project, "Dynamic Employer-Household Data and the Social Data
    Infrastructure," is a sociological and economic analysis of means
    whereby low-income women, the employment rates of whom have reached
    all-time highs, can be encouraged by policymakers to pursue
    strategic job ladders that move them out of poverty
    ([9978093](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=9978093)).

Each of these projects has a direct impact on an area or areas that have
in recent years been identified as a national and/or regional priority.
Indeed, several of these illustrate the global nature and potential
effects that research in the areas of critical technologies or
sociological imperatives can have.

There is relevance and high risk in each of the examples cited above.
The impact of the large, multidisciplinary initiatives such as LTERs and
ERCs is unquestionably enhanced well beyond the individual sum of the
parts involved. The marriage of life sciences with engineering expertise
provides a particularly potent approach to formerly intractable problems
and is yielding promising results. Moreover, in the case of the fifth
example cited above, the potential impact on society at large of the
novel approach of focusing on employer strategies and practices rather
than on employee characteristics has the potential to transform how we
craft future social policies and manage workforce and workplace issues.

**[IDEAS GOAL \-- Indicator I4]{.underline}**: Increase opportunities
for underrepresented individuals and institutions to conduct high
quality, competitive research and education activities.

NSF programs such as the Louis Stokes Alliances for Minority
Participation (LSAMP), Centers of Research Excellence in Science and
Technology (CREST), Alliances For Graduate Education and the
Professoriate (AGEP), the Minority Postdoctoral Fellowship Program, and
Research Experiences for Undergraduates have historically provided a
stimulus and increased opportunities for women and underrepresented
minorities to participate in all stages of the research process. These
programs have been successful, and now NSF's portfolio contains a number
of examples of projects that involve the full participation of
underrepresented individuals and institutions in the generation of
ideas. Several overarching themes emerge, including: a) improved access
to STEM (science, technology, engineering, and math) by disabled
persons; b) culturally-based learning projects; c) CAREER awards that
have provided the groundwork for highly successful careers of
underrepresented minorities; and d) the coupling of outstanding science
and strong mentorship.

A number of projects involved the improved access to STEM by visually
and hearing-impaired persons, with a cluster of projects addressing the
needs of blind persons. Involving a totally blind graduate student
researcher, the project "Automated Tactilization of Graphical Images:
Full Access to Math, Science, and Engineering for Blind Students" aims
to automatically create tactile versions of maps, charts, graphs,
diagrams, and other images that are found in math, science, and
engineering textbooks
([0415273](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0415273)).
This is an important problem, as the creation of tactile representations
of data is very time and labor intensive. Another project, "Exploring
New Geometry by Touching, Seeing, and Feeling," explores new geometry by
touching, seeing, and feeling has similar goals
([0430730](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0430730)):
it combines computer graphics with 3D computer haptics (which imitates
the 3D sense of touch) to enable blind persons to perceive geometric
shapes including self-intersecting surfaces. Finally, working under the
mentorship of the PIs of the "Engineering Research Center for Biomimetic
Microelectronic Systems" at the University of Southern California
([0310723](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0310723)),
a high-school student won the top prize at the 2004 Orange County
Science and Engineering Fair for her project, "Intraocular Camera for
Retinal Prostheses: Restoring Vision to the Blind."

Culturally based learning projects are providing a novel approach to the
inclusion of underrepresented minorities in competitive research and
education activities. A new paradigm is emerging, one that involves the
student in STEM by using the student's life experience and culture as a
starting point. Examples include a project, "Agricultural Science Summer
Undergraduate Research Education and Development Project" (ASSURED)
([0244179](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0244179)),
in which the children of migrant workers, who have spent their youth
harvesting onions and chili peppers in the field, are now studying these
plants in a laboratory. They are looking at ways to improve yield and to
understand fundamental characteristics of the plants at the genomic
level. Developed in cooperation with Yup'ik Eskimo elders, another
project, **"**Improving Alaska Native Elementary Students' Math
Performance"
([0138920](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0138920)
and
[9618099](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=9618099)),
produced a culturally based mathematics curriculum for elementary school
students. As an example, students learned the mathematical properties of
shapes that they made as bookmarks. Students participating in this
curriculum had significantly higher test scores than those students in
the standard curriculum.

Similarly, there is another project, "Sustainability and Stewardship in
Alaska," that addresses Alaskan Natives and is organized along lines
parallel to NSF\'s Integrative Graduate Education and Research
Traineeship (IGERT) program but is focused on undergraduate education
and research
([0331261](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0331261)).
This undergraduate to graduate pipeline approach invigorates the
students by infusing them with real-world research concepts. They
participate in hands-on research involving the integration of natural
and social sciences for natural resource conservation. The focus is on
sustainability and stewardship of the land.

Women and underrepresented minorities who have received NSF CAREER
awards are making significant contributions to STEM and are becoming
outstanding mentors, as well. For example, Janice A. Hudgings developed
a 2-D thermoreflectance microscopy technique that enables thermal
measurement of optoelectronic devices on the nanoscale in the project,
"High Performance Thermal Profiling of Photonic Integrated Circuits"
([0321449](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0321449)
and
[0134228](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0134228)).
She established the first engineering and physics research lab at Mount
Holyoke College, an ideal context in which to encourage a diverse group
of women undergraduates to participate in science and engineering. To
date, 19 women have performed independent research in her lab, nine of
which are underrepresented minorities.

Kathleen Pickering is using the Pine Ridge Lakota Indian Reservation as
a starting point to study how pre-industrial indigenous societies
organized economic production on a \"subsistence\" level, based on the
family and different from that of market-based industrial capitalism in
"CAREER: Cash and the Social Economy of the Pine Ridge Indian
Reservation: Labor Allocations, Consumption, and Economic Development on
the Periphery"
([0092527](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0092527)).
Her research advances theoretical understandings of the
subsistence-market distinction, trains students in research design and
methods and encourages local Lakota students to consider advanced
studies at the university.

CAREER awardee Kim Venn, in collaboration with researchers at University
of Texas at Austin and University of Texas, El Paso, has analyzed the
chemical composition of stars in a sample of local dwarf galaxies and
compared them to published datasets for stars in the Milky Way in the
projects, "The First Stellar Abundances in Local Group Galaxies" and
"Collaborative Research: Chemical Evolution Beyond the Milky Way"
([0306884](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0306884),
[0307534](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0307534),
and
[9984073](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=9984073)).
They find distinctive differences; their results challenge basic ideas
about the formation of galaxies.

Finally, CAREER awardee Kristi Anseth of the University of Colorado,
Boulder, received the 2004 Waterman Award, which is the highest prize
the NSF offers to scientists from all fields who are not more than 35
years old and seven years since their doctorate. In her pioneering work
in the field of tissue engineering, "CAREER: Photocrosslinkable Polymers
for Fracture Fixation"
([9734236](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=9734236)),
she created polymeric scaffolds that serve as specific templates for the
attachment, growth, and proliferation of cells, and has also developed
novel polymeric materials for the fixation of fractured bones.

A number of projects illustrate that strong mentorship, especially by
and of women and underrepresented minorities, is a very positive
by-product of outstanding STEM accomplishments. For example, Casonya
Johnson is a female African-American who, after graduate and
post-doctoral work at the Johns Hopkins University, returned to her alma
mater, Morgan State University, where she serves as an important role
model for her students. Her research involves functional
characterization of a novel class of genes, discovered through analysis
of the *C. elegans* genome sequence. Her project, "Genetic and Molecular
Characterization of Dual HLH Domain Proteins in *C. elegans*"
([0212336](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0212336)),
supports the integration of quality research and education at a
historically black university.

Two of the graduate student researchers in Frank Bates' (winner of the
prestigious Turnbull Award of the Materials Research Society) laboratory
at the University of Minnesota who contributed to the discovery of a
totally new phase in soft matter were African Americans. The project
team for "Phase Behavior and Network Morphologies in ABC Triblock
Copolymers"
([0220460](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0220460))
synthesized tri-block copolymers, in which the three molecular
components segregate themselves into continuous nanoscale pathways that
are intertwined in a regularly structured way. In this manner they may
find unique applications as membranes, templates, or composites. These
students now have outstanding careers in industry and academia.

Using nanoparticle-mediated assembly of crystals, Jennifer Lewis at the
University of Illinois at Urbana-Champaign has reported, for the first
time, a new directed-assembly route that allows for the creation of
crack-free, single region (or domain) colloidal crystals of high
quality. Her research, "Novel Colloidal Routes to Photonic Band Gap
Materials"
([0071645](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0071645)),
may lead to new optical devices for chemical/biological sensing,
optoelectronics, optical computing, and telecommunication networks.

**[IDEAS GOAL -- Indicator I5:]{.underline}** Provide leadership in
identifying and developing new research and education opportunities
within and across S&E fields.

NSF supports a broad array of research projects that promote the
identification and development of new research and educational
opportunities in science and engineering fields. Many of the projects
demonstrate leadership and novelty and represent new and ingenious ways
of approaching research. Much of the work in this indicator is
interdisciplinary, requiring input by a number of researchers from
different areas. Further, many of the studies involved a combination of
fundamental and applied research with high potential for practical
outcome.

For example, NSF funded, "Renewable and Resource Efficient Composite
Materials for Affordable Housing"
([0229731](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0229731)),
the research of Professor Chandrashekhara at the University of Missouri,
Rolla, and his team of mostly undergraduate students to develop new
fiberglass-epoxy composite materials from soy products. These materials
are suitable for structural use in floors, roofs, and walls and in the
form of a foam for use in insulation panels. This project delineates an
innovative approach to utilizing a waste product to form low cost and
environmentally friendly construction materials. This creative research
involves a multidisciplinary team with backgrounds in polymer chemistry,
composite manufacturing, structural mechanics and environmental
engineering.

Another project, "Multiscale Virtual Reality of Diffusion-Induced
Deformation Processes"
([0313346](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0313346)),
an Information Technology Research (ITR) project, shows leadership in
developing a novel approach to educating today's students for tomorrow's
jobs by supporting the development of joint doctoral programs between
San Diego State University and the University of California, San Diego
(in applied mechanics and materials science) and between San Diego State
University and Claremont Graduate University (in computational materials
science). These joint doctoral programs provide a link between research
universities with those more oriented toward teaching and community
service-based education. These programs will produce students who are
well versed in the technological challenges of today while being
equipped with an extensive background in the fundamental sciences. Both
joint programs enhance the flow of innovative ideas that will provide
San Diego's booming technology economy with a more creative and
inventive workforce.

A project led by Kenneth Beard at the Carnegie Institute, "Investigating
the Origin and Early Evolution of Primates in Asia"
([0309800](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0309800)),
challenges earlier interpretations whereby most or even all of the major
events in primate and human evolution were thought to have occurred in
Africa. The team has uncovered evidence for a broad range of early
primates in Asia, including the oldest and most primitive primates and
anthropoids yet to be discovered. This project has attracted a
substantial amount of attention from popular media and has fostered
international collaborations among American, Chinese, French, Thai, and
Burmese scientists. This research demonstrates leadership because it
challenges the long-held hypothesis that primate and human evolution
took place only in Africa. This work has the potential to change the way
we think about where the evolution of humans began.

Research by Caroline Ross and colleagues at Massachusetts Institute of
Technology on controlled self-assembly of nanostructures,
"Nanostructured Surfaces with Long-Range Order for Controlled
Self-Assembly"
([0210321](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0210321)),
a Nanoscale Interdisciplinary Research Teams (NIRT) project, is hoped to
generate a set of methods and processes to impose precise long-range
order nanostructure arrays over large areas. These methods are designed
to be scaleable and compatible with low-cost, high-volume manufacturing.
The educational goals of this work are to contribute to the public
understanding of nanotechnology and to the training of skilled
researchers.

Another project that demonstrates significant leadership is one that
engages diverse students in developing nuclear physics tools for
unraveling the mysteries of subatomic particles, **"**Precision
Measurements with Pions **"**
([0354808](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0354808),
[0245407](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0245407),
and
[0114343](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0114343)).
This work is a collaboration involving three interactive projects:
Research in Intermediate Energy Physics, Study of Electromagnetic
Structure of Light Pseudoscalar Mesons via the Primakoff Effect, and
Center for the Study of the Origin and Structure of Matter. This
collaboration includes several Historically Black Colleges and
Universities (North Carolina A&T and Hampton University), as well as
scientists from China, Russia, Ukraine, Armenia and Brazil.
Undergraduate and graduate students from five different universities
have been involved in the project. This effort brings nuclear physics to
students often underrepresented in this challenging area.

A team led by S. J. Yoo at UC Davis is working on a project, "Protocol
Agile Optical Networking for the Next Generation Internet"
([9986665](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=9986665)),
that explores new research opportunities in high-speed optical
networking by creating new switching technologies. This project
contributes to knowledge in the area of networking architectures by
developing and demonstrating a new optical networking approach. This new
networking technology can be integrated with campus networks to form the
basis for future cyberinfrastructure. This research group is committed
to integrating research and education and has directly trained 14
graduate students and educated 150 graduate and 250 undergraduate
students.

[IDEAS GOAL -- Indicator I6:]{.underline} Accelerate progress in
selected S&E areas of high priority by creating new integrative and
cross-disciplinary knowledge and tools, and by providing people with new
skills and perspectives

The NSF supports a wide variety of projects that create new integrative
and cross-disciplinary knowledge while providing researchers with new
skills and multi-disciplinary perspectives.

One extremely innovative project fosters cross-disciplinary knowledge by
developing a new graduate program in astrochemistry at the University of
Hawaii, "Untangling the Energetics and Dynamics of Atom-Radical and
Radical-Radical Reactions"
([0234461](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0234461)).
This project is the first of its kind in the United States and has been
spearheaded by Ralf Kaiser, an assistant professor and CAREER awardee.
This program features a curriculum that relates chemical dynamics to
astrochemistry, planetary sciences, laboratory astrophysics,
astrobiology, and combustion chemistry in reaction dynamics and
astrochemistry. Participating units include the Department of Chemistry,
the Department of Physics & Astronomy, the Institute for Astronomy
(IfA), the Hawai\'ian Institute of Geophysics and Planetology (HIGP),
and the Astrobiology Institute (NAI).

In the interest of fostering highly integrative knowledge exchange, NSF
supported a project that utilized a series of workshops aimed at
unifying the cross-disciplinary knowledge of complex networks in order
to generate a text describing that nascent field, **"**First
Crossdisciplinary Text on Optimal Adaptive Management of Complex
Systems,"
([0223696](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0223696)
and
[0224592](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0224592)).
These workshops, organized by Jennie Si at Arizona State University,
brought together experts in neural networks, control theory, operations
research, artificial intelligence, electric power and fuzzy logic. The
new text focuses on adaptive systems that learn to optimize performance
with foresight to manage complex systems prone to unexpected
disturbances like power grids, critical infrastructure and financial
systems.

William Kaiser from UCLA is building a networked infomechanical systems
(NIMS) robotic sensor system to operate continuously in the forest at
the James San Jacinto Mountain Reserve that will provide accurate
environmental
([0331481](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0331481)).
NIMS systems have generated the first three-dimensional characterization
of solar radiation on the space and time scale of forests, waterways and
wetlands. These new robotic sensing systems are suspended on cable
infrastructure and may move, sense, draw water samples from a stream, or
collect images high in the forest canopy while responding suddenly to
events by moving immediately to acquire detailed imaging of compact
objects at centimeter ranges. NIMS research is a convergence between the
computer science and engineering fields of networked sensing and
robotics along with the science application fields of biology and public
health that enables fundamental investigations of ecosystem energy,
water and carbon budgets critical to global change. The NIMS project
includes a summer REU program involving students from universities
throughout the U.S.

The Particle Engineering Research Center (PERC) at the University of
Florida is developing a major new alternative drug transport technology
([9402989](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=9402989)).
This involves collaboration between chemical engineers, materials
scientists, and pharmaceutical researchers. This technology is designed
to deliver drugs specifically to diseased cells, thereby greatly
reducing doses needed by patients while providing a more effective
treatment. Potential applications include drugs used to treat
life-threatening human maladies such as cancer, heart disease, and AIDS.
This significant new application of nanotechnology is the result of a
multi-disciplinary team working in an Engineering Research Center.

NSF is taking the lead on supporting a collaborative research platform
of geographically distributed infrastructure that will be connected via
information technology to address pressing environmental questions on
regional to continental scales. The National Ecological Observatory
Network (NEON) will be a large-scale multi-disciplinary effort led by
the American Institute of Biological Sciences that involves biologists,
engineers, computer scientists, social scientists and educators in a
collaborative effort. NEON will generate knowledge of complex
environmental processes by applying emerging sensor, analytical,
communication and information technologies to investigate the structure
and dynamics of ecosystems and to forecast biological change, such as in
the project, "Infrastructure for Biology at Regional to Continental
Scales"
([0229195](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0229195)).
Example environmental questions that will be addressed include
evaluating the ecological effects resulting from climate-driven changes
on global water and carbon cycles and the emergence of infectious
diseases and invasive species resulting from anthropogenic activities.

# TOOLS Strategic Outcome Goal

The Committee concluded that there has been [significant achievement in
all indicators]{.underline} of the TOOLS strategic outcome goal. The
Committee also concluded that the projects contained in the TOOLS
portfolio exhibit both high quality and high relevance to important
national goals.

# [Innovative/High-Risk /Bold Research]{.underline}: A more thorough discussion of this issue is found elsewhere in this report. However, the Tools subgroup endorses the definitional efforts of the Organizational Excellence Subgroup on this topic. Additionally, we offer three observations: 

-   First, it may be useful to look at [NIST\'s
    ATP](http://www.atp.nist.gov/) (Advanced Technology Program) risk
    rating system, which has been developed over years of experience.

-   Second, one of the mechanisms used by NSF to encourage "bold"
    research, Small Grants for Exploratory Research (SGERs) is not, in
    our view, effectively addressing the innovative research issue.
    Although program officers have considerable latitude to employ SGER
    grants to foster innovative research to counter what might be
    unwarranted caution in review panels, in fact SGERs are used
    relatively rarely. Foundation-wide, divisions may use up to 5
    percent of their budget on SGER grants, but in reality only 0.4
    percent of these budgets are used in this way. In our view, SGER
    grants are not a significant fraction of the overall portfolio
    therefore, they may not be playing a significant role in increasing
    the amount of highly innovative research. The reason(s) for this is
    (are) unclear. We encourage NSF to re-examine the purpose and use of
    SGER grants.

-   Third, it does not appear that clear data exist which demonstrate
    that NSF either does or does not fund enough innovative research.
    With respect to the TOOLS portfolio, we found that many of our
    nuggets indeed reflected bold/innovative research efforts. On the
    other hand, some directorates that use a number of different
    mechanisms may not be making such awards with a full understanding
    of the implications for the entire portfolio, or, conversely, may
    not be using the full suite of mechanisms available to them to
    encourage and fund innovative research efforts. The bottom line is
    that it is important to have a clear definition in hand as the
    necessary precursor to collecting reliable data to form a more
    accurate picture of the portfolio mix with respect to innovative or
    transformative research.

[Multidisciplinary Research Projects]{.underline}: More and more,
forefront science sits between traditional disciplines, and some of the
more innovative ideas involve investigators from very different fields
collaborating on "terra incognita." NSF has a structure that, for the
most part, has been established to fund single principal investigators.
While many of the new, targeted solicitations and priority areas
encourage or require multidisciplinary activities, these are often
short-lived programs (e.g., Information Technology Research, Nanoscale
Science and Engineering, and Biocomplexity in the Environment). We
encourage NSF to develop ways to encourage and fund multi- and/or
inter-disciplinary activities through its ongoing programs. .

We point out the difficulty of parsing projects to fit into a single
indicator "box". Many of the large, NSF-funded centers and networks
impact many indicators not only in the TOOLS strategic goal, but
sometimes including indicators from the IDEAS and PEOPLE strategic
goals. The Network for Earthquake Engineering Simulation (NEES) is an
example. Since we will refer to it several times, to minimize repetition
in the text, we describe it here, before we turn to the individual
indicators.

From the Pacific coast to our nation\'s interior, more than 75 million
Americans in 39 states live in towns and cities at risk for earthquake
devastation. While scientists are digging into the origins of seismic
waves, engineers are pushing the boundaries of design to create
structures that remain safe when an earthquake ultimately occurs. The
George E. Brown, Jr. Network for Earthquake Engineering Simulation
(NEES)
([**0126366**](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0126366)[,]{.underline}
[**0117853**](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0117853),
and
[**0402490**](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0402490))
integrates 15 experimental facilities located at academic institutions
across the United States, including shake tables, geotechnical
centrifuges, a tsunami wave basin, large strong floor and reaction wall
facilities with unique testing equipment, and mobile and permanently
installed field equipment.

NEES is a major state-of-the-art facility and important for the
discipline (Indicator T1). Indeed, it goes beyond the state of the art
in developing the prototype of next-generation ways of doing science
(Indicator T2). It is a distributed \"virtual instrument\" for
earthquake engineering research ([www.nees.org]()). It has enabled a
large community of earthquake engineers, computer scientists, and other
disciplinary specialties to share resources in a unique way. It provides
the necessary tools for remote data acquisition, for sharing data
through metadata management software, for remote simulations, virtual
laboratories, even for telepresence. The interface is friendly enough to
support K-12 teachers and be usable by the general public. This effort
is serving as a model for other distributed scientific instrumentation.
As such, it demonstrates the potential for cyberinfrastructure
(Indicator T3) to transform the way that researchers do research and
that teachers teach.

**Other Important Issues**

[Expanding the NSF community beyond research-focused
institutions]{.underline}: In the past, research-focused institutions
have stood out as being the primary recipients of NSF funds. In order to
meet the future needs of the nation for scientists, engineers, and
technically trained people, NSF must redouble its efforts to expand its
constituency to include predominantly undergraduate institutions
(typically teaching-intensive) and minority-serving institutions as well
as research-focused institutions. NSF has made significant progress on
building infrastructure capacity at many of these other types of
institutions. However, it is clear that a primary barrier to making
continuing progress towards enhancing the research capacity at
institutions educating a large percentage of underrepresented groups is
the high teaching workload of faculty at these institutions. The NSF
should examine the relative balance of its investments in aimed at
enhancing infrastructure, encouraging student pursuit of STEM fields,
and supporting the professional development of faculty in the community
colleges, predominantly undergraduate, and minority-serving
institutions.

[Sustainability]{.underline}: We continue to be concerned about the
sustainability of a number of the tools developed with NSF funding. This
issue was also raised by several COV reports that we reviewed. For
example, databases whose collection, organization and initial
presentation, often on web sites, must be maintained after the duration
of the grant or upon departure of a PI, graduate student or other
technical staff from the institution that hosts that database. Another
example would be facilities that are funded and built to provide access
to a user community, but then the funding is reduced or eliminated
either due to termination of the program (for example, large projects
funded by Information Technology Research) or funding cycle. The
accomplishment descriptions of the various projects mentioned in our
report do not provide any indication about what the institution will do
when the NSF funding runs out. We suggest that merit review panels
should, where appropriate, consider the quality of the proposer\'s plan
for the long-term sustainability of the site or facility.

[TOOLS GOAL \-- Indicator T1:]{.underline} Expand opportunities for U.S.
researchers, educators, and students at all levels to access
state-of-the-art S&E facilities, tools, databases, and other
infrastructure.

NSF supports and provides a wide variety of accessible, state-of-the-art
science and education facilities, tools and infrastructure, and in most
cases is the only support for such instrumentation in academia. These
tools provide opportunities for researchers, educators, students,
citizens and policymakers. NSF supports large state of the art
facilities, and nearly all of the US's land based astronomical
facilities, and tools that push the forefront of science and
engineering. It supports databases and acquisition/analysis software
that present and synthesize large amounts of data collection by numerous
researchers around the US and the world. Through a variety of funding
mechanisms on different scales, NSF addresses both the needs of
researchers to have and develop facilities and infrastructure that
enables scientific discovery and educators to develop innovative means
of disseminating science to students and the public. NSF has made a
significant achievement with respect to indicator T1. A selection of
examples follows:

Through FabLab, which is an educational outreach component
([0122419](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0122419)[),]{.underline}
the Center for Bits and Atoms provides outreach facilities to bring the
ideas of fabrication and micro-manipulation to the US public and
includes modules in Kenya and South Africa. The tools of this large
center are made available to the public through these activities and the
Fab Lab serves as a model for Centers that are more than the total of
all the science that occurs there because the science is disseminated
and brought to the public in meaningful, hands-on methods. CBA\'s
laboratory research on technologies for personal fabrication is
complemented by the field \"Fab Lab\" program. The FabLab brings
prototype capabilities to under-served communities that have not had
access to the reach of conventional and modern technology development
and deployment.

"Expanding National Library of Virtual Manipulatives (NLVM) Reaches U.S.
and International Audiences of K-8 Math Learners"
([0352570](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0352570)[)]{.underline}
and "National Library of Interactive Web-based Virtual Manipulatives for
K-8 Mathematics"
[(]{.underline}[9819107](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=9819107)),
are projects that enhance the mathematics education in grades K-8 in
both the US and abroad. This project provides on-line, web based tools
and databases that have more than 1 million hits a day as students
access the information on the web. Not only does the program provide
state-of the art educational tools for students, but it also provides
pre-service teacher training in a field where innovation on a K-8 level
that is solid and rigorous is hard to come by. Accessibility will be
increased as well as the team is working on creating a version in
Spanish. The outreach of this activity is expanding, has free access and
can help to increase mathematics literacy by providing manipulatives via
the internet that are formal curriculum tools as well as informal
learning environments.

Materials Science as a field has developed to the point where scientists
are beginning to predict macroscopic properties from atomic or
microscopic structure. However, in order to have this capability, the
tools of cyberscience---algorithms and computational expertise---are
needed. Researchers at the University of Illinois at Urbana Champaign
have begun to address this important cyberinfrastructure need by
developing software and education cyberinfrastructure
([0325939](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0325939)[)]{.underline}.
From a small group award, the seeds of this idea grew to a larger
proposal in FY03 awarded through the Information Technology Research
solicitation and is funded through the Division of Materials Research
with co-funding from the Chemistry Division and the Division of
Computing and Communications Foundations in the Computer and Information
Sciences and Engineering Directorate. This program has provided software
dissemination openly, developed new software tools, and hosted a
workshop to promote the exchange of ideas and new advances in algorithms
for computational materials research and a computational summer school
to help train the next generation of computational materials researchers
in state of the art computational methods. This project is an example of
the cyberscience tools being developed through the NSF that will enable
the forefront science of the next generation.

The Protein Data Bank
([0312718](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0312718)[)]{.underline}
promotes international cooperation and is the authoritative,
international repository for 3D structural information for biological
macromolecules. Indeed, anyone in the US or abroad looking for the
structure and classification of a protein can access all published
information on the web. This database not only provides information, but
is coupled with tools for visualizing the protein structure as well. In
addition, storage of this data, archiving and backup is pushing the
frontiers of international collaboration as well as the issues of
permanent or long term storage and ownership/responsibility for long
term maintenance. Suzanne Richman from Rutgers University writes about
her work in Japan on this project \"Despite our differing cultures and
languages, working at PDBj felt like home. We are all working on the
same project, half a world apart, but with the same thoughts and
feelings about it, and in an annotation room that can be just as eerily
quiet, as we all work and concentrate hard.\" Science can bring people
together and break down barriers of language, culture, and geography.
The Protein Data Bank provides an excellent example of the unifying
force of science.

The Cyber Defense Technology Experimental Research (DETER) Network
([0335298](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0335298))
is a facility funded by NSF. The DETER network and test-bed serves as a
center for interchange and collaboration among security researchers, and
as a shared laboratory in which researchers, developers, and operators
from government, industry, and academia experiment with cyber security
technologies under realistic conditions. It provides an infrastructure
that would not otherwise exist to both aid in the development of tools
for protecting cyberinfrastructure and for training students and the
next generation of cyberinfrastructure researchers. This is a unique
facility with broad outreach to a diverse community involved in network
security evaluation.

Microsystems Packaging is a key component of all consumer electronics,
and yet as a field has not yet been developed. This innovative program
has developed textbooks, innovative curricula and develop and important
field
([9402723](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=9402723)).
Students from the program at Georgia Tech have been highly sought by
industry. Two of the largest professional societies, IEEE and IMAPS have
helped develop 15 new courses for the Internet that are accessible
internationally. This access as well as the adoption of the textbook at
47 universities shows the importance of MSP and the need for the tools
and curricular databases provided by this program.

Through a joint collaboration between U.S. and Indian astronomers, a
spectroscopic fingerprinting of over 1200 stars has been funded and will
be provided openly to the scientific community
([0114536](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0114536)).
This is a huge undertaking as the current largest star mapping is about
200 stars. This library will include spectral data over the largest
wavelength range available as well. The star library is a unique data
resource for our international scientific collaborations, as for the
whole astronomical community. The scientific potential of the library is
that certain spectra can be used as building blocks for analyzing the
evolution of galaxies. As a database, this library will be unparalleled
in the astronomical community.

[TOOLS GOAL \-- Indicator T2]{.underline}: Provide leadership in the
development, construction, and operation of major, next-generation
facilities and other large research and education platforms.

The development, construction, and operation of major, next-generation
research facilitates many essential discoveries that advance fundamental
knowledge and enhance the American economy. Innovative facilities and
research tools often open unique opportunities for collaborative
research across institutions, nations, and disciplines. Facilities and
other large research and education platforms provide the long-term
infrastructure for creating new knowledge that serves society. The NSF
has made significant achievement in providing leadership in the
development, construction, and operation of major, next-generation
facilities and other large research and education facilities.

The Arctic Ocean is a crucial region determining the present and future
state of the world's oceans and climate. The extreme conditions of the
Arctic environment have limited scientific observations to a relatively
few locations and seasons of the year. The design and implementation of
**an observational array for Arctic oceanographic measurements through
"An Observational Array for High Resolution, Year-round Measurements of
Volume, Freshwater, and Ice Flux Variability in Davis Strait"**
**([0230381](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0230381))
will provide a highly integrated and interdisciplinary perspective on**
the role played by the Arctic and sub-Arctic in steering decadal scale
climate variability. The observing system to will provide the first
year-round measurements of the total water volume, influence of
freshwater, and ice fluxes across Davis Strait between Greenland and
Canada. The ocean, ice, and atmospheric observations from this facility
will be essential for understanding and documenting the influence of
future climate variability and change on Arctic environments.

In February 2004, the Global Biodiversity Information Facility (GBIF)
([**0301149**](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0301149))
went online with a prototype data portal (www.gbif.net) that provides
digital access to data from the world's natural history collections,
herbaria, culture collections, and observational databases.
Participation in the GBIF consortium is open to any country or relevant
international organization. The consortium currently consists of 72
participating institutions. This revolutionary capability for sharing a
treasure of unique data collected from important ecosystems across the
entire planet will promote scientific collaboration and dramatically
improve fundamental understanding of the state of the world's
biodiversity. Science and society stand to gain much from the GBIF data.
Data mining will turn up gems of insight and understanding that cannot
be predicted but are likely to lead to fruitful new directions for both
research and commercial applications of natural substances. Such
insights are vital to creating better futures for both people and
nature.

From the Pacific coast to our nation\'s interior, more than 75 million
Americans in 39 states live in towns and cities at risk for earthquake
devastation. While scientists are digging into the origins of seismic
waves, engineers are pushing the boundaries of design to create
structures that remain safe when an earthquake ultimately occurs. The
George E. Brown, Jr. Network for Earthquake Engineering Simulation
(NEES)
([**0126366**](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0126366),
[**0117853**](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0117853),
and
[**0402490**](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0402490)),
integrates 15 experimental facilities, located at academic institutions
across the United States, including shake tables, geotechnical
centrifuges, a tsunami wave basin, large strong floor and reaction wall
facilities with unique testing equipment and mobile and permanently
installed field equipment. A NEESgrid connects these experimental
facilities via the Internet2 to form the world\'s first prototype of a
distributed \"virtual instrument\" for earthquake engineering research
([**www.nees.org**]()). NEES also provides national resources for
developing, coordinating, and sharing new educational programs and
materials to excite and support future generations of the earthquake
engineering workforce.

Scientists and engineers at the University of Texas at Austin, Center
for Space Research (CSR), Mid-American Geospatial Information Center
(MAGIC) lead the development of cyberinfrastructure that rapidly
integrates and distributes crucial environmental, engineering, economic,
and social data necessary to disaster mitigation, response, and recovery
in their project "Extensible Terascale Facility (ETF): Enhancing the
Capabilities, Scope and Impact of the Extensible Terascale Facility"
([**0338629**](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0338629)).
This timely and usable information is quickly provided to state and
federal agencies, regional and local governments, academic institutions,
and the public. This accomplishment is a stunning example of translating
fundamental earth science observations and research into operational
uses that will reduce the loss of life and property caused by hurricane
winds, storm surges, tsunamis, floods and other disasters. This project
involved collaborations with the Texas Advanced Computing Center, Oak
Ridge National Laboratory, and Purdue University.

The Cyber Defense Technology Experimental Research (DETER) Network
([**0335298**](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0335298))
is a new center for collaboration among information technology
networking and security researchers. This facility encourages
collaborative research and education efforts - across organizations and
disciplines - by involving six universities and four industrial
institutions in an effort that spans both networking and security
issues. This project provides leadership in the future networks and
computational infrastructure that will be necessary to the emerging
knowledge society. This project also expands opportunities for U.S.
researchers, educators, and students at all levels to access
state-of-the-art network security evaluation infrastructure.

[TOOLS GOAL -- Indicator T3:]{.underline} Develop and deploy an advanced
cyber infrastructure to enable all fields of science and engineering to
fully utilize state-of-the-art computation.

Cyberinfrastructure constitutes the research environments that make
advanced computation, data acquisition, and collaborative services
available through high-speed networks. NSF has built up the country\'s
cyberinfrastructure through a variety of programs, notably the PACI
(Partnerships for Advanced Computational Infrastructure) supercomputer
centers, the Middleware Initiative, the Information Technology Research
(ITR) program, and the Teragrid.

There was significant achievement in the cyberinfrastructure goal
through the combination of these facilities, indicated by progress in
several funded activities, falling roughly under two headings:

-   [Successful applications of the existing
    cyberinfrastructure]{.underline}. Several project accomplishments
    attest to how the recently developed infrastructure is supporting
    many scientific projects, from access to astronomical surveys (the
    National Virtual Observatory), to parallelizing existing useful
    software, like the Harvard CHARMM code for molecular mechanics. Two
    nuggets are exemplary in this regard: "Computing dark energy" and
    "Using Grid platforms to better understand neuro-transmission."

-   [Development of new tools to extend the reach of the
    cyberinfrastructure]{.underline}. We highlight two project
    accomplishments among the several that fall into this category:
    "Rocks Cluster Management Software" and "Workflow Scheduler for
    Distributed Computation."

The greatest concern is stable funding and management of these resources
in the future.

Successful applications:

The recent conclusion that the expansion of the universe is accelerating
likely due to the presence of "dark energy" was initially supported only
from supernova data. Now a second line of evidence from the projects
"Statistical Data Mining for Cosmology" and "Searching for Correlations
in a High Dimensional Space"
([0121671](https://www.fastlane.nsf.gov/servlet/showaward?award=0121671)
and
[0312498](https://www.fastlane.nsf.gov/servlet/showaward?award=0312498))
bolsters the same conclusion, based on the so-called Sachs-Wolfe effect.
The faster expansion rate of a universe that contains dark energy would
leave its mark on photons that gain energy passing by gravitational
potentials. This effect has been observed with the help of statistical
data mining algorithms developed to search the massive astrophysical
surveys.

MCell is a Monte Carlo simulator of cellular microphysiology. It
simulates the dynamics of biochemical reactions in 3D microenvironments,
and in particular, of neurotransmitters in synapses. Current demands are
of the order of 2CPU-months of computation and 35GB of memory. The
project, "Virtual Instruments: Scalable Software Instruments for the
Grid"
([0086092](https://www.fastlane.nsf.gov/servlet/showaward?award=0086092)),
altered MCell to MCell-K to permit it to run in parallel, distributing
the work onto large Grid platforms. Clusters at the San Diego Super
Computing Center, the Tokyo Institute of Technology, and the IBM
BlueHorizon supercomputer, were all used for large-scale simulations
previously unapproachable by serial MCell.

Development of tools:

There is a need to make stable and manageable parallel computing
platforms available to a wide range of science and engineering research,
as the project "National Partnership for Advanced Computational
Infrastructure"
([9619020](https://www.fastlane.nsf.gov/servlet/showaward?award=9619020))
may demonstrate. An impediment has been the difficulty of setting up a
cluster, and then managing it, e.g., ensuring all nodes have a
consistent set of software. Rocks addresses this need by making it easy
to create, manage, and upgrade a Linux cluster. The basic idea is to
make complete OS installation on a node the basic management tool, which
is faster and easier than determining the software synchronization of
all nodes. Rocks software clusters use a MySQL database for site
configuration. The software builds a cluster by installing a Linux suite
of software, and provides tools for easy upgrades and extensions. Rocks
has quickly developed an extensive worldwide user base, and won several
awards at the 2004 Supercomputing Conference.

Scheduling the flow of work in a distributed computation is a critical
issue for heterogeneous tasks, which are more challenging than
tightly-coupled parallel computations. This new workflow scheduler
([0331645](https://www.fastlane.nsf.gov/servlet/showaward?award=0331645))
seeks to minimize the "makespan" (overall job completion time). It
creates a task graph, and ranks each eligible resource against subtasks,
incorporating information (some automatically estimated) about
communication and memory costs. Optimization heuristics then choose a
mapping of components to nodes. Experiments indicate significant
improvement over randomized scheduling.

[TOOLS GOAL -- Indicator T4:]{.underline} Provide for the collection and
analysis of the scientific and technical resources of the U.S. and other
nations to inform policy formulation and resource allocation.

Our examination of the nuggets and other background information
indicates that the NSF and its grantees contribute to a great extent to
the national need for information needed to inform policies and budgets.
This information is produced in three basic ways, which we will discuss
in turn. First, the NSF\'s division of Science Resources Statistics
(SRS) and its contractors collect and interpret a great deal of
information themselves. Second, a variety of programs within the NSF
make grants that result in a number of databases that scientists,
educators, and citizens can use. Third, some grants made by the NSF
either deliberately or accidentally produce policy-related information
that is useful for dealing with specific issues. We find that that the
NSF program merits the designation of \"significant achievement\" in the
T4 area.

NSF\'s SRS unit gathers a great deal of data on Research and Development
(R&D) which forms the statistical basis for the familiar volume *Science
and Engineering Indicators*, published every other year under the
imprimatur of the National Science Board. Surveys cover such topics as
Industrial R&D, Federal Funds for R&D, Federal science and engineering
support to universities, colleges, and nonprofit institutions, academic
R&D, and science and engineering research facilities. SRS works with
other units of the Federal Government, most particularly the Census
Bureau, in developing these data.

A committee, convened in 2002 by the National Academy of Sciences,
reviewed the performance of SRS and issued a report in 2005. This report
contains 32 separate recommendations that largely deal with ways in
which SRS could improve and/or extend the kinds of data, which it does
collect (Brown, Plewes, and Gerstein 2005). Given the scope of this
group\'s review and the integrity of the National Academy review
process, our subcommittee chose to simply accept the positive review by
the National Academy committee at face value and did not make our own
independent evaluation of SRS.

The second way that the NSF supports the development of useful policy
data is to support projects that include as all or part of their mission
the development of websites that either contain some data themselves or
have links to websites that contain data. We cite here three examples of
such projects that came to the committee\'s attention as being examples
of particularly noteworthy endeavors.

Number range CHAPTERThe Math and Science Partnership program (MSP),
developed in conjunction with the President\'s "No Child Left Behind"
education initiative, has generated among other things a pooled database
of successful practices that will be very useful both to people within
the MSP community and beyond it, for example, in the project "Program
Evaluation for the Math and Science Partnership"
([0456995](https://www.fastlane.nsf.gov/servlet/showaward?award=0456995),
[0335334](https://www.fastlane.nsf.gov/servlet/showaward?award=0335334),
and
[0445398](https://www.fastlane.nsf.gov/servlet/showaward?award=0445398)).
Exploration of one of the many websites supported by this project
([[http://hub.mspnet.org/]{.underline}](http://hub.mspnet.org/))
revealed that already, only a few years after the MSP projects began,
there are a considerable number of papers presenting Results which are
of interest to practicing science teachers. The links were easy to
follow and information on particular areas of interest was easy to find.

Scientists and science instructors occasionally find themselves
interested in some very specific areas that suddenly come on to their
radar screen. For example, a university scientist who has been asked to
visit a school for deaf children would do well to visit the website of
the NSF-funded project COMETS (Clearinghouse on Mathematics,
Engineering, Technology, and Science). The COMETS website, developed at
the National Technical Institute of the Deaf, aims to contain virtually
everything ever published that is related to deaf education in STEM
fields
([0095948](https://www.fastlane.nsf.gov/servlet/showaward?award=0095948)).
This website
(<http://www.rit.edu/~comets/pages/featurespages/biblio/bibliopage.html>)
has information on a great deal of individual investigations, and a
complete list of scientists who were deaf or hard of hearing.

Another example deals with a particular environmental niche, the cold
regions of our planet. A scientist who had a need for information on the
work that had been done in arctic and sub-arctic regions of the planet,
and who was not already familiar with the network of literature and
investigators in this area, could simply go to
[[http://www.coldregions.org/)]{.underline}](../(http://www.coldregions.org/))
. This website, prepared with NSF support
([9909727](https://www.fastlane.nsf.gov/servlet/showaward?award=9909727))
apparently contains links to almost everything published on these parts
of the planet.

A third way that the Foundation supports policy studies is to support
basic research projects which not only have significant policy
implications but which seem to be influenced by the need to develop data
with policy implications. For example, Dr. Robbie Luliucci at Washington
& Jefferson College studies the aging of silica-reinforced polymers used
in weapons systems
([**9909727**](https://www.fastlane.nsf.gov/servlet/showaward?award=9909727)).
The materials that age are not the materials that explode, but the
plastic and rubber-like materials that are equally important to the
integrity of a weapon. Undergraduate students develop skills that can be
used in industry, particularly as is related to homeland security. As
another example, Marina Alberti of the University of Washington led an
interdisciplinary study of urban development, land-cover change, and
bird diversity, a study that could certainly be useful to any land-use
planner who was interested in the relationship between environmental
integrity and the intensity of urban or suburban development in any
particular area
([**0120024**](https://www.fastlane.nsf.gov/servlet/showaward?award=0120024)).

# 

# [TOOLS GOAL -- Indicator T5]{.underline}: Support research that advances instrument technology and leads to the development of next-generation research and education tools.

An important part of NSF\'s research strategy is to provide new and
advanced tools as a \"backbone\" that can position our nation to
investigate and develop \"next-generation\" research programs further
advancing science and technology. Perhaps of equal significance is the
training and development of students and academia to new methods and
processes that enable us to do things tomorrow that are just being
imagined today thus leading to \"development of next generation research
and education.\" We find that the NSF efforts in this area are worth
describing as \"significant achievements.\"

In evaluating the various research nuggets for 2005, it is clear that
there are many NSF-funded programs producing results for putting in
place new instruments that can and will provide opportunities for great
advancements in the fields of biology, medicine, materials, and computer
technology.

For example, at the Center for Bits and Atoms at MIT, scientists are
developing new methods that fundamentally will change the way a computer
works integrating both \"living\" software and hardware that changes to
meet the computational needs at hand. This program is innovative in that
it seeks to fundamentally revisit the notion of what a computer is, and
what a computation is. By taking a more holistic approach and a
radically new view of the process, the program seeks to revolutionize
the computing process. If successful, this program has the potential of
creating a new foundation for much more advanced computing and
management of much larger amounts of information at higher speeds than
ever before possible. This program expands the narrow \"hardware\" focus
of current computational techniques and methods. The vision is to
include in the computational approach \"context\" information. The goal
is to overcome the very real scaling limits of \"data crunching only\"
that creates an obstacle to designing and managing very large-scale data
and information systems. The program is high risk, multidisciplinary,
and has already achieved some positive results, one of which is a new
type of analog to digital converter. Under NSF funding for "Center for
Bits and Atoms"
([0122419](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0122419)),
Neil Gershenfeld and his team have produced an extremely
energy-efficient version of the versatile analog-to-digital converter.
Conversion of analog readings to digital signals is becoming extremely
important, not only in technology advancement, but also in everyday
life. This device applies new methods to increase speed of conversion
and energy efficiency over previous technology. This instrument has an
array of applications in the computer, automotive and communications
industries.

As another example, NSF has taken a leadership role in developing
nanotechnology and instrumentation. In the project "Nanotechnology Moves
into Production at IBM"
([0213618](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0213618)
and
[0213695](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0213695)),
Curtis Frank of Stanford University and Thomas Russell of University of
Massachusetts Amherst have developed a new tool for high-density
lithography. Collaboration with IBM scientists has led to the
application of this technology to increasing the lifetime of flash
memory over 100-fold, compared to previous technology. As with the
analog-to-digital converter, this application of nanotechnology is
important in technological research as well as improving everyday life.

Carl Wieman and Thomas Perkins, at the University of Colorado at
Boulder, in their project "Watching Proteins Bend DNA with Subnanometer
Resolution"
([0404286](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0404286)
and
[0096822](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0096822)),
have created another breakthrough nano-scale instrumentation as a tool,
which allows biologists to follow the motion of a single molecule. Until
this advancement, scientists needed to rely on the average of a set of
measurements on a group of molecules in order to study molecular
behavior. Now, molecular motion can be measured with ten-fold greater
resolution at times on the millisecond level. This project opens up the
opportunity to measure the motion of enzymes replicating.

Another new tool in the field of biology and medicine allows for early
detection of esophageal cancer. Adam Wax of Duke University in the
research project, "Low Coherence Light Scattering for Biophotonics"
([0348204](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0348204)),
developed a method, which has been proven successful in experiments with
rats, using the scattering of light to detect an enlarged nucleus, one
of the earliest signs of pre-cancerous cells. Measurements of light
scattering can be taken in 40 milliseconds, making diagnosis possible in
less than a second (compared to the many minutes it takes using current
methods). Time is of the essence in cancer diagnosis, so shorter
diagnosis times combined with earlier detection capabilities are great
strides in instrument technology for cancer treatment. Not only is this
a valuable platform tool, but also extension from rats to humans, if
successful, could save many lives.

At Carnegie-Mellon University, "Synchronized Transatlantic Synchrotron
Research"
([0079996](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0079996))
has yielded a new tool that can increase our ability to predict and
control the properties of ceramic and metallic materials. The instrument
developed in this program uses x-rays that can penetrate through
centimeters of solid samples, allowing scientists to measure the shapes
and orientations of grains in the material and how they change with
time. An increased understanding of material structures and properties
can lead to improvements in fabrication of products from bridges to
microscopes to prosthetics.

The Materials Research and Science Engineering Center
([0079996](http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0079996)),
as well as the previous four projects described, advances in instrument
technology are creating opportunities to better understand and improve
products and processes in the fields of biology, medicine, materials,
and computer technology. Thus, through achieving success in Indicator
T5, NSF-funded programs are enabling "discovery, learning and
innovation," one of the National Science Foundation's five main goals.

Reference

Brown, L.D., Plewes, T.J., and Gerstein, M.A. (eds.). 2005. Measuring
Research and Development Expenditures in the U.S. Economy. Washington,
D.C.: National Academies Press.

## 

## ORGANIZATIONAL EXCELLENCE (OE) Strategic Outcome Goal

An agile, innovative organization that fulfills its mission through
state-of-the-art business practices. NSF is successful when significant
achievement is demonstrated for the following performance indicators:

-   ***Human Capital Management:** Develop a diverse, capable, motivated
    staff that operates with efficiency and integrity.*

-   ***Technology-Enabled Business Processes:** Utilize and sustain
    broad access to new and emerging technologies for business
    application.*

-   ***Performance Assessment:** Develop and use performance assessment
    tools and measures to provide an environment of continuous
    improvement in NSF's intellectual investments as well as its
    management effectiveness.*

-   ***Merit Review:** Operate a credible, efficient merit review
    system*

**[Introduction]{.underline}**

The OE strategic outcome goal was added to the NSF Strategic Plan for
FY2003-2008. This was a major step forward in recognizing the linkages
between excellence in advancing science and excellence in organizational
development. Within the OE goal, the indicators "mirror" the people,
ideas, and tools structure of the other strategic outcome goals. The
Human Capital indicator is the "people" dimension of OE, the
Technology-Enabled Business Processes is the "ideas" dimension, and the
Performance Assessment and Merit Review indicators together are the
"tools" dimension.

At the time the OE goal was added, it was determined that the Advisory
Committee for Business and Operations (AC/B&O) would provide an
assessment of the first three OE indicators (Human Capital,
Technology-Enabled Business Processes, and Performance Assessment) and
the AC/GPA would conduct an assessment of the Merit Review indicator. As
part of its OE analysis, the AC/GPA reviewed data and information from:

1.  ***Report to the National Science Board on the National Science
    Foundation\'s Merit Review Process \-- Fiscal Year 2004 (NSB
    05-12).*** This is a report on the statistics and processes related
    to the proposal and award activity for the fiscal year 2004 and the
    merit review process (number of proposals received, numbers of
    reviewers, etc).

2.  ***Committee of Visitor (COV) Reports.*** NSF convenes panels of
    relevant community experts to maintain high standards of program
    management and to provide advice for continuous improvement of NSF
    performance, in part through their assessment of how NSF programs
    evaluate and process proposals (i.e., merit review). Each NSF
    program undergoes a COV review approximately every 3-5 years. These
    COVs are also asked to comment on organizational excellence and
    multidisciplinary and high-risk proposals. Table 1 shows which NSF
    areas had COVs in 2004 (shown in bold type). Directorates decide
    what level to do these COV reviews (e.g., program, division,
    directorate, etc). The schedule for all COV program evaluations and
    COV report template and questions can be found in the 2004 NSB Merit
    Review report in Appendices 12 and 13.

3.  ***The NSF Advisory Committee on Business and Operations 2005
    Assessment of Organizational Excellence.*** This report looks at the
    non-merit review elements of organizational excellence (human
    capital, business processes, and performance).

**[2005 AC/B&O Assessment]{.underline}**

The 2005 AC/B&O assessment supports NSF's conclusion that the agency has
demonstrated significant achievement for the three indicators it
considered (human capital, business processes, and performance
assessment). The AC/GPA agrees with this conclusion. The AC/B&O also
made a number of comments to improve the approach, methodology and
analysis for the assessment of performance in subsequent years. The
AC/B&O report can be found in an Appendix to this report. For our part,
we conclude that the MRP is effective in the processing and reviewing of
a large and increasing volume of proposals and in the engagement of a
broad and diverse segment of talent in the NSF's science and engineering
enterprises. While the MRP will always, in our view, require vigilance
and a commitment to continuous improvement, when taken as a whole and
when one looks at the results as illustrated in the People, Ideas, and
Tools portfolios, clearly, the process remains a major positive force in
advancing the frontiers of science, mathematics, and engineering. [From
this review, we concluded that NSF has demonstrated significant
achievement for this OE indicator.]{.underline}

**[2004 Committee of Visitors OE Assessment]{.underline}**

The 2004 COVs listed in Appendix 2 were asked to comment specifically on
NSF\'s ability to provide \"an agile, innovative organization that
fulfills its mission through leadership in state-of-the-art business
practices\" (i.e., organizational excellence). The COVs were also asked
to comment on the merit review process (MRP), which will be discussed in
the next section. While less than half of the COVs actually provided OE
comments (something NSF needs to work on), those COVs that did answer
concluded that the respective programs were meeting the OE challenge.
Some comments and issues of note include:

-   **COV Process.** The COV process itself was cited by COVs as an
    innovative practice that ensures the quality of NSF\'s grants and
    awards.

```{=html}
<!-- -->
```
-   **Crosscutting Programs.** Crosscutting programs involve NSF staff
    distributed in diverse directorates, divisions, and programs (e.g.,
    Biocomplexity in the Environment involves seven directorates and two
    offices). The COVs found the crosscutting management structure,
    while not excellent, is \"good enough\". NSF staff has learned to
    use the current crosscutting structure even though it is cumbersome.
    NSF should examine how its internal processes could be improved for
    this kind of matrixed management. Such an analysis should include
    staffing requirements, fund management, decision-making, time use
    efficiency, preparation and review of program announcements, and
    outcome assessments. Most NSF staff involved in these crosscutting
    programs does so in addition to their normal disciplinary program
    responsibilities. Streamlining processes and providing program
    assistance would help reduce or manage this often overwhelming
    workload.

-   **Benchmarking.** One COV suggested that NSF should consider
    periodic benchmarking with other government agencies, academic, and
    private sector organization to maintain state-of-the-art business
    practices.

-   **Impact on External Organizations.** A few COVs mentioned the
    impact that NSF\'s financial management requirements have on other
    organizations (e.g., universities, school districts, etc.) and
    encouraged NSF where possible to provide these organizations with
    much simpler and more "user-friendly" financial management
    guidelines to avoid additional administrative costs.

-   **Review Process.** A few COVs noted that the increased reliance on
    panel reviews over *ad hoc* mail reviews has enhanced the timeliness
    of reviews. COVs also noted that the reliance of the review process
    on leading scholars, the rotation of Program Officers which leads to
    the infusion of new ideas, and improvements to Fast Lane have
    substantially contributed to OE.

-   **Program Officers.** Many of the COVs concluded that effective
    Program Officers (POs) were pivotal in ensuring OE and described POs
    as dedicated, hard-working, and innovative staff who are agile and
    as responsive as the Congressional budget process will permit (i.e.,
    lack of needed funds, appropriations late in the year, etc). Several
    COVs cited POs who have been particularly creative in developing
    programs to address areas of increasing importance (e.g., in
    response to 9-11). NSF should recognize these outstanding POs and do
    as much as possible to provide training and mentoring to new POs.

**[2004 Multidisciplinary Committee of Visitors
Assessment]{.underline}**

In its 2004 report, the AC/GPA discussed the effectiveness of the MRP
for the review of interdisciplinary or multidisciplinary proposals. At
that time, we debated without resolution whether the organizational
structure of NSF (relatively autonomous directorates with disciplinary
divisions) promotes effective reviews and uniform processes for such
proposals. We now suggest that this issue is ripe for additional
discussion by NSF's senior leadership with an eye toward creating
consistent review practices for these types of proposals across the
Foundation. For 2004, the COVs mentioned in Table 1 were asked to
comment on whether NSF program portfolios have an appropriate balance of
multidisciplinary proposals. Most of the COVs reports on this topic
replied either "yes" or "appropriate," or provided few comments. In
future years, NSF should work more closely with the COVs to address this
topic more fully. Of the comments that were received, two themes
emerged:

-   **Program Officers.** The ability of a multidisciplinary proposal to
    be successful is a direct reflection of the willingness of POs to
    consult with experts outside their range of expertise when
    collecting reviews and to think broadly about how their programs
    relate to the broader needs of science and society.

-   **Budget Constraints and Review Process.** The budget environment
    may be acting as a constraint on the amount of multidisciplinary
    work that can be funded; further the review process does not
    preclude funding these projects, but it also makes no special effort
    to single them out.

The AC/GPA believes that neither the budget nor the review process
should create barriers to funding high quality multidisciplinary
proposals.

#  [2004 Merit Review Process Assessment]{.underline}

In general, the MRP continues to be impressive and effective. Some
statistics and issues taken from the NSB merit review report and COVs on
merit review (Table 1), include:

-   **Proposal and Success Rate.** In 2004, over 96% of NSF\'s awards
    were selected through the competitive MRP. NSF took action on 43,851
    competitively reviewed proposals (40,075 in 2003, 35,165 in 2002,
    and 31,942 in 2001) and provided funding to 10,380 of them. While
    the number of awards funded has not changed much in the past few
    years, the number of proposals acted upon has risen by 38%
    since 2001. This resulted in an overall NSF-wide average funding
    rate of 24 percent (27% in 2003, 30% in 2002, and 31% in 2001). The
    2004 funding rate is the lowest rate in 15 years and largely due to
    the substantial increase in the number of submitted proposals and
    budget constraints. The funding rate ranges between 16-45% depending
    on the NSF program. Funding rates for female PIs is one percent
    above the overall average and one percent below this average for
    minorities. The funding rate for new PIs is 17% (down from 19% in
    2003). Some COVs cited this declining rate as also contributing to
    larger numbers of proposals being resubmitted and re-reviewed with
    little gain in quality and further burdening the review process.
    However, in response to questions raised by the AC/GPA in 2004, NSF
    analyzed resubmitted proposals and the data show that these
    proposals are not disadvantaged in the review process -- thus, the
    issue for NSF is one of proposal volume.

-   **Sector Distribution.** The distribution of NSF awards among
    institutions of varying size remains relatively unchanged over the
    past four years. The top 100 universities still receive the
    overwhelming majority of all NSF awards (74 percent). The TOOLS
    subgroup also commented on this issue.

-   **Grant Size and Duration.** The average annualized award amount for
    individual investigator research grants in 2004 was \$139,522 (3
    percent more than in 2003 and 22 percent more than in 2002). The
    average award duration for 2004 was 2.96 years. NSF\'s goal has been
    3.0 years.

-   **Proposal Reviews.** Proposals are reviewed primarily by three
    mechanisms: *ad hoc* mail reviews only, panel reviews, and
    mail-panel combination. The use of the panel and mail-panel
    combination is increasing while the use of mail reviews only
    continues to decrease (down to 7% in 2004.) Including all mechanisms
    of review, the average number of reviews per proposal for 2004 was
    6.0, slightly lower than the 2003 level of 6.3.

```{=html}
<!-- -->
```
-   **Reviewer Pool.** Selected from the electronic database of about
    300,000 reviewers, 58,000 reviewers participated in the MRP, and of
    those, 13,000 were first time reviewers (up from 8,000 in 2003). In
    2004, 7092 or 17% of the distinct 41,263 reviewers returned
    demographic information. Out of these, 2,449 or 35% indicated they
    were members of an underrepresented group (i.e., minority or women).
    Provision of demographic data is voluntary and, given the low
    response rate, there is not enough information to establish a
    baseline. Thus, there is no apparent way to judge balance between
    racial, ethnic, and gender representation among the reviewer
    population. The AC/GPA is concerned about this situation both for
    reviewers and for PIs and urge NSF to redouble its efforts to
    collect this data and engage a diverse pool of reviewers in the MRP.
    We understand that the PI database and the reviewer database are
    currently separate, but easy and comprehensive retrieval of
    systematic and holistic demographic information about reviewers and
    PIs remains desirable and necessary. There were also concerns in
    many of the COVs about the potential for overuse of reviewers (i.e.,
    burnout, low response rate, etc) and the impact on the reviewer pool
    from the substantial increase in the number of proposals. COVs made
    a number of suggestions on improving the reviewer pool:

    -   Include private sector reviewers especially when tool
        development is a major objective of a proposal to ensure that
        proposals benefit from private sector expertise and are not
        duplicating projects already underway in the private sector.

    -   Create an on-line system to allow potential new reviewers to
        register, describe their interests, and be certified that they
        are prepared to be reviewers.

    -   Promote the benefits of reviewing to prospective reviewers
        (e.g., there is no better way to learn how to write a good
        proposal) and develop ways to publicly recognize reviewers
        (e.g., distinguished service award). This could help change the
        social norm of reviewing from a burden to something that is
        valued by one\'s peers (the same way teaching awards have
        changed the norm in many universities). A panel provides the
        opportunity to bring developing scientists into contact with
        established researchers and help these developing researchers
        obtain insights about proposal preparation and review standards.

    -   Seek more flexibility and mechanisms for appointing review
        panels, such as having six standing members on three-year
        rotations and two short-term members appointed on the basis of
        proposals received for each round of solicitations.

    -   Increase the number of young qualified reviewers and ask panel
        members to provide lists of review candidates.

    -   Encourage Program Officers to write journal articles to help
        increase reviewer return rates.

-   **Reviewer Response Rates.** In 2004, 59% of requests for mail
    review elicited positive responses, up slightly form 58% in 2003.
    Many COVs mentioned the decline in review return rates and thought
    that this was attributed to the great number of proposals and fewer
    available reviewers. Many COVs mentioned the impact of this low
    reviewer return rate and that it could limit the number of outside
    reviewers, particularly if those who do respond are only somewhat
    familiar with the research area. Several COVs postulated that some
    the lower return rate might be related to a \"Fast Lane learning
    curve\" and that Fast Lane has shifted the burden on reviewers to
    download and print proposals. Some COVs suggested that further
    testing be done on Fast Lane to help non-technical people use it \--
    although the AC/GPA has not seen the Fast Lane interface as an
    impediment. NSF might consider modifications to Fast Lane that would
    provide a convenient way to automatically monitor responses by
    prospective reviewers and remind reviewers as deadlines approach.

```{=html}
<!-- -->
```
-   **Dwell Time.** The NSF goal of processing 70 percent of all
    proposals within six months of submission was once again exceeded in
    2004 (77 percent, about the same as 2003 and up from 63% in 2001).
    This achievement is particularly significant given that the number
    of proposals has increased 38% since 2001.

-   **Review Feedback.** COV reports also indicate that more specific
    feedback to the PIs is desirable, including strengths, weaknesses,
    and suggestions for improvement. Several of those reports
    recommended tutorials on review preparation with examples of helpful
    reviews (for prospective reviewers and panel members). In 2003, the
    AC/GPA mentioned the need to ensure the panel summaries in the
    proposal jackets fully capture the rationale for funding decisions
    and the make up of panel experts (e.g., short bio). This lack of
    concise summaries was mentioned again repeatedly in the 2004 COV
    reports. While this is may be a PO workload issue, the AC/GPA
    believes that it is very important that these rationales be included
    or the COVs process will be negatively impacted. There were some COV
    comments that the \"Summary\" section of the review forms varies
    considerably. In some reviews, the summary synthesizes all comments,
    while in others the summary simply reiterates reviewers' comments.
    The former is more desirable and NSF should make this clear to
    reviewers. One COV suggested that reviewers should take the
    perspective \-- \"if I were the PI on this proposal, what feedback
    would help me turn this into a successful proposal\". Some reviewers
    organize their reviews according to NSF criteria and others do not.
    NSF should insist on a consistent format that easily relates to the
    MR criteria.

-   **Merit Review Criteria.** One of NSF\'s original GPRA goals was to
    increase reviewer and PO attention to both review criteria
    (scientific merit and broader impacts). It has been noted in the
    past three AC/GPA reports that consideration of the broader impact
    of the proposed research (i.e., Criterion 2) continued to be
    somewhat inadequate. In 2004, 92 percent of all external reviews
    addressed aspects of both MR criteria (90% in 2003, 84% in 2002, and
    69% in 2001). In 2004, 236 proposals were returned without review
    due to failure to address both criteria (276 in 2003). There is
    clear improvement in the statistics on this issue.

> While most COVs mention this improvement, they also all continue to
> cite the uneven attention of reviewers to Criterion 2 because
> reviewers, proposers, and POs still don\'t fully understand and apply
> these criteria consistently. It was widely held by the COVs that the
> *broader impact* criterion needs better definition, since its meaning
> can vary substantially depending on the background and perspective of
> both proposers and reviewers. For example, how does one judge a
> proposal impacting 20 students at a small liberal arts college and
> another proposal impacting 1,700 students at a large university?
>
> Suggestions offered by COVs to improve the quality of Criterion 2
> include: (1) redesign Fast Lane so reviewers must fully address both
> criteria; (2) provide clear and repeated guidance to review panels to
> comment on both criteria; (3) better define the criteria; (4) make
> available models of good and bad reviews; and, (5) insist that program
> announcements specify what sorts of broader impacts are relevant to a
> particular program. Ideas for providing examples of broader impacts
> are presented throughout the COVs reports and the COVs suggested that
> NSF should consider conducting Panel Review Workshops at national
> meetings to help train and certify reviewers. The AC/GPA finds that
> the review of the *broader impacts* criterion remains a challenge for
> most reviewers and we recommend that NSF intensify its efforts on this
> issue.

-   **Program Officers.** Many of the COVs found the thoughtfulness and
    thoroughness of the POs\' review of proposals and reviews
    extraordinary. They balance competing priorities, manage the review
    process, and many other things. The AC/GPA recognizes that POs are a
    key element in the success of the MRP in that they select the
    reviewers, compose the panels, and manage the process of review. The
    typical PO processes 105 proposals each year and spends 55 percent
    of his/her time on the review process (source: Booz, Allen, Hamilton
    workload survey of NSF staff).

> Since this survey was the first of its kind ever performed, we have no
> basis on which to assess whether more than half of an average PO work
> year on proposal review is too much or too little, but it does seem to
> us an inordinate amount of time, especially given the other important
> duties that a PO should be attending to (e.g., program development,
> award management and oversight, outreach and communication,
> performance assessment). In the past ten years the budget of NSF has
> nearly doubled and the number of proposals has increased
> significantly, yet the number of NSF staff has only increased by four
> percent.
>
> The AC/GPA recommends that NSF continue to track the amount of the
> PO's workload that is devoted to the MRP and consider providing the
> POs with additional resources (e.g., portfolio assistants). Throughout
> the COVs there are references to outstanding POs. NSF should analyze
> why these particular POs are so effective, reward them, and develop
> training for other POs based on these findings. Some of the attributes
> of a good PO include: long-term experience in the MRP, well respected
> in the field, close ties to leading scholars, no perceived personal
> biases, and ability to run a very effective review process. COVs also
> mentioned the importance of non-rotator POs that bring \"long-term
> memory\" into the process and this should be balanced with the \"fresh
> ideas\" that rotators bring to NSF.

-   **Efficiency and Effectiveness.** Some COVs mentioned that for very
    small awards, the peer review process might be excessive and add
    delay and cost to the process. NSF might want to consider a level
    under which some proposals can be handled with a less cumbersome
    procedure. It was also suggested that the number of required
    reviewers should vary with the amount of requested funding.

-   **Lost Capacity.** In 2004, over \$2 billion of declined proposals
    were rated as high as the average rating for an NSF award (4.2 on a
    5-point scale). These declined proposals represent a rich portfolio
    of high quality, yet unfunded research and education opportunities
    that we hope will not be lost to the nation.

Based on our analysis of NSF's merit review process, we offer these
comments and recommendations that we hope will be helpful and that we
suggest NSF consider for additional action:

-   **Crosscutting Programs.** As crosscutting programs and initiatives
    increase, NSF must focus on the associated infrastructure and
    management to assure continued OE (i.e., the structure is balky and
    should be overhauled or streamlined).

-   **COVs.** The COV reports have proved to be extremely valuable in
    the AC/GPA analysis. However, COVs too often either did not respond
    at all or responded solely in monosyllables (i.e., yes or no)
    regarding the review process for the multidisciplinary proposals and
    OE. As the number of the proposals increases, the importance of
    effective program review by the COVs grows. Thus, NSF should ensure
    that all COVs address MRP effectiveness for multidisciplinary
    proposals and OE topics.

```{=html}
<!-- -->
```
-   **Program Officers.** The positive effects of outstanding POs are
    critical to the success of the MRP. NSF should develop and recognize
    these POs and use their best practices in the training and mentoring
    of new POs. Although the numbers of proposals have increased
    significantly, the staffing managing the MRP has not increased
    proportionately, causing heavy workloads for POs. NSF is encouraged
    to continue to track the PO workload and to assign resources based
    on these results.

-   **Budget Constraints.** The AC/GPA noted the COVs\' contention that
    the funding of multidisciplinary proposals is budget constrained or
    negatively impacted by the MRP itself. The ACGPA encourages NSF to
    not let the budget and/or review process limit the funding of these
    proposals.

-   **Proposal and Success Rate.** The 2004 funding rate is the lowest
    rate in 15 years and largely due to the substantial increase in
    proposals. Many of the declined proposals are rated well above
    average (\~\$2 billion) and are costly to develop. Thus, NSF will
    need to develop strategies on how to take advantage of this
    important national intellectual capacity and avoid discouraging the
    pursuit of important research and education efforts.

-   **Reviewer Pools.** Demographic data allowing the determination of
    the balance of racial, ethnic, and gender representation among NSF
    reviewers remain scarce. The NSF is urged to redouble its efforts to
    broaden the numbers and diversity of the review pools. Reviewer
    response rates are roughly flat and there are signs of reviewer
    overuse and burnout. NSF should consider developing an on-line
    system to register and certify reviewers and develop a mechanism for
    Fast Lane to automatically monitor reviewer responses and deadlines.
    NSF should also promote the benefits of being a reviewer.

-   **Review Criterion 2.** An imbalance in the quality and thoroughness
    of the responses to Criterion 2, the *broader impacts* criterion,
    persists. We recommend that NSF continue to focus on this issue,
    including considering conducting Panel Review Workshops at national
    meetings to help train and certify reviewers.

**[Assessment of Transformative/Bold/Innovative-High Risk Research and
Education Proposals]{.underline}**

In its 2004 report, the AC/GPA discussed the issue of whether the MRP
may filter out "high risk" and "innovative" proposals. The AC/GPA came
to no conclusion about this issue in the brief time they had to consider
it and encouraged NSF to study this issue. Over the past year, NSF
created a rubric to define this type of research (Appendix 3) based on
analyzing a set of 150 accomplishments that POs thought should be
considered as \"innovative-high risk\" (Table 3). The AC/GPA compared
the definitions to the 150 proposals and also reviewed COVs comments on
the use of the designation of \"innovative-high risk\" to proposals in
their reviews. From this analysis, the AC/GPA concluded the following
regarding this topic:

-   **Goal of the \"Innovation-High Risk\" Category.** The Committee
    considered the type of research that would be captured in this
    category. Much of this discussion was focused on whether an approach
    or theory is truly new, might be very difficult to pursue, and might
    offer a very high reward if successful. These criteria describe much
    of NSF's portfolio, therefore this category would be the \"exception
    to the rule" \-\-- not just projects that are inherently complicated
    and that push the intellectual or technical boundaries. In reading
    the program officer comments for the 150 selected nuggets (see
    Appendix 4), it was difficult to determine how many of these
    projects truly met the \"exception to the rule" test\-- largely
    because the POs didn\'t have a uniform set of criteria to consider.

```{=html}
<!-- -->
```
-   **Current NSF Draft Definitions and Program Officers' Comments.** We
    conclude that NSF has made a good start in developing a rubric to
    define transformative research. However, based on our attempts to
    apply the draft definitions to the 150 proposals, NSF should
    consider refining the draft rubric to make it clearer and simpler.
    For example:

    -   Types 1 and 3 might be combined. Out of the 150 PO comments,
        there were very few proposals that seemed to fit only type 3
        and, in our view, type 3 seems to be a subset of type 1.

    -   It was very difficult to distinguish between types 1 and 2.
        These two descriptions appear to mix innovative, risk, and
        return-on-investment (reward) in unintended ways. For example,
        the current definitions tie together \"novel and untried\",
        \"high reward and technical challenge\", and \"innovative and
        contrary to current theory\". Something could be very high
        reward and have little technical risk. That is why NSF should
        consider separating innovation, risk, and reward.

    -   It would seem that \"risk to society if not successful\" should
        be in nearly all NSF grants, whether they are "high risk" or
        not. While important, \"PI safety\" is not in the direct
        \"spirit\" of what we are trying to measure, it should be
        considered part of technical risk. \"Risk to society if a result
        is harmful\" is a very important issue and should be tracked \--
        but it might also be beyond the spirit of what we are trying to
        measure here. Thus, while types 4 and 5 are important \-- they
        are probably not as relevant to the particular designation of
        \"transformative/bold/innovative-high risk\" and NSF should
        consider eliminating them from this rubric.

-   **COVs Comments on Innovative-High Risk Research and Education.**
    For 2004, the COVs were asked to comment on whether NSF program
    portfolios have an appropriate balance of
    transformative/bold/high-risk proposals. Many of the COV reports on
    this topic either replied \"appropriate\" or provided few comments.
    In future years, NSF should work more closely with the COVs to
    address this topic more fully. It was also difficult to judge or
    apply this category to many of the 2004 programs reviewed by COVs
    (e.g., NSF-NATO post-docs, cyber scholarships for service, etc).
    COVs did note the following:

> **High Risk Definition.** Many of the COVs mentioned the need to
> further clarify the concept of \"risk\" or characteristics of a
> "risky" proposal so that reviewers can apply the rubric more
> effectively. Most COVs noted that in many cases funded proposals
> appear to be the more conservative ones, with some evidence within the
> proposal jackets of a tendency toward risk-aversion. Some COVs that
> examined very similar programs said that it was relevant to judge risk
> while others said it was not possible. A few COVs mentioned that panel
> reviews could have an inherent bias to seek consensus, which may
> overlook or deemphasize high-risk, high-potential projects.
>
> **Budget Constraints and Competing Priorities.** Several COVs said
> that NSF's ability to fund high-risk proposals was substantially
> reduced due to budget constraints and the many priorities that POs
> must balance (e.g., providing grants of sufficient size and duration,
> funding new and underrepresented investigators, core disciplines,
> etc). Many \"gems\" reside in the \"could fund\" (but not funded)
> category because the projects are controversial by nature.
>
> **Program Officers.** COVs noted the importance of the PO\'s role in
> funding high-risk proposals. Typically, high-risk proposals had
> favorable reviews on the basic science ideas, but low ratings due to
> reviewer skepticism regarding the feasibility of the project or lack
> of track record of the PI. In many cases, it was the intervention of
> the PO that provided that type of proposal a chance to succeed.

Based on our analysis, the AC/GPA recommends: (1) a simpler, yet
effective \"innovative-high risk\" definition, (2) more effective
guidance and training for POs to apply it, particularly given the
critical role POs play in supporting this type of proposals, and (3)
clearer guidance to POs that budget constraints should not influence
their ability to make reasonable investments in this type of proposal.

The AC/GPA suggests that NSF consider the \"innovation- risk-reward\"
criteria below to develop a revised rubric:

-   **Innovation.** Significantly more forefront, novel, and
    transformative than other proposals in the field.

-   **Risk.** The proposal may have significant technical risk
    (including risk to the PI safety), a high probability of failure, it
    may be untried, and/or it may be contrary to current theory. If
    there is significant risk to society that a result might be harmful
    (e.g., health, environmental, safety, etc), this should be
    explicitly documented, and flagged for a policy-level decision prior
    to funding.

-   **Reward.** The proposal has significant economic, intellectual,
    societal, etc. return on investment.

-   **Overall Rating.** A proposal would be considered \"innovative-high
    risk\" if it is clearly innovative beyond other proposals in the
    field, might have substantial risk, and offers the potential for
    outsized returns on investment.

If NSF finds the above idea promising, we urge that it be further
studied and tested internally to make sure that users of the revised
rating system find it to be an improvement. One challenge in this regard
will be to ensure that POs and COVs are able to use the new rubric in a
comparable fashion to facilitate comparisons across organizations and
time. It would make sense to do some retrospective analysis to look for
trends as part of the internal \"testing\". During our deliberations, it
was brought to our attention that the Department of Commerce NIST
Advance Technology Program may have developed this type of high-risk
rating approach and NSF should examine whether the NIST approach may be
applicable for NSF. We also discussed the value of exploring whether the
current Small Grants for Exploratory Research definitions and award
process needs to be updated. The SGER process was created some time ago
and might need to be updated to reflect current research trends.

**COMMENTS ON THE PROCESS AND THE COMMITTEE'S WORK**

The AC/GPA Committee is pleased with the continued improvements that NSF
has made in the process. Not only were we able to do much of our work in
advance of the meetings, but NSF also provided periodic updates on the
progress of each subgroup in its analyses of indicators. This Committee,
with NSF's invaluable assistance, has, we believe, modeled the very
organizational excellence behavior that we seek from NSF. We also note
that this committee is, perhaps, unique within the Federal government.
We commend NSF's past and current leadership for their foresight and
their commitment to an assessment process that is both thorough and
impactful.

Although the Committee believes the AC/GPA process is working
effectively, the Committee has some additional comments that we offer in
the spirit of the continuous improvement that distinguishes
high-performing organizations:

-   Over the course of our two-day meeting, many members of the
    Committee expressed the view that they would have benefited from a
    brief overview session on the first day that would have encompassed
    the philosophy and structure of NSF's Strategic Plan, the
    relationship of the four strategic outcome goals to the Plan and to
    each other and the relationship of the individual indicators in each
    strategic outcome goal to the achievement of that particular goal.
    We characterize this session as "peeling the onion," with each layer
    enriching and deepening our understanding of the context in which we
    are doing our assessments. This contextual information will be
    especially important in years where NSF has revised its Strategic
    Plan or where indicators have been modified.

-   Continuing in this vein, we again note that we are being asked to
    determine whether NSF has reached a threshold of "significant
    achievement" toward its goals. The principal source of evidence for
    this determination is the extensive and rich database of
    accomplishments (aqua: nuggets). However, we would like to know more
    about how these accomplishments relate to the indicators they are
    supposed to support. In the absence of more contextual information,
    we are often left wondering how strong the linkage is between the
    accomplishments and the outcome goals. It might be worth
    experimenting with a "feedback" loop in which NSF program staff
    assign project accomplishments to specific indicators, but then
    allow principal investigators to comment on how they believe their
    project supports (or doesn't support) the indicator for which it has
    been identified.

-   Some panel members felt it would be beneficial to have access to
    basic program data in a spreadsheet format that would permit some
    additional analysis for specific indicators. Additional data to be
    included in such a spreadsheet might include: division, directorate,
    program, project title, grant number, hyperlinks to project summary
    and annual reports, award amount award duration, numbers of direct
    participants, numbers of beneficiaries (such as students), and PI
    names and institutions. We recognize that this suggestion might not
    be workable in this form given the workload already inherent in
    creating and validating the accomplishments database. However, the
    Committee makes this suggestion with the hope that, over time, more
    information, especially about numbers and types of participants and
    beneficiaries can be added to the existing information about project
    accomplishments. While simply counting things (outputs) won't in and
    of itself drive a determination of significant achievement, it is
    very helpful to understand the breadth and depth of the impact of a
    program or project -- especially in the PEOPLE goal.

-   In addition, the Committee believes that it might benefit from
    access to external, contractor-performed formal evaluations of
    programs. These are common in some directorates, e.g., Education and
    Human Resources, but they also exist for other programs, e.g., the
    Science and Technology Centers. These evaluations could provide
    important information to the Committee during its review and
    deliberations. Since these reports are often quite voluminous, it
    would be helpful if contractors preparing them could be asked to
    include a summary discussion of how their evaluations relate to
    NSF's strategic outcome goals. This would increase not only the
    utility of such evaluations, but would also serve to "close the
    loop" between program development, execution and assessment.

```{=html}
<!-- -->
```
-   The Committee was honored and grateful for the opportunity to
    discuss our work with Director Bement. However, we suggest that this
    discussion might be more meaningful and, perhaps, more helpful to
    the Director if it occurred on the final day of our meeting. By that
    time, the entire Committee would have had time to synthesize and
    reflect on our preliminary findings and recommendations and on the
    strategic issues that we would be highlighting in our report.

-   Over the last three years NSF has streamlined and focused the
    process of reviewing accomplishments in PEOPLE, IDEAS, and TOOLS
    portfolios. This has expedited completion of the report during the
    Committee's meeting. Over the next year, we recommend that NSF now
    consider how to make corresponding improvements to the task of the
    OE Subgroup. For example, it would be helpful to have:

> \(1\) more guidance and context for the form and structure of the
> final OE product, given the variety of issues that must be covered
> (e.g., merit review, multidisciplinary and transformative research);
>
> \(2\) more information on the relative priority of key data sources
> that need to be analyzed;
>
> \(3\) information on any outstanding OE issues that have not been
> resolved in the previous year; and
>
> \(4\) more information on the COV reports that will be used in a given
> year and whether those particular COVs might contain gaps or biases
> (i.e., the COVs in a particular year are not broadly representative).

-   It might also be helpful for the OE Subgroup chair to attend the
    meeting of the AC/B&O committee (at which the other three OE
    indicators are discussed). This meeting occurs about 2 months before
    the AC/GPA meeting. In that way, the OE chair could have the benefit
    of that discussion as the subgroup looks at the totality of the OE
    goal.

Lastly, the Chair and Vice Chair specifically wish to thank all of their
committee members and especially the able chairs of each of the
subgroups, Harry Shipman, Sally Mason, Gloria Rogers, and Jack Fellows.
It is always a pleasure to work with a group of capable, motivated, and
hard-working individuals and this Committee typifies those traits. Each
person "did their homework" and came to the meeting prepared to discuss,
debate and synthesize the collective results of their work. NSF is
fortunate to have such people in its "corner" and it has been an honor
to serve as their leaders.

**Appendix 1: Advisory Committee For Business and Operations Letter**

National Science Foundation

*Advisory Committee For Business and Operations\
*

May 25, 2005

# 

# 

# Mr. Anthony A. Arnolie

# Director, Office of Information and 

#  Resource Management

# 

# Mr. Thomas N. Cooley

# Director, Office of Budget, Finance,

#  and Award Management 

# 

# Dear Mr. Arnolie and Mr. Cooley:

# 

# The National Science Foundation's (NSF) Advisory Committee on Business and Operations met May 5-6, 2005, to consider ongoing issues at NSF, with special emphasis on the 2005 assessment of Organizational Excellence*.*

# 

# *The committee offers the following thoughts and perspectives on the items discussed at the meeting.*

# 

# CIO, IRM, and BFA Updates

The committee again appreciated the updates on CIO, IRM, and BFA
activities. A number of important observations came to light through the
presentations and ensuing discussions:

-   NSF's leadership on major eGov activities, especially Grants.gov and
    the Grants Management Line of Business, remains vital to the science
    and engineering community. Because the "customers" for Federal grant
    programs are as varied as the programs themselves, it is important
    that NSF and other R&D agencies work to ensure that the emerging
    systems adequately support competitive, investigator-initiated
    activities.

-   The Knowledge Management Pilot Projects discussed appear to hold
    great potential for improving both programmatic and operational
    efficiency at NSF. For example, the work in advanced searching and
    clustering could greatly benefit key parts of the merit review
    process, as it could lead to improved methods of assigning reviewers
    and panelists to proposals.

-   An important attribute of IRM's major initiatives is the involvement
    of a broad cross-section of the NSF workforce -- in terms of both
    staffing level and organization. This emphasis on outreach and
    communication, as was highlighted in the discussion of the
    Administrative Functions Study, is vital to the success of these
    activities.

-   The committee looks forward to future discussions on how best to
    update the Organizational Excellence component of the NSF GPRA
    Strategic Plan. The BFA Update noted that the planning environment
    is being shaped by a complex set of internal and external drivers,
    such as the President's Management Agenda, new guidance related to
    internal controls, a growing and more detailed set of metrics and
    scorecards across the government, and the findings from NSF's own
    Business Analysis, to name but a few. In this light, it would be
    especially valuable to further explore the "people" and "process"
    synergies that cross the five major PMA initiatives.

# 

# Report of Facilities Subcommittee

Dr. Tom Kirk, the subcommittee chair, reviewed the activities of the
facilities subcommittee. He noted that the subcommittee held its first
meeting on March 25, 2005. At that meeting, the subcommittee received
presentations from NSF staff on issues related to facilities management.
Following the meeting, the subcommittee developed a written report,
which the full committee agreed should be provided to NSF (following a 2
week period for committee comments, as is noted in the Minutes).

Dr. Kirk noted that a first task for the subcommittee was defining its
role, recognizing that the subcommittee's activities are framed by three
major policy documents:

-   The 2004 National Academies (NA) Report, *Setting Priorities for
    Large Research Facility Projects Supported by the National Science
    Foundation,*

-   The October 2004 draft NSF/NSB report, also titled, *Setting
    Priorities for Large Research Facility Projects Supported by the
    National Science Foundation,\
    *

-   The July 2003 NSF policy document, *National Science Foundation
    Facilities Management and Oversight Guide.*

The subcommittee decided to assess the interaction of these three policy
documents with the actual functioning processes of the NSF as the
Foundation carries out the assessment, approval, funding and oversight
of the large initiatives undertaken as "Major Research Equipment and
Facilities Construction" (MREFC) projects.

Dr. Kirk described the three main observations that emerged from the
subcommittee's review:

1\. The implementation of adequate project management methods for MREFC
projects during the Development Stage seriously lags the National
Academies Report recommendations as well as NSF policy guidance
currently provided for MREFC projects; accordingly, the needed 'Baseline
Project Definitions' have not been achieved with adequate reliability as
projects move towards New Project status and are submitted by NSF to
Congress for funding.

2\. The failure of NSF to regularly achieve adequate Baseline Project
Definitions during the Development Stage is closely tied to the agency's
under-investment in professional engineering, cost estimating and
project management support for projects during this period; a useful
guideline for the necessary level of pre-project engineering/project
management investment is between 10% and 25% of the total project cost
to provide reliable cost and schedule estimates at this stage.

3\. The Deputy for Large Facilities Projects at NSF is a properly
conceived role but one that has not been adequately empowered, staffed
and supported within the agency to this point in time; a strengthening
of the Deputy position plus significant staff enhancement are needed to
realize the purposes recommended in the National Academies Report and
accepted by the Foundation.

In addition to agreeing to provide the subcommittee report to NSF, the
committee also agreed that the materials presented to the subcommittee
by NSF staff at the March 25 meeting should be made publicly available.

**NSF 2005 Assessment of Organizational Excellence**

The principal focus of the meeting was NSF's 2005 Assessment of
Organizational Excellence. This was the second year that NSF developed a
self-assessment of activities related to its OE goal, and the committee
was again asked for input on three of the four indicators used to
determine significant achievement in OE:

-   Human Capital,

-   Technology-Enabled Business Processes, and

-   Performance Assessment.

The fourth OE indicator, Merit Review, will be assessed by the Advisory
Committee for GPRA Performance Assessment (AC/GPA).

The committee's discussion focused on two questions:

1.  Does the evidence presented support NSF's determination that it has
    (or has not) "demonstrated significant achievement" for the
    indicator?

2.  Should any changes in approach or methodology be considered for
    future OE assessments?

Regarding the first question, the committee concluded that the evidence
presented does support NSF's determination that it demonstrated
significant achievement for the three indicators. The committee in
particular commended the BFA and IRM staff for incorporating the
observations provided last year on improving the assessment process.

This year's presentations were especially valuable in highlighting 1)
the major activities that addressed the three indicators, 2) the
mechanisms used to determine success, and 3) the relative level of
success (fully successful, partially successful, etc.). The committee
appreciated the high quality of the presentations and how they
demonstrated clear leadership in the broad and complicated area of
Organizational Excellence.

Through its discussion, the committee also provided a number of
observations to NSF that address the second question related to
potential improvements to the assessment process. One recurring theme in
the committee's discussion was the need for NSF to clarify the
connection between activities that address the OE goal and NSF's overall
mission. A number of the committee's observations directly addressed
this point:

-   The OE assessment should be vigilant in reminding people about NSF's
    mission and purpose. The current assessment framework presents NSF's
    business processes in isolation. This runs the risk of rewarding NSF
    for the processes themselves and overshadowing the agency's actual
    mission of promoting progress in science and engineering.

-   The OE assessment should also distinguish between "strategic"
    accomplishments and "tactical" ones. If the assessment focuses
    disproportionately on achievements at the tactical level, it loses
    the strategic connection to the agency's mission and long-term
    vision. This can lead to NSF's assessment appearing
    indistinguishable from those of other agencies, as the underlying
    processes are somewhat generic. It is important for the assessment
    to demonstrate NSF's unique role among Federal agencies.

-   A third area that should receive increased attention in future
    assessments is the impact of NSF's leadership in government-wide
    initiatives, especially those in eGovernment. The committee
    recognized that NSF's leadership in these areas is often more
    valuable to the initiatives than to NSF itself, at least in the near
    term. One possible approach to this issue would be for future
    assessments to highlight the elements of government-wide activities
    that benefit NSF and the elements that NSF is pursuing primarily to
    meet specific requirements.

Finally, the committee noted that NSF's presentations and the ensuing
discussion outlined the key set of issues that will need to be addressed
when presenting the OE goal in the next update of the agency's GPRA
Strategic Plan.

The committee understands that NSF will provide these observations to
the AC/GPA.

# Transforming the NSF Academy

The presentation, *Transforming the NSF Academy,* provided the committee
with an excellent overview of the Academy's goals and major activities.
It also underscored NSF's motivation in establishing the Academy, namely
to serve as a catalyst for a continuous learning organization. A number
of issues for NSF consideration arose in the discussion following the
presentation, such as the importance of involving all parts of the
agency in planning the Academy's activities, how best to highlight the
links between the Academy's activities and the agency's strategic goals,
ensuring that the Academy's leadership competency models appropriately
complement other such models in use at NSF, and the importance of
blended learning environments for e-learning activities.

# The NSF Business Analysis

The committee found the approach used in the presentation on the NSF
Business Analysis -- following a single topic (merit review in this
case) from baseline analysis through findings and recommendations -- to
be a productive way to frame the committee's discussion. NSF is
encouraged to continue using this approach in future presentations.

In its discussion, a central issue the committee raised for NSF to
consider is whether there may be opportunities for increased integration
across the analysis and for putting the analysis in a broader context.
The findings seem to reflect discrete lines of analysis, focused on
workload, or the merit review process, or the results of the applicant
survey, but lacking integration that would answer questions like "what
should be done about the rising proposal workload?" "How do workload and
staffing issues affect the advancement of science?" For example, could
the workload survey clarify what tasks are shared across employee
categories, and which tasks are most important to NSF's core mission?

In terms of context, the business analysis should communicate more
explicitly the integral importance of NSF's many external relationships.
Unlike most federal agencies, NSF's accomplishing of its core mission
depends on extensive and extended interaction with individual scientists
and their institutions. Business analysis results should emphasize how
NSF's own activities affect and are affected by these relationships. For
example, the committee's discussion underscored the importance of NSF
program officers taking time to work directly with first-time applicants
to NSF. This arose in the context of discussing the possibility of
creating a customer service center to answer investigator inquiries
currently handled by program managers. The committee noted that
educating and guiding investigators, especially young or unsuccessful
investigators, is a valuable part of NSF's culture. NSF should therefore
continue to approach this kind of proposed activity as one that
complements the role of the program officer.

# Meeting with Dr. Bement

The committee again appreciated the chance to meet with NSF Director Dr.
Arden Bement. Dr. Bement noted that a major challenge for the agency is
to address the increasing proposal workload on program managers. He
described selected strategies under discussion with senior management to
slow the increase in the number of proposals received each year. He
added that the most important function of the NSF program officer is to
serve as "signal processors" for science and engineering -- ensuring
that the agency remains focused on the frontiers of research and
education.

He also discussed the importance of aligning NSF's staff functions and
business systems with the agency's strategic goals, especially in light
of the revision of the NSF Strategic Plan scheduled for next year. In
addition to asking the committee to help NSF develop the next iteration
of its Strategic Plan, he asked it to give attention to the agency's
internal controls and their importance in achieving public trust. The
committee shared with Dr. Bement the major issues raised previously
during the meeting.

In closing the meeting, the committee noted that an appropriate topic
for the Fall meeting would be how to foster a deeper integration of the
OE strategic goal with the NSF mission, in the context of the upcoming
revision of the agency's Strategic Plan.

The committee hopes these observations help to inform and guide the
Foundation as it addresses the range of issues discussed at the meeting.
We would like to thank the staff that helped make this meeting a
successful one. We look forward to reviewing anticipated progress on the
various issues discussed at this meeting and to discussing other
mission-critical issues at our next meeting.

On behalf of the committee,

Dr. Norine Noonan Dr. Peter D. Blair

*Co-Chair* *Co-Chair*

May 5-6, 2005 Meeting

**Appendix 2: 2004 NSF Committees of Visitors**

**NSF-Wide**

Biocomplexity in the Environment (BE) February 25-27, 2004

Nanoscale Science and Engineering (NS&E) May 11-13, 2004

**Biological Sciences**

Division of Biological Infrastructure (DBI) June 16-18, 2004

Plant Genome Research Program August 11-13, 2004

**Education and Human Resources**

Teacher Preparation (CETP and STEMTP) October 9-10, 2003

Federal Cyber Scholarships for Service (SFS) November 19-20, 2003

NSF-NATO Postdoctoral Fellowships in Science and Engineering August
9-10, 2004

Centers for Learning and Teaching (CLT) May 3-5, 2004

Tribal Colleges and Universities Program (TCUP) May 20-21, 2004

Rural Systemic Initiative (RSI) May 20-21, 2004

Urban Systemic Program (USP) August 25-27, 2004

**Computer and Information Science and Engineering**

Civil and Mechanical Systems (CMS) March 22-24, 2004

###  {#section-31 .unnumbered}

### Geosciences {#geosciences .unnumbered}

Lower Atmosphere Research Section (LARS) September 14-16, 2004

Earth Sciences Instrumentation and Facilities (EAR/IF) August 18-20,
2004

**Mathematical and Physical Sciences**

Division of Chemistry (CHE) February 3-4, 2004

Division of Mathematical Sciences (DMS) February 11-13, 2004

**Social, Behavioral and Economic Sciences**

Economics, Decision and Management Science March 29- 31, 2004.

> Economics
>
> Decision, Risk, and Management Science Program (DRMS)
>
> Innovation and Organizational Change (IOC)

Methods, Cross Directorate, and Science and Society March 29-31, 2004

Methodology, Measurement and Statistics (MMS)

> Science and Technological Studies (STS)
>
> Societal Dimensions of Engineering, Science & Technology (SDEST)

Social and Political Sciences March 18-20, 2004

> Law and Social Science Cluster
>
> Political Science Cluster

Sociology Cluster

Note: All executed and approved COV reports are posted on the COV
webpages in the NSF Office of Integrative Activities

**Appendix 3: NSF Draft \-- Innovative-High Risk Research Definitions**

# 1. Forefront, Novel or Transformative but Untried or Untested 

-   This work is high risk since it explores the actual working of a
    living organism. We have very little information and knowledge in
    this area. An understanding will bring revolutionary changes to
    information processing. On the other hand we do not know the "right
    approach" until we try out many and it is possible that we will be
    unable to unlock this mystery that nature holds.

-   This is an exploratory research project. It is high-risk because it
    is based on a novel idea that has not been tried before.

-   For the first time it involves both state-of-the-art computational
    and experimental techniques.  It has the potential of a large
    impact.

# 

# 2. High Reward but Significant Technical Challenges and/or High Probability of Failure

-   The technological hurdles are so immense that the project may never
    succeed.

-   The work is high risk because many unknown factors may complicate or
    hinder the scale-up of the process by several orders of magnitude.

-   This work involves new technology and deployment in high risk
    environments, so there is a risk of failure and loss.

-   The return of data from a hostile alien environment hundreds of
    millions of miles from Earth is a risky endeavor \[how NSF radio
    telescopes saved the wind data collected during decent on Titan
    after configuration problems on the remote NASA/ESA Cassini
    spacecraft\]

# 3. Innovative and Contrary to Current Theory or Conventional Paradigms

-   There is always risk when exploring new areas of science, especially
    if the research may run contrary to current theory.

-   This work involves fundamental issues involving computing
    architectures of the future that may be a radical departure from the
    way computing machines are designed today.

-   This work challenges the common understanding of how water molecules
    interact. These researchers are using new, cutting-edge tools to
    re-examine the structure of liquid water.

#### "High Risk", but not necessarily Innovative-High Risk

# 4. Risk from Not Succeeding

-   Any study of severe storms (including floods) is high risk (for the
    P.I.s) and for the speedy results that could save lives. The risk is
    in the criticality of the research.

-   Failure could mean less minority students will enter the scientific
    workforce.

-   The risk is that delays in the research can result in disastrous
    events that may have been prevented.

# 5. Risk to Principal Investigators/Others 

-   To drill 48 boreholes on a glacier can be very dangerous and risky.

-   The PI undertook lengthy, dangerous fieldwork in the Golden Triangle
    area of Southeast Asia.

-   With our research ever involving new ideas and technologies as a
    technical society we must always be vigilant in protecting society.
    At the moment there are many unknowns surrounding nanotoxicology and
    still we continue to produce these products. The risk is our lack of
    knowledge and understanding for the products being produced.

**Appendix 4: 150 Innovative, High Risk Projects \-- NSF Program Officer
Assessments**

  -----------------------------------------------------------------------
  **Title**

  A Manifold Learning Toolbox For Nanostructure Simulations

  A Materials Prediction Map for Tailoring Crystal Chemistry to Bulk
  Modulus

  A New Camera for Extrasolar Planets

  A New Paradigm for Cell Biology: Putting Some Muscle Into RNA
  Transcription

  A New Sun-to-Earth Coupled Model

  A Search for Dark Matter

  A Wireless Stent That Measures Arterial Blood Flow

  Adelante!: Focus on Latinas in Math, Science and Engineering

  Advanced Optical Instruments for Monitoring Asthma

  Air-Sea Exchange Measurements by Eddy Correlation

  An Electrically Regenerated Diesel Particulate Filter

  An Inexpensive Vertically Profiling Float for Seasonal-High-Resolution
  Measurements in P-BECS

  An Observational Array for Arctic Oceanographic Measurements

  Analysis of a Novel Mechanism of Genetic Reversion in Arabidopsis

  Automated Housing Construction with Contour Crafting

  Baby Cosmos Grows Up

  Bioengineers Harness Prehistoric Marine Microorganisms to Produce
  Nanoscale Semiconductor Composite Materials

  Brain Function and Economic Decisions

  Brain-Like Learning Can Boost Electricity to Users

  Breakthrough in Learning to Save Babies\' Lives

  Breakthrough Structural Information on Viruses through Combination
  Techniques and Methods

  Bring Statistical machine Translation to the Real World

  California State University at Fresno develops Certification Program
  with NSF Support

  Cardiovascular Informatics

  Catch a wave from space: Einstein@home

  Challenging our Understanding of Water

  Collaborative Research: Formal Methods for Behavioral Subclassing and
  Callbacks

  Complex Reaction Networks on a Chip

  Computer-Aided Design Algorithms and Tools for Nanotechnologies

  Controlling Cell Shape

  Cooperative Wireless Communications

  Dark Matter in a Lambda CDM Universe

  Design for Near-Term Cost Breakthrough in Access to Space

  Designing New Materials to Prevent Protein Adsorption Fouling

  Devices and Architectures for Neuromorphic Circuits with Nanoelectronic
  Components

  Dynamic Evolution of DNA Supercoils and Loops

  Dynamic Scheduling for Parallel Server Systems

  EdGCM: A Global Climate Model (GCM) for the Classroom

  Education Technology Software Promises to Ease Difficulties in Studying
  Science

  Enhancing High-Speed Machining of Difficult-to-Machine Materials

  ET Might Write Not Radiate

  Event Driven Software Quality

  Examination of Natural Products using X-Ray Crystallography

  Exploiting Fading in Wireless Channels to Make Data Services Feasible

  Fibers and fabrics that change color when illuminated

  First Arctic Ocean Drilling Reveals Subtropical Past

  First Crossdisciplinary Text on Optimal Adaptive Management of Complex
  Systems

  First light for the Advanced Modular Incoherent Scatter Radar (AMISR)
  in Peru

  Flames for Nano-Manufacturing

  Foundations of Autonomous Biomolecular Computation

  Fractures as Main Pathways of Water Flow in Temperate Glaciers

  Gaffers Guild gives Glassblowing Demos to Thousands of K-12 Students
  Annually

  Helical Configurations of Large Molecules

  High Throughput Manufacturing of Nanoporous Films

  IGERT Research Effectively Crosses Fields

  Implicit Discrimination: New Insights into the Source of Discriminatory
  Behavior

  Influence of Nano-particle Additives on Boiling in Liquids

  Innovative Ocean Wave Energy Extraction Devices

  Inside the San Andreas Fault

  Internal Cooling of Turbine Blades for High Efficiency Propulsion and
  Power

  ITR/SW: STRATEGIC SOFTWARE DESIGN: Value-Driven Software Definition,
  Development, Deployment, and Evolution

  ITR: A Community Based Partnership for Community-Based Education
  (COPIRE)

  ITR: New Approaches to Human Capital Development through Information
  Technology Research

  ITWF: Culturally Situated Design Tools

  Judicial Pioneers: Litigants in the Moscow Theater Hostage Case

  Kidney Exchange: A Life-Saving Application of the Theory of Two-Sided
  Matching

  Laser-Controlled Production of Antihydrogen

  Lasers Talk with One Single Atom

  Learning gains from Triarchic Instruction

  Long-term turbulence observations and modeling in the coastal ocean

  Loss processes for ozone in forests

  Microbial Fuel Cell Generates Power From Waste

  Modeling Impacts on Spacecraft

  Modulating light on a CMOS chip

  Molecular Computation in Ciliates

  Molecular Storage by Charge Trapping

  Molecule Fluorescence Methods and Myosin V

  Multiple-Dye Luminescent Solar Concentrators

  Nanoporous Silica Sluries

  Nautical Knots and Engineering Design

  NEES: George E. Brown, Jr. Network for Earthquake Engineering
  Simulation

  NEESgrid: Cyberinfrastructure for Earthquake Engineering Research

  NER: The First Rational Approach to the Preparation of Single Molecule
  AFM Tips

  New Environmental Technology to Treat Sludge

  New Evidence for Hope of Breakthrough in Fuel Cells

  New Techniques in Cryopreservation Optimize Survival of Cells and
  Tissues

  New Valve Based on Magnetic Coupling

  New World Record Magnet at the National High Magnetic Field Laboratory

  NGRIP Basal Ice

  Novel DNA Nanostructures for Targeted Molecular Scale to Micron Scale
  Interconnects

  NSF Radio Telescopes Save Saturn Moon Data

  Observing Cosmic Rays at the Energy Limit

  Ocean Drilling Probes the Deep Biosphere

  Pipelines to the Stars

  Polymer Nanocomposite Foams Produced by Environmentally Benign Carbon

  PRAM-On-Chip

  Predicting Storm, Tsunamis, and Flood Emergencies

  Pretzels, Noodles & Mini-Proteins

  Probabilistic and Statistical Methods in Machine Learning

  Programmable Nanoscale Self-Assembly on Solid Surfaces

  Protein Motors Incorporated

  Quantum-mechanical Features of Thousands of Biological Macromolecules

  Rapid Response Telescope Array Sees First Light

  Reactive Mounting of Heat Sinks

  Reactive Nanotechnologies to Enable Lead-Free Soldering

  Reliability of Defect Tolerant Architectures using Probabilistic Model
  Checking

  Replicating the Earth\'s Magnetic Field in the Laboratory

  Research Experiences for Undergraduates in Chemistry at California
  State University Los Angeles

  Research Experiences for Undergraduates in Chemistry at James Madison
  University

  Research Experiences for Undergraduates in Environmental Sciences at
  Northern Arizona University

  Researchers Fabricate Nano Test Tubes

  Retwistered Twister

  Reverse Engineering Cellular Pathways from Human Cells Exposed to
  Nanomaterials

  Ring Arrays for Magnetic Storage Applications

  Risky Project Helps Arabidopsis Genome Annotation

  Safety Analysis for Critical Product Lines

  Scientists Discover Undersea Volcano Off Antarctica

  Security of Elliptic Curve Cryptography

  Self-Adaptive Software\'

  Self-assembled polymer nanofibers for nerve repair

  Self-Perfecting Genetic Circuits

  Semiconductor Devices for Control of Solid-State Laser Dynamics

  SGER: "Game Based Software Adaptation"

  SGER: Biomimetic Wet Attachment Mechanism for Miniature Climbing Robots
  in Unstructured Environments

  SGER: Development of the MERMAID Float: an Autonomous Vehicle for
  Recording Regional Earthquakes within the World\'s Oceans

  Single Photon and Single Atom Sources

  Smart Nanotubes for Selective Biomolecule Delivery to Living Cells

  Solid-State NMR and Plant Structural Biology

  Soy-Based Renewable Composite Material for Affordable Housing

  Spin-Dependent Processes in Organic Semiconductor Devices

  Stress-Induced Nucleation of Nanocrystals

  Strong-Coupling Emission from a Quantum-Dot Photonic-Crystal-Slab
  Nanocavity

  Structure of an RNA catalyst in its Entirety

  Synchronized Transatlantic Synchrotron Research

  Synthesis of new electrooptic polymers for DNA and RNA sensors

  Technology-EDA Integration Through Nanometer Interconnect Design Tools

  Test of the nuggets submission process \-- not for approval

  The 2004 NSF Waterman Award

  The LIGO Livingston Observatory says "Goodbye" to Noise from Logging

  The Long-Term Effects of Technology on Microprocessors

  The Opium Trade in the Wa Area of the Golden Triangle

  The Self Reorganization of Matter: Beyond the Standard Theory of Phase
  Transitions

  Toward an international network of gravitational wave detectors: LIGO
  scientist participates in Virgo

  Ultraviolet Photonic-Crystal Laser

  Undergraduate Research for Freshmen and Sophomores at Xavier University
  of Louisiana

  Uptake of Anthropogenic Carbon Dioxide by the Ocean During the
  Industrial Age

  Using a Moss to Understand Chloroplast Biogenesis

  Using Biomolecules to Assemble Building Blocks of Matter

  Visualization of Proteins

  X-ray Reflectivity of Liquid/Liquid Interfaces
  -----------------------------------------------------------------------