**ENG Advisory Committee Report of Engineering Directorate Performance
for GPRA for FY 2001**

A. **A description of the role of the directorate in support of research
and education; relevance of the programs to the mission of NSF and
NSF\'s goals.**

The research and education agendas of the Directorate for Engineering
(ENG) focus on strengthening the nation\'s engineering science base;
specifically in the areas of engineering systems, devices, and materials
and associated processes and methodologies. The Directorate catalyzes
the discovery, guides the development and nurtures the future of
innovations that *advance the national health, prosperity, and
welfare.*..and.*..secure the national defense.* It is a key component of
this nation\'s scientific and technological infrastructure. ENG played a
critical role in the recent tragic events of September 11, 2001. The
Directorate quickly organized itself to avail itself of its resources in
hazard response, making \$0.3 million immediately available in *Quick
Response Research Awards*. Through its innovative, integrative, and
cooperative programs, it supports the transformation of research results
into those enabling technologies that generate new products and services
and support the educational needs of this nation. ENG leads the
Foundation's Nanoscale Science and Engineering Priority Area and manages
and supports the National Nanofabrication Users Network. ENG programs
support a broad range of research and other activities aligned with the
National Science Foundation\'s information technology and biocomplexity
in the environment priority areas.

Its role in education is to provide the means to perpetuate the
generation of this nation\'s engineering talent through programs that
enhance student participation in research, graduate training, curricular
reform, the transfer of new research results into education, and
outreach to K-12 students. ENG also supports student and faculty
partnerships with industry through the Grant Opportunities for Academic
Liaison with Industry (GOALI), the Engineering Research Centers (ERC)
and Groups, and the Industry/University Cooperative Research Centers
(I/UCRC) programs. ENG has made substantial commitments to the Faculty
Early Career Development (CAREER) program that provides support to
outstanding young faculty with promising approaches to the integration
of research and education. ENG Research Experience for Undergraduates
(REU) programs reached out to approximately 1000 students in FY 01, and
the newly-launched Research Experience for Teachers (RET) (about 200
teachers funded in the first year) will effect systemic change in the
Nation's engineering education programs. These programs contribute to
the NSF goal of developing *a diverse, globally-oriented workforce* of
*scientists and engineers, and well-prepared citizens*.

The ENG Directorate takes the lead in supporting the National
Nanofabrication Users Network (NNUN), which provides the nation's
researchers with access to advanced Nanofabrication equipment and
practical expertise. These enabling tools are extremely timely in view
of the current priority area in *Nanoscale Science & Engineering* (NSE);
it is important to note that ENG made its investment in this area long
before NSE was singled out as a priority area: NNUN was first
established in 1994. ENG also supports the Network for Earthquake
Engineering Simulation collaboratory (NEES), a visionary program that
provides shared databases, tools, and expertise for the understanding
and mitigation of earthquake hazards. This ENG program, that promotes
the innovative development of engineering science and information
technologies with applications to the social and economic segments of
society, may serve as a model for a future program on *integrated hazard
measurement*. Certainly, the need for such a program was well
underscored by the events of September 11, 2001.

B.  **Executive Summary of Results Being Reported**

The ENG Advisory Committee believes that the ENG Directorate is doing an
excellent job in promoting and addressing NSF's goals related to
*People, Ideas,* and *Tools.* We provide numerous, specific examples in
our report of ENG's investment in human resources and education, in
developing effective outreach to the public, and in contributing
engineering and scientific insights into large-scale issues of public
policy. ENG has been, and continues to be one of the leaders in
promoting *partnerships* in research and education, partnerships among
academic institutions, between academia and industry, and between
academia and governmental and policy-making agencies. ENG has also
demonstrated its continuing commitment to the development of critical
databases, enabling tools and educational approaches made broadly
accessible to the engineering community. We applaud ENG's recent rapid
and proactive approach in applying its infrastructure of expertise and
tools in the rapid response to the tragic events of September 11, 2001.
We also applaud ENG's continued commitment to supporting innovative and
visionary solutions to engineering challenges, witness ENG's leadership
of the Nanoscale Science and Engineering priority area.

Against this background of excellent performance, the AdCom cites some
areas in which improvements may be cultivated. Merit Criterion 2,
addressing the *broader impacts* of supported research is still not
being fully and consistently applied. The full extent of ENG's
contributions to the development of *tools* is not being accurately
represented (for example, refer to Table 1.1, *Selected activities of
the Engineering Directorate by outcome goal, FY 2001,* in the ENG
Performance Report for FY 2001), and we believe that this does a
disservice to the Directorate. The capability of ENG to attract and fund
truly 'high risk' (with high pay-off) projects may be difficult to
achieve within the current review process, and with the present
limitations on funding. Better guidance from the AD, Program Directors
and other ENG staff will be required to truly broaden ENG's portfolio to
include an appropriate balance of high risk projects. Finally, although
there has been an improvement in the accessibility of information for
the preparation of the GPRA reports, there are many things that could be
instituted to further improve this process.

C. **A description of the approach used in making the assessment and
list of sources of information and input.**

The assessment is based primarily on Committee of Visitor (COV) reports,
the ENG Directorate annual report, and relevant discussions conducted
during this and past ENG Advisory Committee meetings. These discussions
included assessments of previous GPRA reports for the Engineering
Directorate. This assessment also considers information in the NSF data
systems, program evaluations, and information from the media.

Findings from the following reports prepared by two COVs are considered
in this report.

Civil and Mechanical Systems (CMS)

Engineering Education and Centers (EEC)

D. **For each performance goal and each relevant area of emphasis
provide a summary of the overall performance of the directorate in
achieving NSF\'s goals. Utilize indicators in crafting assessments and
address whether the directorate demonstrated sufficient progress toward
achieving the NSF-wide performance goals for results.**

The Advisory Committee believes that the Directorate has been effective
in achieving NSF\'s performance goals through the successful
implementation of the strategic directions developed by its six
divisions. We believe that ENG has demonstrated excellent progress in
meeting the criteria of the Outcome goals and indicators. The discussion
below highlights the individual goals, and gives illustrations in
support of our assessments. We were also asked to assess the balance of
innovative, risky, and interdisciplinary research. We believe that the
portfolio of programs demonstrate a good balance of innovative research,
and are exceptional in the interdisciplinary, interactive quality of the
projects. With respect to *risky* research programs, there appears to be
some evidence (CMS COV report) that high-risk proposals suffer from
unrealistic expectations by members of panel review teams that the
proposed methodology and outputs should (must) have a high probability
of success. Therefore panelists may be unwilling to make awards to truly
\"risky\" proposals during periods of constrained budgets.

*Strategic Outcome: People---Development of "a diverse, internationally
competitive and globally engaged workforce of scientists, engineers, and
well-prepared citizens"*

ENG continues to demonstrate its commitment to the successful
realization of performance goals related to the strategic outcome
related to *People*. The Directorate continues to play a leadership role
in the CAREER program; in FY 2001 ENG made 119 out of the total 387 NSF
CAREER awards. Through its *Research Experience for Undergraduates
(REU)* programs, ENG has reached out to approximately 1000 students in
FY 01. More recently, ENG has launched its program in *Research
Experiences for Teacher (RET),* with FY 2001 awards exceeding \$2
million. This program should provide one of the best ways of bringing
the excitement and realities of the engineering and science research
world into the classroom. Through innovative partnerships with museums,
industry and other academic institutions, ENG programs are bringing the
immediacy and excitement of engineering out to the public. We address
the particular Performance Indicators below.

#### Indicator 1--- Improved mathematics, science, and technology skills for U.S. students at the K-12 level and for citizens of all ages: [successful]{.underline} 

Although all programs in ENG evidence and emphasize the integration of
education with research, not all programs focus on the K-12 level.
However, there are outstanding examples of significant programs in this
area throughout the Directorate. For example,

-   The Engineering Research Center (ERC) for Particle Science and
    Technology at the University of Florida (EEC-94-02989) encourages
    secondary students' interest in engineering through day-visit tours
    and *summer engineering camps*. Through hands-on laboratory
    experiences students learn about careers in particle science and
    technology. An intensive 8-week residential laboratory research
    experience offered to high school students through the university's
    *Student Science Training Program* hosts up to four students each
    year and places them on multidisciplinary research teams together
    with university students. These talented high school researchers
    present their ERC projects at the university's annual junior high
    school science and humanities symposium and at school district
    science fairs.

-   In project CMS-96-25161, participants were engaged in the INROADS
    outreach program for minority high school students in St. Louis.
    Also under this grant, research results were transformed into
    undergraduate and graduate laboratory demonstrations to provide
    students with 'outside-the-textbook' experience.

-   The ERC for Environmentally Benign Semiconductor Manufacturing
    (EEC-95-28813 and SRC 96-EC-425) has sponsored three-week high
    school teacher conferences during the summers of 1997-2000. Teams of
    teachers learn about Center research programs, tour facilities,
    shadow faculty and graduate students, develop lesson plans to
    augment their curricula. In the summers of 1998-2000, the Center
    offered a no-cost, three-day high school teacher conference, to
    introduce the teachers to the Center, high tech job opportunities,
    and to give them kits containing experiments and computer modules
    developed at the teacher institutes.

#### Indicator 2---A science and technology and instructional workforce that reflects America's diversity: [successful]{.underline}

The examples below show the success of ENG programs in recruiting and
retaining a more diverse engineering student population, that will
ultimately comprise the nation's engineering workforce.

-   Retention of African American engineering students from the freshman
    to the sophomore year in the universities of the *Gateway Coalition*
    (EEC-97-27413) increased from 67 percent in 1992 to 87 percent
    in 2000. For women students, retention increased from 75 percent to
    90 percent, and for all engineering students from 79 percent to 86
    percent.

-   NSF statistics show that large fractions of the students involved in
    the ERCs for FY 2000 were female and under-represented minorities
    (URM). The 600 pre-college students included 48% female and 36%
    under represented minority (URM) students; the 4000 undergraduate
    students included 29% female and 12% URM students; the 2200 graduate
    students included 22% female and 5% URM students; and the 344 REU
    students included 39% female and 35% URM students.

#### Indicator 3--- Globally engaged science and engineering professionals who are among the best in the world: [successful]{.underline}

In addition to promoting the internationally-linked engineering research
activities, ENG has been proactive in fostering international
initiatives, and collaborations-at-a distance. Some specific examples
include:

-   Geotechnical reconnaissance report on Hyogoken-Nanbu earthquake
    (CMS-95-20204). Eighteen U.S., Japanese, and Canadian researchers
    were rapidly deployed in the area affected by the Hyogoken-Nanbu
    earthquake to prepare a document of substantial value to U.S.
    practitioners and researchers. Shortly after the earthquake, the
    report was available on the Web. The work represented exceptional
    international collaboration and provided invaluable reconnaissance
    training for young researchers who have subsequently become leaders
    of reconnaissance efforts after major earthquakes in Turkey, Taiwan,
    and Washington State.

```{=html}
<!-- -->
```
-   New distance-learning courses (EEC-99-01565): The Maintenance and
    Reliability Center at the University of Tennessee-Knoxville (an
    Industry/University Collaborative Research Center) obtained support
    from industry and CRCD to develop four new academic courses.
    Distance-learning versions of the courses were presented at
    participating universities, including the University of Sao Paulo in
    Brazil.

#### Indicator 4---A public that is provided access to the benefits of science and engineering research and education[: successful]{.underline} 

ENG has provided benefits to the public in a wide variety of ways,
ranging from the development of displays and workshops set in public
venues, to direct consultation on the timely engineering and societal
issues. Perhaps the most dramatic examples of ENG's participation are
related to the response of its researchers in the recent tragic events
of September 11, 2001. With a record response time, ENG allocated \$300K
within 3 weeks to support over half-a-dozen awards to allow researchers
to access critical data from 'Ground Zero' in a timely manner. Projects
included the 'Forensic Study of Steel and Fire Protection from WTC
Collapse'(A. Astaneh-Asi, CMS 0139542), and 'Palm Pilot/GPS-Digital Data
Collection for Damage Assessment (D. Frost, CMS-0139258). As shown in
Figure 1, early access to the WTC site allows critical analysis of
structural failure, and the development of future strategies to prevent
such failures. CMS also worked with the *Quick Response Research
Program* of the Natural Hazards Research and Applications Information
Center (NHRAIC, CMS-9811388) to fund studies of the aftermath of the
September 11, 2001 terrorist attacks in New York City and Washington,
D.C. Further details of the projects funded can be accessed at the
NHRAIC website:![](media/image1.wmf){width="2.1222222222222222in"
height="1.6076388888888888in"}

<http://www.Colorado.EDU/hazards/qrsept.html>.

-   *Consultation on National Natural Hazards*: A major research project
    for the *2nd National Assessment of Research and Applications on
    Natural Hazards* (CMS-95-41969, CMS-95-44211, and CMS-96-41063)
    involved over 130 national experts in hazards and disasters from all
    fields of science, engineering, policy, and practice. The results
    greatly influenced policies for hazard-loss mitigation in FEMA's
    Project Impact, which will establish a national risk assessment,
    meet the need for computer-aided systems to inform local hazard
    decision-making, and assist local efforts to design safer
    communities. The document was subsequently used internationally. For
    example, it provided the basis for redrafting New Zealand's
    environmental and hazards legislation to link sustainable
    development with environmental management and hazards mitigation.

-   Professionals who plan and manage *reconstruction after major
    foreign earthquakes* were invited to describe their experiences to
    the planning directors of major U.S. cities subject to earthquakes
    in a symposium funded by the directorate (CMS-98-18494,
    CMS-94-16416, CMS-90-25125, CMS-90-8032). Before the symposium, both
    the Whittier Narrows and Loma Prieta earthquakes struck. Planners
    from the affected cities attended the symposium to share their
    experiences with early recovery and to learn from their foreign
    counterparts. The symposium was an early attempt to bring
    information about post earthquake reconstruction directly to the
    global planning community. NSF funded the development of a training
    exercise in which planners confronted the major reconstruction
    decisions, problems, and opportunities typically faced in the first
    year or two after an earthquake, including preparing a
    reconstruction plan and defining mitigation actions that could make
    each of the tasks easier to carry out. The exercise has been
    distributed by FEMA to state mitigation officers and flood,
    earthquake, and hurricane program managers.

-   Besides creating opportunities for professionals and policy planners
    to benefit from ENG investments in engineering research, the
    Directorate also sponsors projects that bring these day-to-day
    engineering issues to the public eye. The ERC for Reconfigurable
    Machining Systems at the University of Michigan (EEC-95-291925) is
    creating an *exhibit for a local museum* that will enable visitors
    to explore the interdependence and impact of product design,
    manufacturing processes, and business issues. The Earthquake
    Engineering Research Center (EEC-97-01568) at UC Berkeley is
    supporting an *Earthquakes!* Exhibit at the California Academy of
    Sciences to educate the public about the use and advantages of
    performance-based earthquake engineering design.

## *Ideas---Enabling "discovery across the frontiers of science and engineering, connected to learning, innovation, and service to society*"

ENG has demonstrated consistent support to promote new visionary
research initiatives whose impact will be of ultimate benefit to
society. ENG has been the leader in the Nanoscale Science and
Engineering (NSE) area, making a total FY 2001 investment of \$67.5
million. It has also been a major contributor to the Information
Technology Research (ITR) priority area, as well as the Biocomplexity in
the Environment (BE) priority area. A more detailed discussion of the
particular indicators is given below.

#### Indicator 1---A robust and growing fundamental knowledge base that enhances progress in all science and engineering areas, including the science of learning: [successful.]{.underline}

The examples below give evidence of important ENG investments that have
led to the establishment of knowledge bases and expertise that will
continue to have growing influence on enabling progress in engineering
and science applications.

-   The 1987 Whittier Narrows earthquake revealed a significant gap in
    disaster research literature: the need to improve understanding of
    the impact of major natural disasters on the *business sector.* Most
    of the existing research had focused on the economic impacts of
    disasters on households and families. With funding from the National
    Center for Earthquake Engineering Research (NCEER), the PIs were
    able to carry out a large-scale survey of businesses in
    Memphis/Shelby County, TN on business owners' perception of
    earthquake hazard, on their judgment on the vulnerability to and
    effects of lifeline disruptions, and on their preparedness measures.
    Similar data was collected from the Northridge earthquake. Another
    NSF grant (CMS 96-32779) extended such studies to Santa Cruz County,
    CA and Dade County, FL. Overall, these studies have produced data on
    nearly 5000 businesses of all sizes and types, comprising the
    largest database on business disaster readiness and effects in the
    world (CMS-98-18494, CMS-94-16416, CMS-90-25125, CMS-90-8032).

-   The Bio-Engineering Technology Engineering Research Center
    (EEC-98-76363) has established a Java module library of common
    components, common visualization tools as part of a 'broad-based
    attack on the barriers to... an integrated effective system... for
    engineering education'. They have also developed model curricula for
    undergraduate and graduate education, as well as a wide range of
    innovative educational technology delivery systems and interactive
    learning software.

#### Indicator 2---Discoveries that advance the frontiers of science, engineering, and technology: appropriate balance of high risk...innovative research areas across all NSF programs: [successful]{.underline}. 

There is a need to encourage the development of programs that have
innovative, high-risk concepts that can result in new frontiers in
emerging areas such as nanotechnology, as well as in traditional areas
such as structural systems. An example of the latter includes so-called
\"smart structures\" that have electronic communications and
interrogation devices that are integral to the structural integrity of
the system. Another example is provided by research in fiber reinforced
reactive powder concrete that is currently under development by foreign
industry, resulting in proprietary products. To promote a more
widespread and competitive national and worldwide implementation, such
new materials should be developed under NSF-supported research.
Certainly the investments made in ENG have produced discoveries that
have the potential to provide a leading edge for possible commercial
developments and investments. For example:

-   The Data Storage Systems Center (an ERC at Carnegie Mellon
    University, EEC-89-07068) has developed two-element actuator control
    systems for improved tracking speed and accuracy in high-density
    magnetic and optical disk drives. The center has played a pivotal
    role in the industry, as evidenced not only by industrial funding,
    but also by the establishment of a major industrial research
    facility (by Seagate Corporation) near the center in Pittsburgh,
    directed by the ERC's founding director.

#### ENG invested in research (CMS-96-25161) that developed the theory and method to anticipate pathological nonlinear vibrations through precursor sensor measurements that were applied to the detection of cardiac arrhythmia in collaboration with Cardiac Pacemakers, Inc. The method was further applied to the detection of dysfunctional behaviors of drilling and milling tools in collaboration with Boeing.

In addition, ENG has made awards in truly innovative research areas. One
example is:

-   An innovative proposal (CMS-99-41148) will explore the design of
    *palm-size airplanes*, among the world's smallest -- by using
    flexible wings made of composite materials. An illustration of such
    an airplane is shown in Figure 2. The flight characteristics of
    these airplanes are expected to be similar to biological
    counterparts. These aircraft could be employed in unmanned
    surveillance and reconnaissance missions.

#### Indicator 3---Partnerships connecting discovery to innovation, learning, and societal advancement:[successful]{.underline}

ENG has been consistent in its development of innovative partnerships
across a variety of different sectors. We have already listed some
examples of such partnerships in our prior discussion of indicators: for
example, the *Consultation on National Natural Hazards*. Below, we
provide examples of other partnerships, spanning partnerships with city
government, with industry, and with other sectors of the university.

-   One ENG-funded project (CMS-95-26094) established a partnership with
    the City of Houston to produce fundamental research on new materials
    and technologies that support an ambitious ten-year Greater Houston
    Wastewater program to rehabilitate old pipes and add new pipelines
    and facilities.

-   The ERC for Neuromorphic Systems Engineering at Caltech (EEC-
    94-02726) envisions a new era of machines whose behavior is inspired
    by the sensing and control systems of biology. Results have been a
    stimulus for substantial commercial investment, with the creation of
    seven spin-off companies, participation in both the spin-offs and
    the center by a variety of established companies, and the potential
    for creating a neuromorphic systems industry.

-   The University of Washington Engineering Biomaterials ERC
    (EEC-95-29161) has developed curriculum options and degree programs
    to prepare students for the cross-disciplinary field of
    biomaterials. They have developed an interdisciplinary Minor in
    Engineering Biomaterials for undergraduates, a graduate Biomaterials
    Certificate Program has been approved by the College of Engineering
    and the School of Medicine, and a Professional Certificate Program
    has been developed for practicing engineers.

#### Indicator 4---Research and education processes that are synergistic: [successful.]{.underline} 

The integration of research with education has been a hallmark of ENG
programs. Some specific examples are provided below:

-   In an innovative interdisciplinary program focused on real
    engineering issues, graduate and undergraduate students in
    transportation and structural engineering, seismology, economics,
    and other disciplines studied the *effects of earthquakes on urban
    highway infrastructure productivity* (CMS-96-33386). The project is
    a model of interdisciplinary research.

```{=html}
<!-- -->
```
-   *The Engineering Projects in Community Service* (EPICS) program
    (EEC-99-03195) partners teams of engineering undergraduates who earn
    academic credit for multi-year projects that solve challenging
    technology-based problems for local community service organizations.
    Based at Purdue University, and now involving eight universities and
    a Corporate Partners Program, over 90 products have been delivered
    since its inception in 1995 to service organizations including the
    Wabash Centers Children's Clinic, the Homelessness Prevention
    Network, and Habitat for Humanity.

### Tools---Providing "broadly accessible, state-of-the-art information bases and shared research and education tools" 

Through the Engineering Research Centers (ERCs), the National
Nanofabrication Users Network (NNUN), and the Network for Earthquake
Engineering Simulation (NEES), ENG has made a consistent contribution to
the development of broadly-enabling shared resources in the form of
information databases and shared tools. As shown in Figure 3, the easy
availability of costly, state-of-the-art equipment to the general
research community opens the possibility for broader participation and
contributions in critical areas of science and technology. The Advisory
Committee judges that ENG has been overwhelmingly successful in
implementing this strategic outcome. We note with some concern, however,
that the 'accounting' of this part of the ENG portfolio may not have
been given the emphasis or prominence that it deserves. The FY 2001
allocation to 'TOOLS' was listed as \$2.8 million out of a total budget
of \$430.84 million (or .65%). We believe that in fact there are
tremendous investments in TOOLS not made apparent through this kind of
accounting, which are part of ENG's portfolio throughout its programs,
in particular through the ERC program. We recommend that a more accurate
accounting be made of ENG's contributions to the development of broadly
accessible, enabling tools, and that serious consideration be given in
the development of next-generation tools to be supported by the
Directorate. To highlight this point, we give specific examples relative
to the indicators below.
![](media/image3.jpeg){width="1.2819444444444446in" height="1.9875in"}

#### Indicator 1--- Shared-use platforms, facilities, instruments, and databases that enable discovery and enhance the productivity and effectiveness of the science and engineering workforce: [successful]{.underline}

ENG's contribution to shared-use tools encompasses databases,
computational tools and fabrication capabilities.

-   Predicting the ground motion of large basins during strong
    earthquakes (CMS-93-18163, CMS 94-19696, CMS 95-03991,
    CMS-99-80063): The Quake project, funded by NSF and DARPA under the
    High Performance Computing and Communication/Grand Challenge
    Application Grant program, is a joint effort by seismologists at the
    University of Southern California and the Department of Civil and
    Environmental Engineering and the School of Computer Science at
    Carnegie Mellon University. The goal is to develop the ability to
    predict, by computer simulation, the ground motion of large basins
    during strong earthquakes, and to use that ability to study the
    seismic response of the Greater Los Angeles Basin.

```{=html}
<!-- -->
```
-   The National Nanofabrication Users Network (NNUN) \[99-87915\]
    provides the nation's researchers with access to advanced
    Nanofabrication equipment, techniques, and practical expertise and
    leverages its facilities with distributed design tools.

#### Indicator 2--- Networking and connectivity that take full advantage of the Internet and make science, mathematics, engineering, and technology information available to all citizens: [successful]{.underline}

-   The VaNTH ERC (EEC-98-76363) has developed graphics, simulations and
    text that can be used over the web to support learning. These items
    are modular in nature, allowing easy enhancement of the modules as
    well as straightforward integration of the module into current
    learning support materials. They are available through a variety of
    digital repositories.

-   The development of OpenSees from the Berkeley EERC (EEC-97-01568)
    enables engineers and scientists to work in an interactive fashion
    on seismic analysis of structures via the Internet.

-   The Center for Innovation and Product Development (EEC-95-29140) has
    developed a Distributed Object-based Modeling Environment (DOME) to
    enable product designers to share information pertaining to product
    design over the 'World Wide Modeling Web'. Several major companies
    have implemented DOME type systems.

#### Indicator 3- Information and policy analyses that contribute to the effective use of science and engineering resources: [successful]{.underline}

We believe that with respect to this particular indicator, ENG has made
contributions that are *exceptionally strong.*

-   The University of Iowa I/URC for Virtual Proving Ground Simulation
    (EEC-97-04242) has developed fundamentally new computationally
    high-speed dynamics formulations that enable real-time,
    operator-in-the-loop vehicle dynamics simulations. Because of this
    technological development, the U.S. National highway Traffic Safety
    Administration (Dept. of Transportation) and the University
    committed a total of \$65 million collectively for the construction
    of a *National Advanced Driving Simulator.* When operational, this
    facility will be used for the conduct of highway safety research,
    aggressively seeking to reduce the number of vehicle crashes that
    result in over 42,000 deaths per year in the U.S. at an annual cost
    to society of \$150 billion.

-   The Power Systems Engineering (EEC- 96-16221) multi-university
    initiative will develop policy analyses and market principles for
    management of electrical power systems to support competition and
    efficient use of resources in a deregulated industry.

-   The I/UCRC for Health Management (EEC-97-96320) manages a number of
    projects that inform~~s~~ members about key issues and challenges in
    health management. Their work influences administrative and clinical
    leadership in member organizations.

E. **Assess the performance of the directorate in achieving the
performance goal on the use of the new NSF merit review criteria. NSF\'s
performance is successful when (a) reviewers address the elements of
both generic review criteria appropriate the proposal at hand, and (b)
program officers take the information provided into account in their
decisions on awards.**

1.  [Goal for Reviewers]{.underline}: As, in previous years, the
    Advisory Committee found that the *intellectual merit* criterion was
    adequately addressed in the comments by reviewers, and played an
    important role in their decisions. However, the COVs continue to
    find examples of uneven application of the review criterion related
    to *broader impact* (Merit Criterion 2). The CMS COV found that
    while NEES and CAREER reviewers addressed both criteria, Criterion 2
    was *not addressed* for most unsolicited proposals. The EEC COV also
    found that less detail was provided in addressing Criterion 2,
    although the reviewers for the ERC program addressed this criterion
    'to a greater extent than many other NSF programs'. We are aware
    that the ENG Division Directors and the Program officers have been
    instructed to inform reviewers of the importance of *both criteria*.
    However, it appears that further guidance needs to be given to
    reviewers about the meaning, importance and application of the
    *broader impacts* criterion. Based on the evaluations of the COVs
    regarding the variable use of the second criterion for merit review,
    the Advisory Committee rates performance on the use of the *broader
    impacts criterion* as [not successful]{.underline}.

2.  [Goal for Program Officers]{.underline}: In general, the COVs found
    that the Program Directors have acted on the recommendations of
    their reviewers. Therefore, where minimum emphasis was given by
    reviewers to Criterion 2, a similar minor emphasis was given by
    Program Directors. The Advisory Committee thus similarly rates this
    performance as [not successful]{.underline}. We were pleased to
    learn, however, at our October 2001 meeting with members of the ENG,
    that the Directorate has itself started its own survey and
    assessment of the utilization of Criterion 2 by both reviewers and
    Program Directors across the entire division. 292 reviews were
    assessed, involving 61 program officers. We look forward to the
    final results of that analysis, and the Directorate's own decisions
    on how to better promote the usage of Criterion 2.

F. **Summarize how well the directorate is positioned to attain NSF\'s
outcome goals. Identify opportunities for future improvement.**

The COVs and the Advisory Committee anticipate that the future
performance of the Engineering Directorate can continue to be successful
with respect to each of the NSF\'s outcome goals. Specific opportunities
to further enhance performance include:

[Actions Suggested to Enhance Future Performance for Strategic Outcome
1-]{.underline} Building a Diverse, Globally-engaged workforce of
scientists, engineers, and well-prepared citizens

In our report of the Directorate\'s Performance for FY 2000 for GPRA,
the AdCom commended the development of a new initiative, Research
Experience for Teachers (RET). We are delighted that ENG has gone ahead
with this program, with an initial \$2 million investment, and continues
to develops the strength of its REU programs, and its investment in the
CAREER program. We believe that ENG is pursuing a multitude of ways in
which it interacts with, and engages the public, and recommend that
these activities continue to grow. We especially commend the leadership
ENG exercised recently, in rapidly marshalling its resources to help
address engineering issues associated with the events of September 11,
2001.

As we mentioned in last year's AdCom report, the determination of
success in meeting this or any of the strategic outcomes would be
facilitated by the collection of longitudinal data. In the case of
Strategic Outcome 1, this would include data on the participation of
women, minority and disabled individuals in NSF-sponsored programs that
tracks their contributions throughout programs and career paths. We were
delighted that such data is accessible through the ERC program, and has
been made available to the AdCom.

[Actions Suggested to Enhance Future Performance Toward Strategic
Outcome 2]{.underline} -- Enabling discovery across the frontier of
science and engineering, connected to learning, innovation and service
to society.

ENG has developed several initiatives to support NSF outcome goals, and
has taken the leadership in the *Nanoscale Science and Engineering*
(NSE) program, as well as responding with exceptional speed and
performance to the events of September 11, 2001. We applaud their
performance in these areas, and suggest that ENG has an even stronger
leadership role to play in the environmental area. The considerable
investment in infrastructure and in NEES may give rise to important new
networks that can deal with the engineering implications of hazard
assessment and response. We note that the structure and organization of
the review and award process may not be as successful as desired in
selecting out truly high-risk/high-return projects, and we recommend
that more consideration be given to exploring ways of being more
responsive to such projects. Finally, to successfully implement any of
the Strategic Outcomes, it is important for a Directorate and its
Divisions to have a strong 'sense of self', arrived at by internal
strategic planning, with a compelling vision, and the enthusiastic
commitment of its AD, Program Directors and all of its staff.

[Actions Suggested to Enhance Future Performance Toward Strategic
Outcome 3]{.underline} -- Providing broadly accessible, state-of-the-art
information bases and shared research and education tools.

As we stated in the text above, the Advisory Committee judges that ENG
has provided a diversity of broadly enabling tools through its portfolio
program. We have two principal concerns, however. One concern is that
the actual investment in tools is not being represented appropriately:
the FY 2001 ENG allocation to 'TOOLS' was listed as \$2.8 million out of
a total budget of \$430.84 million (or .65%). We believe that ENG's
contribution to Strategic Outcome 3 encompasses more than just the NNUN
and NEES programs, but are broadly distributed throughout its programs,
in particular with the ERC program. We therefore recommend that a more
accurate means be found of representing this investment. Secondly, we
recommend that ENG carry out some strategic planning to examine possible
future areas of investment in this Strategic Outcome area, and exercise
leadership in the development of new, shared engineering tools.

**G. Identify areas where there is a lack of sufficient information/data
to complete the report. Suggest types of information/data appropriate
for future reporting.**

The Advisory Committee has seen improvements since last year in the
methods of securing information for the GPRA report. However, there is
still a great deal of room for further improvement. We believe that
there are ways to accomplish this without introducing substantial,
increased administrative burden on the PI\'s and Directors. Suggestions
include restructuring the proposal submission (outcomes of previously
sponsored work), and particularly, annual review and project close-out
formats to better capture the Principal Investigators\' knowledge of new
discoveries arising from ENG-supported research and their impact on
society. The annual reports could be structured to provide the
information that explicitly addresses GPRA-related questions presented
to the COVs. The importance of providing such information should be
emphasized by the NSF to the PIs. Additional guidance to the COVs would
also be useful, so that their assessments and reports can be most
usefully integrated into the GPRA report. The Advisory Committee also
strongly encourages retrospective studies that may best be accomplished
by chartering COVs to identify a set of division-sponsored programs that
should be assessed by a future COV.

Some additional information/data that could facilitate preparation of
the Committee\'s report are in the list following. We note that the
availability of the information below would allow a true and valuable
assessment of the Merit Criterion related to the *broader impacts* of
the research supported:

-   Data on citations stemming from NSF supported research,

-   Data on further grants from other public and private organizations
    following the period of NSF support,

-   Data on new commercial ventures or public/private partnerships,

-   Information associated with spin-off applications that differ from
    the original intent of the program, but are based on the same
    scientific principles identified in the original research (these
    data might be found most easily through the long-term, center
    programs such as the ERC\'s),

-   A list of key research results, technologies developed and
    transferred to industry, patents, and other success stories for all
    proposals funded,

-   Longitudinal studies that can determine the scientific, engineering,
    or societal impact of the resources expended by the Directorate.