June 13, 2001

Dr. Mario J. Gonzalez

Department of Electrical and Computer Engineering

ENS 511, Mailcode CO803

The University of Texas at Austin

Austin, TX 78712-1084

Dear Dr. Gonzalez:

This is in response to the report of the Committee of Visitors (COV) for
the Division of Engineering Education and Centers, which you presented
on May 9, 2001 and which was adopted at that meeting.

First, I must thank you and your Committee for a thorough and thoughtful
job well done. The report stimulated a lively discussion at the Advisory
Committee meeting. I have passed your report to Dr. Bruce Kramer, the
Division Director who gave a brief response at the Advisory Committee
meeting, asking that he give full attention to your recommendations,
many of which he has already acted upon.

With regard to your comments and recommendations on refining the review
process, Dr. Kramer is working to implement your recommendation about
using a template to guide reviewers. I look forward to seeing if this
helps us get more specific reviews. Your other recommendation about the
"broader impacts" review criterion echoes what I am hearing from other
parts of the Foundation. Getting more attention given to this topic both
by reviewers and staff is a pervasive problem for the whole Foundation.

Regarding reviewer selection, this is a continuing concern. You
recommend that we seek more female, young and industry-connected
reviewers. We plan to make more efforts to do that and to increase the
participation of industry experts.

In responding to the outcomes questions, i.e., those that deal with
People, Ideas and Tools, you have provided many examples to reinforce
your rating of a "successful" performance in meeting these goals. I find
the examples you give to be quite compelling.

In your discussion of various issues of concern, you emphasize the
diversity issue of which we are continually mindful. I recognize that we
must continue our efforts and not be satisfied

with the progress we have made. I understand and share your frustration
with the limited data we are able to keep on diversity, but we will work
within NSF to rationalize the COV criteria with the data collection we
are able to perform.

The comments that you and your Committee offer on the COV process itself
will be transmitted to the developers of that system, as was discussed
at the Advisory Committee. Your observations on data needed and
preparations for the meeting itself are valuable in helping us prepare
better for future meetings. One of our divisions which is also having a
COV this year, has broken up their meeting into two separate meetings
about a month apart. I will be interested to see if this is a superior
way of relieving some of the compression that your COV experienced.

Comments and recommendations such as yours aid us in two ways. First,
they help us improve the performance of our programs. In this, the
details of your comments are most helpful. Secondly, as we are now using
the COVs to include an assessment of the outcomes of our programs, your
comments enable us to fine-tune our COV procedures so as to give us the
information the Foundation needs to develop its response to the Congress
under the Government Performance and Results Act (GPRA).

On behalf of the Foundation, let me again thank you and the members of
the COV for your efforts in helping us maintain a high standard of
service to the engineering research and education community. Through
activities such as the work of the COVs, we receive the feedback we
require to be more responsive to the needs of that community and the
nation.

Sincerely,

Louis A. Martin-Vega, Ph.D., P.E.

Acting Assistant Director of NSF

for Engineering

Enclosure

Copy:

Director, NSF

Inspector General, NSF

Chief Operating Officer, NSF

Chief Financial Officer, NSF

Chief Information Officer, NSF

Director, OIA

Members, Committee of Visitors

NSF Committee Management Officer, HRM

Bruce Kramer, EEC/ENG

REPORT TO THE NATIONAL SCIENCE FOUNDATION

**Review of the Division of Engineering Education and Centers**

**Committee of Visitors**

## March 12-14, 2001

### COMMITTEE MEMBERS:

## Ronald R. Beck

## Theodore Bogosian

## Michael Buckler

### Warren R. DeVries

#### Susan Finger

# Janie M. Fouke

## Mario J. Gonzalez (Chair)

### Sidney J. Green

### Karen Howard-Nordan

#### Wayne C. Johnson

#### Thomas R. Kurfess

## Carolyn W. Meyers

Gloria Montano

Robert P. Morgan

## Gary W. Rubloff

## John A. Schneider

## Antoinette Torres

James J. Wynne

Review of the Division of Engineering Education and Centers

**Executive Summary**

The Committee of Visitors (COV) for the Engineering Education and
Centers Division met at the National Science Foundation on March 12-14,
2001. The results of the COV deliberations are contained in the attached
report and are divided into three sections: Questions that address the
integrity and efficiency of processes and management; questions that
address results, that is, the outputs and outcomes of NSF investments;
and a feedback section that addresses program areas needing improvement,
the program's performance in meeting program-specific goals and
objectives (non-GPRA outcomes), and the COV review process.

# Processes and Management

In most cases, merit review procedures are highly efficient, well
organized, and effective, although a wide variance in the overall
quality and detail of the reviews was observed. Some of this variance
can be reduced by providing templates that specify the subjects, issues,
and accomplishments to be reviewed. Templates can provide not only a
higher level of review uniformity, but they can also serve to identify
and highlight issues and subjects (for example, review frequency, number
of reviewers) that are important to the review process.

Response to the NSF Merit Review Criteria was mixed. Overall, reviewers
and program officers did address the *intellectual merit* criterion in a
significant and useful manner. In contrast, the *broader impacts*
criterion received much less attention. If these criteria are to
continue to be an integral part of reviews, then reviewer assessment of
the extent to which proposals meet these criteria should be required. A
clearer definition of what the criteria mean and a better understanding
of what their use is intended to accomplish is needed.

Reviewer selection with respect to diversity, geographic distribution,
and professional affiliation appears to be satisfactory, although in
many cases the COV could not provide a meaningful evaluation due to a
number of factors, including missing or incomplete data, and inferences
based solely on a person's first name or place of employment. The pool
of potential reviewers should be enlarged to include more women, younger
reviewers, and more reviewers from industry.

With respect to the portfolio of awards, the COV concluded that awards
are consistent with program guidelines and reviewer recommendations.
Particularly noteworthy are the number and nature of the ERCs, awards in
new and emerging areas, the integration of research with education, and
the discretion given to ERC directors to support new investigators. The
I/UCRC program has been highly successful in promoting collaborations
with industry with relatively small NSF investments. In the future the
NSF should consider larger budgets for fewer high priority projects. An
ongoing challenge continues to be the question of how to maximize
results, productivity, and dissemination during the final phases of
program funding. Although most awards were found to be high risk,
multidisciplinary, and innovative, future evaluation efforts at all
levels will be enhanced by better definitions of these terms and by
specific responses in progress reports.

# 

# Outcomes

Programs administered by the EEC have been highly successful in meeting
the People strategic outcome goal. A wide variety of programs, including
many multi-institutional programs and programs with an international
component, are having significant, and in some cases, a dramatic impact
on diversity, curricula, and pre-college outreach. The COV recommends
that strong, pro-active steps be taken to use media professionals to
share these many successes with the general public and especially the
pre-college population. The NSF should consider budget enhancements that
promote dissemination efforts.

Similar successes have been achieved with respect to the Ideas and the
Tools strategic outcome goals. Recent and ongoing programs are providing
libraries of educational tools, have provided the impetus for the
creation of entirely new degree programs, have produced breakthrough
results that are redefining performance limits in a number of critical
technology areas, and are making significant contributions to economic
development.

# Improvements, Performance, and the COV Review Process

Much of what is included in the report with respect to these issues has
to do with metrics. The entire process of requesting and evaluating
proposals, and reviewing ongoing projects through the use of panels and
site visits should have objective measures that emphasize the importance
and the extent to which grantees address diversity, multidisciplinary,
multicultural, and multi-institutional teams, and industry
participation. Of concern to the COV, however, is that these and other
requirements do not become so onerous that individuals, teams, and
institutions are deterred from submitting proposals due to high proposal
and administrative costs.

Distance education was a prominent issue in the discussions during the
last part of the COV meeting. The issue was not the legitimacy or value
of distance education; the COV considers distance education to be an
integral part of the lifelong learning process for all citizens.
Instead, the COV focused on the ways in which distance education is
likely to impact how research is done and the leadership role that the
Foundation should assume with respect to the use of technology and new
educational models. The COV recommends that results (generated by
programs in other parts of the Foundation) that measure the impact of
technology on research and learning should be integrated into future EEC
guidelines and reviews.

Because of the maturity of the large-scale efforts funded by EEC, the
COV recommends comprehensive studies that attempt to answer the
following questions: What will EEC look like in five to ten years? Have
the Engineering Coalitions truly achieved system reform? What are the
impacts on society of EEC programs?

Finally, the COV recommends that steps be taken to make the COV review
more efficient. These steps include: a streamlining of the Core
Questions document to clarify expectations with respect to process and
deliverables; active interaction among COV members prior to the meeting
in order to specify the roles and responsibilities of each committee
member at the three stages of the COV process---prior to, during, and
after the meeting; and the provision of a context document by the EEC
that summarizes activities for the entire Foundation and includes any
comprehensive evaluations performed prior to the EEC meeting.

# Summary

Although some improvements are recommended, the COV concludes that the
EEC has been successful in meeting its process and management
responsibilities. Programs funded by the EEC have been highly successful
with respect to the most important measures: results and outcomes that
are redefining performance limits and the methods that are being used to
address fundamental engineering and societal problems; the integration
of research and education; outreach to pre-college students and to
society as a whole; and the creation of new and highly relevant
engineering education curricula and degree programs

# I. INTRODUCTION

The Committee of Visitors (COV) for the Engineering Education and
Centers Division met at the National Science Foundation on March 12-14,
2001. Members of the committee were selected to "form an independent
group of credible, external experts, selected to ensure an independent
review that reflects a diversity of perspectives and balanced
programmatic coverage." The committee charge was to review:

-   The integrity and efficiency of processes leading to awards

-   The relationships among award decisions, program goals, and
    Foundation-wide programs and goals

-   The results of awards in the form of outputs and outcomes as they
    relate to the GPRA performance goals

In order to conduct the review, the committee was divided into three
groups that comprise the portfolio of the Engineering Education and
Centers Division:

-   Engineering Research Centers

-   Industry/University Cooperative Research Centers

-   Engineering Education/Human Resources

Groups were provided access to a variety of program materials:
proposals, ad hoc and panel reviews, site visit reports, program officer
evaluations, and so forth. Because of the huge volume of materials that
accumulated during a three-year period, examination and evaluation of
materials by committee members proceeded in two steps. First, Division
staff selected a subset of materials and made them available to the
appropriate groups. Second, group members examined samples of the
subsets to form their conclusions.

These two sampling processes met with mixed reviews from the COV.
Although no one questioned the appropriateness of the representative
materials provided to the COV, in some cases the amount of material was
still huge, sometimes overwhelming. Thus, while some COV members felt
that the sampled sample was truly representative of the entire set of
materials, others felt that due to time constraints the sample that they
examined was so small that it raised concerns about the thoroughness of
the review. At the same time, no one on the COV proposed extending the
duration of the COV review meeting.

In order to comply with the charge to the committee and in order to
provide structure to the review process, committee members were provided
with a set of core questions that are grouped as follows:

-   Group A. Integrity and Efficiency of the Program's Process and
    Management. These questions (core questions 1-4) will be addressed
    in terms of programs, reflecting the manner in which committee
    members were grouped.

-   Group B. Results: Outputs and Outcomes of NSF Investments. These
    questions (core questions 5-7) will be addressed in terms of the
    individual questions, since the individual questions apply to the
    division in aggregate form.

-   Group C. Needed Improvements, Compliance with non-GPRA Outcomes,
    Feedback. These questions (questions 9-11) will be answered in terms
    of the whole division, although observations and recommendations
    that apply to individual programs will be presented.

Group A questions (1-4) were addressed by each of the three groups
during Day 1 and the first half of Day 2. Group B questions (5-7) were
addressed during the second half of Day 2. Day 3 was devoted to Group C
questions (9-11) and to an exit briefing provided to the Acting Asst.
Director.

The remainder of this report is organized as follows:

II. GROUP A QUESTIONS: Integrity and Efficiency of the Program's
    Processes and Management

```{=html}
<!-- -->
```
1.  Effectiveness of the program's merit review procedures

2.  The program's use of the NSF Merit Review Criteria

3.  Reviewer selection

4.  Resulting portfolio of awards

III\. GROUP B QUESTIONS

5.  People: Development of a diverse, internationally-competitive and
    globally-engaged workforce of scientists, engineers, and
    well-prepared citizens.

6.  Ideas: Enabling discovery across the frontier of science and
    engineering, connected to learning, innovation and service to
    society.

7.  Tools: Providing broadly accessible, state-of-the-art information
    bases and shared research and education tools.

```{=html}
<!-- -->
```
III. GROUP C QUESTIONS

```{=html}
<!-- -->
```
9.  Comments on program areas needing improvement

10. Comments on the program's performance in meeting program-specific
    goals and objectives (non-GPRA outcomes)

11. Feedback on the COV review process, format and core questions

# II. GROUP A QUESTIONS

Group A questions (Questions 1-4) address the integrity and efficiency
of the program's processes and management. In this section of the
report, questions will be addressed in terms of programs or sets of
related programs. Comments and observations will be followed by
recommendations, where appropriate.

## 

## [Engineering Research Centers (ERC)]{.underline}

Sixty-five actions were reviewed to assess the integrity and efficiency
of three type of ERC reviews: ERC Competitions, Annual Reviews,
Non-Competitive Renewals (3 and 6 Years) and special competitions
(including the Nano-Energy Research Groups (ERG) and the Equipment,
Connectivity, and Education Special Awards Programs).

1.  **Effectiveness of the program's use of merit review procedures**

Examination of ERC materials led to the conclusion that merit review
procedures are effective. The protocols for ERC competitions, renewals,
and annual reviews show open dissemination of the protocols to program
officers, ERC/EERC management, and site visitors. The review process is
open and provides useful information to the PIs of the ERCs, and
documentation collected from site visits, annual reviews, and renewals
is excellent.

Decisions on ERC non-competitive renewals and feedback from annual
reviews are efficient and timely. However, the COV believes that too
much additional time elapses from the point at which the program
officer(s) recommends funding to the award of an ERC. For the
Engineering Research Group competition in FY 00, panel summaries were
often much briefer than the reviewers' reports and thus did not provide
much guidance as to how the PIs could improve proposals. While examining
the history of proposals, it was not uncommon to find the program
director's recommendation more valuable than the review panel summary in
summarizing the response to the proposal.

Program officers are consistent in using the criteria and priorities in
program announcements and solicitations and demonstrate good judgment
and professionalism. The COV notes positively that the provision of
templates for reviewers is a good practice that reminds reviewers of the
priorities of the ERC program and NSF's two main merit criteria.

Based on the review of a diverse set of proposals and proposal actions,
the COV makes the following recommendations in order to promote process
uniformity and the effectiveness of site visits.

2.  **The program's use of the NSF Merit Review Criteria (intellectual
    merit and broader impacts)**

With respect to this question and ERCs, the COV concluded that reviewers
adequately addressed the *intellectual merit* criterion but provided
less detail when responding to the *broader impact* criterion (although
ERCs address this criterion to a greater extent than many other NSF
programs). In contrast to reviewers, program officers did a good job of
balancing *broader impact* and *intellectual merit* considerations in
their decisions.

Although not directly related to this question, the COV nonetheless
notes positively that the REU program is a powerful adjunct to the ERC
program. In awarding REU grants, the program officers are explicitly
addressing the goal of increasing the participation by underrepresented
minorities and women in the technical workforce.

3.  Reviewer selection

After examining materials to determine whether reviewer selection has
been appropriate for a balanced review, the COV concluded that:

-   The number of reviewers is adequate.

-   Reviewers appear to be qualified (although the only data made
    available to the COV were the departments or job titles and
    organizational affiliation of the reviewers).

-   Geography and type of institution are balanced (although addresses
    were the only information provided to reviewers).

-   There is a good balance between industry and academe.

There are some concerns, however. Materials available to the COV
provided no identification of members of underrepresented groups. COV
members repeatedly asked why they were asked to comment on reviewer
diversity, when NSF staff are not permitted to designate the racial or
ethnic group of a reviewer. While the diversity of reviewers exhibited
by ERCs is good, it could be better. In particular the COV noted a low
number of women reviewers, based on common female given names. The
following recommendations are derived from these observations.

4.  **Resulting portfolio of awards**

The COV's assessment with respect to the portfolio of awards is that the
many centers have benefited from the review process and the NSF
management infrastructure that is in place. Similarly, the size and
duration of awards is appropriate, but there is some concern that as the
required scope of ERCs expands, the limited resources provided by NSF
and the upfront investment in the preparation of a proposal discourages
some submissions. The trend in center awards shows greater appreciation
for the systems approach, one of the stated goals of the ERC program.
Another driving goal of the ERC program---integration of research and
education---has been addressed successfully as evidenced by many
examples of pre-college outreach and changes to engineering curricula.

The COV notes that the size of the ERC awards is small in comparison to
those that are given by other governmental agencies, such as DARPA or
DOE, for less work. For the ERG competition, funding amounts were
prorated down by 20%, apparently to match budget availability, rather
than being adjusted individually. The COV recommends that the program
officers exercise discretion in these negotiations.

The COV commends NSF for the following significant achievements:

-   Identifying and seeking proposals in bioengineering, an emerging
    opportunity area.

-   The discretion given to ERC Directors to provide opportunities for
    them to use ERC funds to support new investigators who are
    initiating their careers.

-   The strong integration of research with education as exemplified by
    the REU program and the recent Bioengineering Education ERC.

Although the COV did not have enough information to find evidence of
increased numbers of applications from underrepresented groups, it did
find that ERC awards are high risk, multidisciplinary, and demonstrate
innovative systems approaches to engineering.

# 

# [Industry/University Cooperative Research Centers]{.underline}

Twenty-two actions were reviewed to assess the integrity and efficiency
of Industry/ University Cooperative Research Centers (I/U CRCs).

1.  **Effectiveness of the program's use of merit review procedures:**

Based on the overall success of the program, the COV concluded that the
review process is indeed working, although a wide variance in the
overall quality and detail of the reviews was observed. Decision-making
in this program is efficient and timely, and documentation was timely
and extensive throughout all phases of the review process. The following
recommendation is provided in order to simplify and add precision to the
review process.

2.  **The program's use of the NSF Merit Review Criteria (intellectual
    merit and broader impacts)**

With respect to the *intellectual merit* criterion, most reviewers (and
all program officers) responded with in-depth comments, addressing
project proposals individually. Other reviewers, however, did not
provide specific information. In contrast, the *broader impacts*
criterion was not addressed in a meaningful manner by most reviewers,
although most reviewers and most program officers recognized the
significant impact of these programs on graduate student research. These
disparities lead to the following recommendation for future COVs.

3\. Reviewer selection

For I/UCRCs, reviewers were represented adequately in terms of numbers
and qualification, although in some instances, reviewers were mostly
from academic institutions. It was not possible to determine if members
of underrepresented groups were included to an appropriate extent due to
the absence of diversity information.

4.  **Resulting portfolio of awards**

The program portfolio is very broad, leading several members of the COV
to conclude that the program could benefit by limiting and prioritizing
the topics that I/UCRCs should address. The quality of science and
applied research is high and focused appropriately. In spite of a high
level of success, the COV concluded that the program should be enhanced
by an increase in award to each center with a concomitant increase in
the program budget.

Centers supported by this program are responsive to their industrial
sponsors, they move rapidly toward emerging technologies, and provide a
prime source of support for young investigators who use this support to
establish industrial contacts and research collaborations. In contrast
to these positive observations, the COV found no evidence or
documentation to address the question of gender and diversity in these
programs.

Projects funded by the I/UCRC program are uniformly high risk
(especially from an industrial perspective), highly multi-disciplinary
(I/UCRCs have been leaders in establishing multidisciplinary and
multi-institutional research programs), and particularly innovative in
creating collaborative research partnerships among multiple industry and
academic institutions and providing opportunities to prepare students to
work in selective industries.

[Engineering Education and Human Resources Development]{.underline}

Thirty-six actions were reviewed to assess the integrity and efficiency
of the following programs: REU, Action Agenda, Coalitions, and CRCD.

1.  **Effectiveness of the program's use of merit review procedures**

An examination of materials led to the conclusion that most of the time
panels are highly efficient, well organized, and effective (this is
particularly true for Action Agenda and CRCD panels); assessments are
thorough and well documented; and the time to reach a decision as part
of the entire review process is excellent. A notable exception was REU
documentation. Generally speaking, REU documentation was incomplete with
respect to the review process. Some reviewers failed to write any
comments at all. Panel rating summaries exhibited a variety of errors
and ambiguous or imprecise recommendations, thereby making it difficult
to follow the justification of the final decisions.

With respect to panel composition, COV members are concerned that some
of the site renewal panels had only two outside members. COV members
concluded that in most cases this number is insufficient. Similarly, the
COV is of the opinion that programs should try to have at least four
reviews to make a decision (instead of the standard three), particularly
for the REUs. These observations lead to the following recommendations.

2.  **The program's use of the NSF Merit Review Criteria (intellectual
    merit and broader impacts)**

Some reviewers in some programs did address the *intellectual merit*
criterion, although as suggested earlier, more attention should be given
to the extent to which both reviewers and program officers address the
*broader impacts* criterion.

3\. Reviewer selection

In most cases, the number of reviewers was adequate (but not for REU) as
was the scientific background and the balance of the reviewers. The
number may have been inadequate for the REU program but was adequate for
the other programs (CRCD, Action Agenda).

4.  **Resulting portfolio of awards:**

Funded projects within these programs display appropriate science and
engineering quality. In most cases, duration size and scope were
appropriate. Of concern to the COV, however, is that the low frequency
of site visits/oversight for coalition programs since the last COV
review was not adequate. More active oversight is necessary for programs
of this size.

The COV review did not provide ample opportunity to address diversity
issues adequately, although the materials that were reviewed did
indicate a low application rate from members of underrepresented groups.
Similarly, examination of sample materials did not provide sufficient
information to assess the high risk, multidisciplinary, and innovative
nature of proposals.

##### GROUP B QUESTIONS (Questions 5-7)

Group B questions (Questions 5-7) address results, that is, the outputs
and outcomes of NSF investments. In this section, each question will be
addressed separately in terms of all programs within the division.

**5. PEOPLE Strategic Outcome Goal: Development of a diverse,
internationally- competitive and globally-engaged workforce of
scientists, engineers, and well-prepared citizens.**

Programs administered by EEC have been highly successful in meeting this
goal. Examples of programs that have contributed to this success are
noted below.

-   The ERC for Reconfigurable Machining Systems (EEC-95-29125) at the
    University of Michigan is creating an exhibit for a local museum
    that will enable visitors to explore the interdependence and impact
    of product design, manufacturing processes, and business issues.

    -   The Earthquake Engineering Research Center (EEC-97-01568) at
        Univ. of California, Berkeley is supporting an *Earthquakes*!
        Exhibit at the California Academy of Sciences to educate the
        public about the use and advantages of performance-based
        earthquake engineering design.

-   NSF statistics show that large fractions of the students involved in
    the ERCs for FY 2000 were female and underrepresented minorities
    (URM). The 600 pre-college students included 48% female and 36% URM
    students; the 4000 undergraduate students included 29% female and
    12% URM students; the 2200 graduate students included 22% female and
    5% URM students; and the 344 REU students included 39% female and
    35% URM students.

-   During the period in which the Gateway Coalition (9727413) was
    established, the retention of African American engineering students
    from the freshman to the sophomore year in the Gateway Coalition
    universities increased from 67% in 1992 to 87% in the year 2000. For
    women students, this retention increased from 75% to 90%, and for
    all engineering students retention increased from 79% to 86%.

-   The goal of the SURE REU Packaging Research Center, an ERC at
    Georgia Tech (EEC-9402723 / REU 9820254), was to expose minority
    engineers to microelectronic packaging research and to interest them
    in graduate study. The success of the program is indicated by the
    fact that 90% of the participants in the program decided to go to
    graduate school.

```{=html}
<!-- -->
```
-   A female graduate student at the Ohio State I/UCRC (EEC-9523358) was
    awarded the Kenan Award from Union Carbide Corp, which is given to
    only ten senior graduate students in the country per year. There are
    seven women co-directors of I/UCRCs. Forty women and minority REU
    grants were awarded at the forty I/UCRCs during FY 2000.

    -   At the University of Tennessee at Knoxville's I/UCRC Maintenance
        and Reliability Center (CRCD 9901565), work was sponsored by
        industry and supported by CRCD grants to develop four new
        courses for academic programs. Distance learning versions of
        these courses were presented at participating universities,
        including the University of Sao Paulo in Brazil.

    -   Workshops utilized the case study from the NSF-sponsored
        learning factory at the University of Puerto Rico, Mayaguez
        (Action Agenda 9812928) to address ABET 2000 accreditation
        criteria. These workshops were held at international conferences
        in Brazil and the Czech Republic.

```{=html}
<!-- -->
```
-   Software developed at the Iowa State I/UCRC for Nondestructive
    Evaluation (CNDE) (9872590) is being used as a teaching tool at both
    the community college and university levels. This has involved CNDE
    industrial sponsors who have an interest in identifying future
    employees with this training.

-   In 1999-2000, I/UCRC Centers had 755 graduate students and 224
    undergraduate students involved in their programs. There are several
    instances of I/UCRC Centers generating Co-op experiences and
    internships. In the same time frame, there were 948 publications and
    1105 presentations from all centers.

Although extensive pre-college outreach is not a required feature of ERC
and EERC programs, significant activity and success have been achieved
in reaching the K-12 population through these programs. A few notable
programs, followed by recommendations, are noted below.

-   The ERC for Environmentally Benign Semiconductor Manufacturing,
    University of Arizona, Arizona State University, Cornell, MIT, and
    Stanford (EEC-95-28813) and (SRC96-EC-425) has sponsored three-week
    high school teacher institutions during the summers of 1997-2000.
    Teams of teacher learned about Center research programs, toured
    facilities, shadowed faculty and graduate students, developed lesson
    plans, and wrote exercises to augment their curricula. In the
    summers of 1998-2000 the Center offered a no-cost three-day high
    school teacher conference in Tucson to introduce teachers to the
    Center, high tech job opportunities, and to give them kits
    containing experiments and computer modules developed at the teacher
    institutes.

-   The ERC for Particle Science and Technology at the University of
    Florida (EEC-9402989) seeks to encourage all pre-collegiate
    students' interest in engineering. In addition to day-visit tours,
    the primary vehicle for this outreach is through a series of four
    middle school and high school Summer Engineering camps designed to
    provide hands-on laboratory experiences that introduce students to
    the field and increase awareness of Particle Science and Technology
    careers. The ERC hosts up to four students each year and places them
    on multidisciplinary research teams consisting of both undergraduate
    and graduate students.

-   The aim of the outreach activities of the ERC for Innovation in
    Product Development at MIT (EEC-9529140) is to provide experiences
    in product development research and practice for minority students,
    women, and their teachers. The objectives of the Center are to give
    teacher-advisors new educational tools and insights, and to promote
    interest in science and technology that will lead high school women
    to careers in engineering, and it has done this through a series of
    two-day workshops and publications.

Through science museum exhibits, science fairs, and K-12 student
sponsored competitions, the public, in particular parents of the
students, have access to the processes and benefits of engineering
research and education. ERCs are encouraged to continue these practices
and to encourage graduates to mass communications careers devoted to
science and technology.

**6. IDEAS Strategic Outcome Goal: Enabling discovery across the
frontier of science and engineering, connected to learning, innovation
and service to society.**

As a result of demonstrating significant achievement in several key
indicators, programs administered by the division were successful in
meeting this goal.

-   The Bio-Engineering Technology Engineering Research Center
    (Vanderbilt, Northwestern, Harvard/MIT, and The University of Texas
    at Austin\--NSF#EEC-9876363) was established to mount a "broad-based
    attack on the barriers to our vision of an integrated effective
    system for . . . engineering education". Innovations include a Java
    module library of common components; common visualization tools;
    model curricula for undergraduate and graduate education; and a wide
    range of innovative educational technology delivery systems and
    interactive learning software.

-   The vision of the ERC for Low-Cost Electronic Packaging at Georgia
    Tech (EEC-9402723) --high performance, low cost Systems on a Package
    (SOP)\--drives the realization of microelectronic systems for
    numerous and diverse applications. This effort has brought these
    methods to a broad spectrum of industry users, including materials
    suppliers, manufacturers of chips and data storage devices, as well
    as telecommunications, automobile, and defense industries. The ERC
    has established a series of courses which constitute a focused
    program of study for undergraduates in the packaging-related
    engineering disciplines. A Practice-Oriented Masters Program is in
    full operation.

-   The Data Storage Systems Center (an ERC at Carnegie Mellon
    University) has developed two-element actuator control systems for
    improved speed and accuracy in tracking in high density magnetic and
    optical disk drives. The Center has played a pivotal role in the
    industry, as evidenced not only by its large industrial funding, but
    also by the establishment of a major industrial research facility
    (Seagate) near the Center in Pittsburgh, directed by the former and
    founding ERC director.

-   The ERC for Neuromorphic Systems Engineering at Cal Tech
    (EEC-9402723) envisions a new era of machines whose behavior is
    inspired by the sensing and control systems of biology. Results
    produced by this center have been a stimulus for profound commercial
    investment, with the creation of seven spin-off companies,
    participation in both the spin-offs and the Center by a variety of
    established companies, and the potential for creating a neuromorphic
    systems industry.

-   The University of Washington Engineering Biomaterials (UWEB) ERC has
    developed curriculum options and degree programs to prepare students
    for the cross-disciplinary field of biomaterials. They have
    developed an interdisciplinary Minor in Engineering Biomaterials for
    undergraduates, a graduate Biomaterials Certificate Program has been
    approved by the College of Engineering and the School of Medicine,
    and a Professional Certificate Program has been developed for
    practicing engineers.

-   The I/UCRC Center for Health Management Research (University of
    Washington/ University of California\--Berkley, 9796320) has several
    projects evaluating quality improvements in hospitals, projects on
    clinical practice patterns, projects addressing information systems,
    and projects which provide prenatal services for teens.

-   The I/UCRC at SUNY Buffalo (9815211) studies biology on surfaces.
    This is an example of research at the interface between the physical
    sciences of surfaces and the life sciences of biological organisms.

-   During 1999-2000, 18 I/UCRC Centers had invention disclosures. All
    the I/UCRCs together had 69 invention disclosures, 30 patent
    applications, 3 software copyrights, 9 patents granted, and 12
    licensing agreements concluded, in addition to the 948 publications
    in the same time period.

```{=html}
<!-- -->
```
-   EmTech is a biotechnology incubator created between Emory University
    and Georgia Tech. With substantial contributions from each campus
    and also from state government agencies responsible for economic
    development, this biotechnology center will focus on the medical
    device-biotechnology interface to take advantage of new economic
    opportunities.

-   The Engineering Projects in Community Service (EPICS) program
    partners teams of engineering undergraduates who earn academic
    credit for multiyear projects that solve challenging
    technology-based problems for local community service organizations.
    Based at Purdue University and now involving eight universities and
    a Corporate Partners Program, over 90 products have been delivered
    since its inception in 1995 to service organizations including the
    Wabash Centers Children's Clinic, the Homelessness Prevention
    Network, and Habitat for Humanity.

```{=html}
<!-- -->
```
-   The Green Design project at Carnegie Mellon University (EEC-9700568)
    has developed modules that enable students to perform lifecycle
    analyses. The modules and case studies enable students to understand
    the environmental consequences of design decisions. The materials
    and tools are available on the web and have been used by students
    from around the world.

####### 

**7. TOOLS Strategic Outcome Goal: Providing broadly accessible,
state-of-the-art information-bases and shared research and education
tools.**

As indicated by the sample of funded projects summarized below, the
division's programs have been successful in meeting this goal.

-   A review of ERC annual reports revealed that over 0.5 million square
    feet of ERC space has been leveraged under the auspices of this
    program initiative, reflecting a significant institutional
    commitment to the goals of shared research. ERCs also reflect the
    joint activity of universities and industry.

-   The VaNTH ERC (NSF#EEC-9876363) has developed graphics, simulations
    and text that can be use over the web to support learning. These
    items are modular in nature allowing easy enhancement of the modules
    as well as straightforward integration of the module into current
    learning support materials. They are available through a variety of
    digital repositories.

-   The development of OpenSees (Berkeley, EERC, EEC-9701568) enables
    engineers and scientists to work in an interactive fashion on
    seismic analysis of structures via the internet

-   Three EERCs involve a total of 39 universities and provide 171
    academic and 111 industrial and government participants access to
    facilities and equipment such as shake tables.

-   The Center for Innovation and Product Development (NSF#EEC-9529140)
    developed a Distributed Object-Based Modeling Environment (DOME) to
    enable product designers to share information pertaining to product
    design over the "World Wide Modeling Web." Several major companies
    have implemented DOME type systems, and DOME developers received
    over 500 requests to give presentations.

```{=html}
<!-- -->
```
-   Iowa State University's I/UCRC for Nondestructive Evaluation,
    (NSF# 9872590) developed new techniques to inspect composite
    materials. The FAA has augmented the program to develop a Computer
    Aided Tap Testing System that is showing great promise to replace
    the old manual tap process. Sixteen field tests have been completed
    at airlines, OEMs, and military depots. Beta tests are currently
    underway at American Airlines, British RAF, Scaled Composites and
    Northwest Airlines. Negotiations are underway with three prospective
    licensees.

-   The University of Iowa I/UCRC for Virtual Proving Ground Simulation,
    (NSF \# 9704242) has developed fundamentally new computationally
    high-speed dynamics formulations that enable real-time
    operator-in-the-loop vehicle dynamics simulations. Because of this
    technological development, the U.S. National Highway Traffic Safety
    Administration of the Department of Transportation and the
    University committed \$54 Million and \$11 Million, respectively,
    for the development and construction of the world-class National
    Advanced Driving Simulator. When operational, this facility will be
    used for the conduct of highway safety research, aggressively
    seeking to reduce the number of vehicle crashes that result in over
    42,000 deaths per year in the U.S. at an annual cost to society of
    \$150 Billion.

```{=html}
<!-- -->
```
-   Rensselaer Polytechnic University's CRCD, Multimedia Based Tools for
    Integration of Technology, Analysis and Design Materials Handling
    Education (NSF #9980261), provides students with the decision making
    and judgment skills exhibited by professionals who have experience
    in integrating engineering analysis with technology selection. Using
    the world's largest knowledge base of materials handling\--from the
    Materials Handling Industry of America\--RPI has developed a
    curriculum that introduces students to categories of material
    handling equipment, provides examples of major material handling,
    and teaches problem solving through use of models.

-   SMETE.org (University of California, Berkeley\-- NSF #9872570) is an
    outcome of the NEEDS Coalition which has evolved as an Action Agenda
    e-learning partnership (of academic institutions, educational
    organizations, and private e-learning companies) and integrative
    organization that collects SMET content and distributes pedagogical
    materials through a federation of digital library content
    repositories. SMETE.org offers direct access and delivery to a broad
    constituency to enable relevant curriculum development.

-   The National Nanofabrication Users Network (NUNN) Program (NSF
    #9987915) housed at Cornell represents an REU that has developed a
    unique strategy to share equipment. NUNN provides the nation's
    researchers with access to advanced nanofabrication equipment,
    techniques, and practical expertise by leveraging its facility by
    distributing design tools. Design tools are available broadly and
    can be submitted to the fabrication center at Cornell.

```{=html}
<!-- -->
```
-   Power Systems Engineering (NSF# IUCRC, 9616221)is a multi-university
    initiative with the goal of developing policy analyses and market
    principles for management of electrical power systems to support
    competition and efficient use of resources in a deregulated
    industry.

-   The I/UCRC for Health Management, (NSF#9796320), a collaboration
    between the University of Washington and the University of
    California, Berkeley, manages a number of projects that inform
    members about key issues and challenges in health management. Their
    work influences administrative and clinical leadership in member
    organizations regarding strategies and tactics to address these
    issues.

# IV. GROUP C QUESTIONS

9.  **Provide comments on program areas needing improvement.**

Suggestions for improvements fall into various areas. Some of the
suggestions have been addressed earlier in this report and are repeated
here for completeness.

9.  [Proposal and Post-Award Review]{.underline}

In general, the COV concluded that more effective proposal and
post-award review processes are needed. In this regard the COV raised a
number of key questions:

-   What are appropriate metrics for educational accomplishment?

-   Who sets the metrics?

-   Who evaluates accomplishments and outcomes with respect to these
    metrics?

After extensive discussion, the COV concluded that the responsibility
for defining metrics and measurable outcomes associated with those
metrics rests with faculty. That is, subject to program guidelines,
faculty submitting a proposal in response to a program announcement
should identify the metrics and the outcomes by which success will be
measured. Responsibility for post-award review falls into two
categories: the institution(s) receiving the funding, and the program
officer(s) with the National Science Foundation.

The COV raised many questions about the appropriate frequency for
various types of evaluations and the duration of funding, particularly
with respect to the Engineering Coalitions. For ERCs and I/UCRCs ten
years of funding may be too long in some cases, especially when the
rapid change of technology is considered. (This question was also raised
with respect to ERCs and I/UCRCs.) Evaluations should be done at a rate
that maintains direction with respect to the original goals and
objectives, yet allows for mid-course corrections prompted by new
knowledge and new technology while not being unnecessarily burdensome.
Particularly vexing is the question of the nature and frequency of
post-award reviews toward the end of a funding stream when issues of
dissemination and implementation are critical.

10. [Impact]{.underline}

As noted earlier, the COV expressed many concerns about two impact
questions: the broader impacts criterion, and diversity.

1.  Broader Impacts. This criterion was ignored by most participants in
    funded programs and by most reviewers. The COV concluded that the
    primary reason for this is that the criterion and its implied
    requirements are not well understood. If the meaning of this
    requirement is not well defined, then it follows that precise
    metrics cannot be provided or assessed. If this criterion is to
    continue to be an integral part of assessment and evaluation
    processes, then considerable effort should be devoted to
    clarification of meaning and expectations.

\(2\) Diversity. The many dimensions of diversity---race and ethnicity,
gender, geographical distribution, academic vs. industry---are key
elements of the entire lifetime of a proposal. These dimensions are of
particular interest to reviewers (panelists, site visitors, COV members)
who are asked to address questions dealing with diversity in very
specific ways. Yet in most cases, reviewers are unable to respond
because the information is not available or it can be inferred only by
indirection (that is, by a person's title or name or by a person's
professional affiliation). The net effect of this situation is to
produce indifference on the part of many program participants and
frustration on the part of reviewers. This leads to the following
recommendation.

(3) Distance Education. Extensive discussion among COV members and with
    division staff led to the following observations and questions:

    -   About half of the programs funded by EEC use distance education
        in some form.

    -   Distance education has the potential to dramatically transform
        education and research environments. Little has been done to
        project, measure, and respond to this transformation within the
        Foundation's funding processes.

    -   Training is very different from education, but is likely to have
        a major impact on the broader issues of education and research.

    -   What does diversity mean in the context of distance education?

In view of these observations and the explosive nature of the education
revolution already underway, the COV makes the following recommendation.

(4) Industry Participation. Because of the nature of the division's
    programs, industrial participation in a sustained and meaningful way
    is crucial. However, with the exception of ERCs and I/UCRCs,
    industry participation in various types of reviews and advisory
    committees has not been at a level that is consistent with the goals
    of the division.

Just prior to adjournment, each COV member was asked to make closing
comments, observations and suggestions. Member responses not covered
elsewhere appear to be most appropriate for Question/Item 9 and are
included here as recommendations that are broader in scope that other
recommendations.

10. **Provide comments on the program's performance in meeting
    program-specific goals and objectives (non-GPRA outcomes).**

After an extensive discussion that considered the materials reviewed by
the individual sub-groups, the COV concluded unanimously that the EEC
Division has been successful in meeting its program-specific goals and
objectives. However, in the spirit of continuous quality improvement,
various aspects of the evaluation process led to the following
recommendations.

11. **Provide feedback on the COV review process, format and core
    questions.**

The feedback provided in response to this request falls into three
categories: Core Questions, preparation before the COV meeting, and the
conduct of the COV meeting.

a.  Core Questions

The core questions provided to COV members prior to the meeting managed
to convey a reasonably clear idea of what the COV was expected to do
during the meeting. However, during the meeting, COV members expressed
some frustration and confusion due to the following:

\(1\) Each of the Core Questions 1-4 contained a number of
sub-questions. Although these sub-questions play a valuable role in the
COV review process, COV deliberations proceeded with an understanding
that all of these sub-questions needed to be answered by all sub-groups.
This led to sub-group draft reports that were unnecessarily lengthy and
did not conform to expectations for the format of the final COV report.

\(2\) The part of the Core Questions document that deals with Questions
5-7 should be reformatted. In its current form, when the COV addresses
the Outcomes part of the review, it is confusing and difficult to
distinguish between STRATEGIC OUTCOME GOALS, FY 2001 Performance Goals,
and the specific questions addressed by the goals numbered 5, 6, and 7.

b.  Preparations Before the Meeting

c.  The COV Meeting