**SRI International**

May, 2000

**Final Report**

**Progress of the Engineering Education Coalitions**

**By:**

H. Roberts Coward, SRI International

Catherine P. Ailes, SRI International

Roland Bardon, SRI International

**Prepared for:**

Engineering Education and Centers Division

National Science Foundation

**1611 N Kent Street • Arlington, VA 22209-2192 • (703) 524-2053 • FAX:
(703) 247-8569**

**Acknowledgments and Disclaimer**

SRI International wishes to acknowledge the extensive and detailed
assistance, as well as the hospitality, of the many Engineering
Education Coalition faculty and other participants -- especially
students, as well as university administrators and staff, who aided in
providing information to the study team that performed this effort. Of
special note are the efforts of the team from SRI's subcontractor COSMOS
Corporation, who performed much of the field research, which involved
eleven campus visits by COSMOS and SRI personnel. The encouragement and
helpful review of NSF staff members, including Ernest Smerdon, Susan
Kemnitzer, and Linda Parker are also greatly appreciated.

The views expressed in this report are those of SRI International alone,
and do not necessarily represent the views of the National Science
Foundation or the U.S. government.

**TABLE OF CONTENTS**

**Table of Contents** i

**EXECUTIVE SUMMARY** iii

**I. INTRODUCTION AND METHODS** 1

A.  Background: the Engineering Education Coalitions Program 1

B.  Study Methods 4

**II. FINDINGS FROM FIELD RESEARCH** 6

A. Curriculum and Instructional Delivery 6

B.  Coalition-building efforts 7

```{=html}
<!-- -->
```
1.  Communication Across Coalitions 7

2.  Concrete Operational Plans 7

3.  Institutional Commitment 7

4.  Reassessment of Goals in Light of Experiences 8

5.  Dimensions and Complexity of Tasks 8

```{=html}
<!-- -->
```
C.  Assessment Practices 8

D.  Impact on Students 10

E.  Faculty and Academic Culture Outcomes 12

**III. FINDINGS FROM CAMPUS-LEVEL PRINCIPAL**

**INVESTIGATOR INTERVIEWS** 15

A.  Interactions 15

B.  Dissemination 16

C.  Challenges 17

D.  Broad Issues and Special Concerns 18

**IV. FINDINGS FROM STUDENT FOCUS GROUPS** 20

A.  Coalition Courses 20

B.  Computer-based Instruction 21

C.  Student Recruitment and Retention in Coalition Programs 21

D.  Design Projects 22

E.  Teaming Efforts 23

F.  Industry Response 24

G.  Students\' Overall Coalitions Experience 24

**V. FACULTY SURVEY** 25

A.  Coalition Inputs 25

B.  Curriculum and Instructional Delivery 27

C.  Student Outcomes 29

D.  Outcomes: Faculty and Academic Culture 31

**VI. SURVEY OF DEANS OF SCHOOLS OF ENGINEERING** 32

**VII. CONCLUSIONS AND RECOMMENDATIONS** 37

A.  Overall Findings 37

```{=html}
<!-- -->
```
1.  Progress Towards Reform Thus Far 37

2.  How Long Should Reform Take? 38

3.  Conditions Impeding Reform 38

4.  Conditions Accelerating Reform 39

```{=html}
<!-- -->
```
B.  Building on the Progress of the Coalitions 39

**ATTACHMENTS**

A.  Survey of Faculty Participants A-1

B.  Survey of Deans/Academic Affairs B-1

# EXECUTIVE SUMMARY

The need for significant changes in the character of engineering
education emerged in a series of studies beginning in the mid-1980s,
culminating with the 1989 Belmont Conference on *Imperatives in
Undergraduate Engineering Education*. The National Science Foundation
(NSF) responded to this challenge in 1989 with the announcement of the
Engineering Education Coalitions, an initiative to support a small
number of major Coalitions of U.S. institutions in a multi-year effort
at a level of \$2-3 million for up to five years to pursue the vision
and goals outlined in their proposals within the NSF Program\'s
objectives. Over the course of three annual competitions, NSF funded six
Coalitions. Program goals included:

-   a dramatic increase in both the quality of engineering education and
    the number of degrees awarded in engineering, including those to
    women and underrepresented minorities;

-   the design, implementation, evaluation and dissemination of new
    structures and approaches affecting all aspects of undergraduate
    engineering education;

-   the establishment of new linkages among all types of U.S.
    engineering institutions, large and small.

At the mid-term of the Program's prospective life span, NSF desired a
review that would synthesize the broad variety of information available
on this complex endeavor in order to provide a description of the
progress of the Coalitions Program during its first five years of
operation. The intent was not to evaluate either the Program overall, or
any individual Coalition or its members, but to capture an overview of
the progress of the coalitions collectively toward achieving the Program
goals. Consonant with the complex character of the Program, SRI
International, working with COSMOS Corporation, conducted a
multidimensional study that used a variety of information gathering
tools and sought to derive from the results a comprehensive picture that
would meet NSF's requirements. The elements of the study consisted of:

1)  Archival review;

2)  Field visits to the lead and one other member institution in each
    Coalition;

3)  Focus group interviews with students using Coalition-developed
    products;

4)  Telephone interviews with the Coalition leader at institutions not
    visited;

5)  A survey of faculty participants in Coalition projects at the
    various member institutions;

6)  A survey of deans of engineering schools and colleges not involved
    in a Coalition.

The time-frame of the study was essentially the prospective midpoint of
the Coalitions' life span -- 1997-1998 -- the point at which they were
being reviewed for their second phase of funding, depending on which
competition had funded the individual Coalitions.

The Coalitions Program has had many important impacts during the first
five years, but these cannot be said to be "the comprehensive and
systemic new models for engineering education reform" anticipated. Most
impacts had been intra-institutional, indeed, intra-disciplinary.
Participating institutions cover less than one-third of engineering
faculty and one quarter of engineering students in the United States,
and not all of their engineering faculty and students participate in
Coalition-influenced courses. Dissemination is nominal at this stage:
there have been many conference presentations and much exchange of
information, but the Dean survey showed very limited evidence of actual
adoption outside participating institutions, and there remains
resistance within even those, ranging from the resistance of
conservative faculty to the "not invented here" syndrome. Participating
faculty are enthusiastic and generate a wide variety of innovations, but
are not temperamentally suited, nor professionally motivated, to
generate the follow-on documentation that aids in adaptation or adoption
at other institutions. Coalition products generally get good reviews,
but primary interaction between the innovators and the rest of the
community is fleeting, documentation limited, and adaptability widely
questioned. Costs in terms of faculty effort, equipment, and customized
classroom configurations are high. Coalition courses add to student
workloads at a time when there is pressure to reduce the number of
credit hours needed for an engineering degree.

Information on student outcomes was derived primarily from a survey of
faculty participants and a series of focus group interviews with
students, which were carried out during the field research visits.
Faculty regarded students taking Coalition courses as rating highly on
Coalition-related criteria such as working in groups and tackling
interdisciplinary projects. Both surveyed faculty and student groups
agreed that the workload associated with Coalition-based courses was
heavier than traditional ones, and was perceived as a negative by both
groups. The students, however, generally recognized that the effort had
given them valuable experience, often to their advantage in the job
market and other long-term professional prospects.

Many problems were identified in the course of the study, but the
Coalitions have made progress. Many materials have been developed,
pedagogical research legitimized as part of the academic reward
structure, and, perhaps most important, significant contributions made
to the efforts of U.S. engineering education to meet ABET 2000 criteria.
The complexity of the task of "comprehensive curriculum restructuring"
is perceived by Program participants as being greater and more difficult
than envisaged in the time-frame imposed on the original Program.
However, given the complex problems facing undergraduate engineering
education when the Program began and the changes that have occurred
since then, the Coalitions Program was widely viewed as the right thing
to do at the time, and as having provided the critical funding increment
to get things moving. There is, in fact, widespread concern about how
the momentum initiated by the Program will be sustained when Coalition
funding is phased out.

With several years of funding remaining in the Coalitions Program, there
are important opportunities to build on the progress that has been made.
A significant base of enthusiastic human resources has been developed,
as well as a wide range of products that meet Program criteria. The
Program can seek to enhance the adoption of the best work produced by
assisting in selecting the "best of the best" and encouraging the types
of documentation and dissemination efforts that will facilitate the
spread of these products through the engineering education system.

**I. Background and Methods**

## A. Background of the Engineering Education Coalitions Program

The need for significant changes in the character of engineering
education emerged in a series of studies beginning in the mid-1980s,
culminating with the 1989 Belmont Conference on *Imperatives in
Undergraduate Engineering Education*. Among the needs identified during
this period were: a shift from disciplinary approaches toward broader
synthesis of knowledge and interdisciplinary modes of operation,
including the ability to work in teams; a systems approach to
problem-solving; greater knowledge and consideration of social,
environmental, and other implications of problem-solving efforts, along
with management skills and a capacity for life-long learning; increased
ability to attract and retain students in fields of engineering, as well
as increased participation of under-represented groups; more emphasis on
engineering practice, including early opportunities to encounter the
design process; and interaction with industry. These needs were
generally viewed within the context of concerns about the ability of
U.S. industry to be competitive in the emerging global economy through
rapid and successful innovation.

The National Science Foundation (NSF) responded to this challenge in
1989 with the announcement of the Engineering Education Coalitions, an
initiative to establish a small number of major Coalitions of U.S.
institutions in a multi-year effort at a level of \$2-3 million for up
to five years, to pursue vision and goals outlined in their proposals
within the NSF Program\'s objectives.[^1] These included:

-   a dramatic increase in both the quality of engineering education and
    the number of degrees awarded in engineering, including those to
    women and underrepresented minorities;

-   the design, implementation, evaluation and dissemination of new
    structures and approaches affecting all aspects of undergraduate
    engineering education;

-   the establishment of new linkages among all types of U.S.
    engineering institutions, large and small.

Over the course of three competitions, NSF funded six Coalitions. The
Coalitions were implemented through co-operative agreements, a funding
arrangement between NSF and the lead institution that included
monitoring of the Coalitions by NSF throughout the five-year term of the
award. While awards were originally intended to be for a single term,
NSF decided to allow each Coalition to submit a proposal for an
additional five-year award as they neared the end of their original
funding.

NSF monitoring included annual site visits to Coalition institutions. An
in-depth review occurred during the third year that determined whether
to continue full funding or phase out funding during the remaining two
years. In the fifth year of each Coalition award, each Coalition had the
opportunity to submit a proposal for up to five more years of funding,
which was required to focus on plans for evaluation,
institutionalization and dissemination. Review of these proposals
determined the level of support that NSF would provide during a second
five-year award. Each Coalition received on average \$2 to \$3 million
per year. While collectively this represents a large block of money,
field interviews pointed out that the structure of Coalitions, with a
lead institution\'s overhead and multiple institutional members
conducting multiple projects, meant that only small amounts -- on the
order of several thousand dollars -- flowed out of the end of the
pipelines to individuals and/or teams for specific projects.

The first competition funded two Coalitions, ECSEL and Synthesis. ECSEL,
which stands for Engineering Coalition of Schools for Excellence in
Education and Leadership, was a geographically and structurally diverse
set of schools intent on the desire to incorporate design throughout
undergraduate engineering education, as well as the recruitment and
retention of women and underrepresented minorities in engineering. Its
membership included:

  -----------------------------------------------------------------------
  **ECSEL** (Funded 1990-94)

  Howard University

  Massachusetts Institute of Technology

  Morgan State University

  CCNY of The City University of New York

  The Pennsylvania State University

  University of Maryland (current lead)

  University of Washington
  -----------------------------------------------------------------------

Synthesis was the second Coalition established in 1990. Like ECSEL,
geographically and structurally diverse, Synthesis has one of the most
abstract and broad visions of all the Coalitions. Founded on the
\"synthesizing" of the teaching and learning process and its
facilitation through advanced information technology. More concretely,
it focused on courses in "mechatronics," the merging of mechanical and
electronic devices in contemporary engineering. Synthesis membership
included:

  -----------------------------------------------------------------------
  **Synthesis** (Funded 1990-94)

  California Polytechnic State University---San Luis Obispo

  Cornell University

  Hampton University

  Iowa State University

  Southern University

  Stanford University

  Tuskegee University

  University of California
  -----------------------------------------------------------------------

The second competition resulted in the 1992 funding of the Gateway and
SUCCEED Coalitions. Gateway sought to develop a combination of
curriculum integration and development (CID), faculty and student
development (human potential development -- HPD), and "embedded
educational technology" (ETM). Its starting point was the "E^4\"^
integrated curriculum developed at Drexel University. Its members were:

  -----------------------------------------------------------------------
  **Gateway** (Funded 1992-96)

  Columbia University

  Drexel University

  New Jersey Institute of Technology

  Polytechnic University--- Brooklyn, NY

  The Cooper Union

  The Ohio State University

  University of Pennsylvania

  University of South Carolina
  -----------------------------------------------------------------------

SUCCEED is a geographically coherent Coalition, the "Southeastern
University and College Coalition for Engineering Education." Its
strategy was the development of a model "curriculum 21" aimed at the
21^st^ century that combined an integrated engineering core with the
sciences and humanities to be spread and institutionalized through its
membership. The latter included:

  -----------------------------------------------------------------------
  **SUCCEED** (Funded 1992-96)

  Clemson University

  Florida A&M/Florida State University

  Georgia Institute of Technology

  North Carolina A&T State University

  North Carolina State University

  University of Florida

  University of North Carolina---Charlotte

  Virginia Polytechnic Institute and State University
  -----------------------------------------------------------------------

The third competition funded Foundation and Greenfield Coalitions
beginning in 1993. Foundation Coalition was thematically based on an
integrated freshman curriculum originally developed at Rose-Hulman
Institute of Technology, combined with similar development efforts at
Texas A&M, which was to be adapted to its diverse membership and its
concepts extended into the upper division and the full engineering
curriculum. Membership was unusual in including a community college and
the need for articulation of courses. Members of Foundation included:

  -----------------------------------------------------------------------
  **Foundation** (Funded 1993-97)

  Arizona State University

  Maricopa Community College District (MCCD)

  Rose-Hulman Institute of Technology

  Texas A&M University---College Station

  Texas A&M University---Kingsville

  Texas Woman\'s University

  University of Alabama
  -----------------------------------------------------------------------

The Greenfield Coalition for New Manufacturing Education is unique. A
principal feature of the Coalition is its relationship to Focus:HOPE, an
organization focused on civil rights and human resource development,
located in Detroit, Michigan, that serves the non-traditional students
who are often ignored by academia. The mission of the Coalition includes
developing three new programs that focus on manufacturing engineering
and engineering technology. All but one of Greenfield's active members
(Lehigh University) are located in the Detroit area, and its
manufacturing focus is primarily on the needs of the automobile and
related industries. In addition to Focus:Hope, the members are:

  -----------------------------------------------------------------------
  **Greenfield** (Funded 1993-97)

  Lawrence Technological University

  Lehigh University

  University of Detroit Mercy

  University of Michigan

  Wayne State University
  -----------------------------------------------------------------------

It is worth noting that three of the Coalitions were curriculum "model
based": Gateway and Foundation with existing models developed by one or
more of their members, SUCCEED with its objective of an integrated
Curriculum 21. The other three were "theme based": ECSEL, Synthesis, and
Greenfield were more conceptual in their basis, although oriented toward
curriculum reform and development.

## 

## 

## B. Study Methods

At the mid-term of the Program's prospective life span, NSF desired a
review that would synthesize a broad range of information in order to
provide a description of the progress of the Coalitions Program. The
intent was not to evaluate either the Program overall, or any individual
Coalition or its members, but to capture an overview of the progress of
the coalitions collectively in toward achieving the Program goals.
Consonant with the complex character of the Program, SRI International,
working with COSMOS Corporation, conducted a multidimensional study that
uses a variety of information gathering tools to derive a comprehensive
picture that would meet NSF's requirements. The elements of the study
consisted of:

1)  Archival review;

2)  Field visits to the lead and one other member institution in each
    Coalition;

3)  Interviews and focus groups of students using Coalition-developed
    products;

4)  Telephone interviews with the Coalition leader at institutions not
    visited;

5)  A survey of faculty participants in Coalition projects at the
    various member institutions;

6)  A survey of deans of engineering schools and colleges not involved
    in a Coalition.

A study strategy had to be devised to provide an intellectual framework
that would pull the wide-ranging aspects, activities, outputs and their
impact of this extremely complex program. This involved developing an
overview of program inputs, activities, outputs and their impacts. To
accomplish this, a networking meeting was held early in the project,
bringing together the study team, representatives from each of the
Coalitions, and NSF Program staff. The objective of the meeting was to
lay out the overall dimensions and context of the program being
assessed, along with its varied outputs and expected impacts, aiding in
the identification of contextual issues by forcing a description of the
attribution of causality among the sequences of interventions and
desired outcomes of the program.[^2]

This report is organized around the various data gathering methods used.
Generally speaking, each method considered the identified program
elements:

-   Inputs, such as Coalition building efforts and faculty participation
    in Coalition-sponsored projects;

```{=html}
<!-- -->
```
-   Outputs, including curriculum materials and instruction techniques,
    and assessment efforts;

-   Outcomes with students, faculty and changes in academic culture; and

-   Impacts, in terms of the Program's goal of introducing comprehensive
    changes and restructuring of undergraduate engineering education.

With the first Coalitions having been funded in 1990 for an initial
period of five years, the time frame of the data gathering aspects of
the study occurred during a period shortly after NSF began the mid-term
reviews of the Coalitions for their second five-year term of funding.
Specifically, these took place during the following periods:

1)  Field research visits (including student focus groups): April,
    1997-February, 1998 -- most in the fall of 1997;

2)  Faculty survey: May 19, 1998-March 11, 1999;

3)  Engineering school dean survey: September 29, 1998-March 22, 1999.

While the full range of data collection required nearly two years, it
should be emphasized that the study could only capture a picture of the
progress of the Coalitions within a limited period near the mid-term of
the program. Moreover, a "snapshot" of the Coalitions was all that was
intended in carrying out the study: it was in no way intended to
evaluate the program overall, the individual Coalitions, or any of the
members of the Coalitions.

# II. FINDINGS FROM FIELD RESEARCH 

## A. Curriculum and Instructional Delivery

Two types of products were of special interest to NSF and therefore the
main topics of data collection: curriculum and related practices
(courses and instructional delivery systems) and assessment practices
(see next section). The main question of interest in terms of curriculum
and related practices was the extent to which the Coalitions extended
the state-of-the-art of engineering education, and whether the
extensions appeared to be modifications, incremental advances, or truly
novel innovations.

The Coalitions supported the development of many new curriculum
practices, mainly taking the form of revised or new courses in
engineering. In general, curriculum practices and other related
activities at the Greenfield Coalition must be considered apart from the
other five Coalitions. Greenfield supports development of a new
curriculum specifically in manufacturing engineering, using
computer-based instruction modules in an institutional setting that is
distinct from their participating institutions. The other Coalitions are
trying to innovate within existing institutional settings.

Aside from the unusual organization of the Greenfield Coalition, among
the other five Coalitions, all of the new courses demonstrated the
variety of features desired by NSF: e.g., students working in teams
rather than independently and including cooperative learning, especially
in the earlier undergraduate years; increased use of contemporary
educational technology, with computer-based methods of delivering
courses increasingly taking the place of traditional lectures; and
integration of engineering with other disciplines, such as biology,
writing courses, and social science and humanities more generally. Many
of the courses resulted in increased interactions with industry, with
firms sponsoring courses and providing equipment, supplies, and guest
teachers. One enhancement is 'just-in-time' teaching, whereby concurrent
lecture and laboratory courses are sequenced so that lecture topics are
covered just as they become needed in the laboratory, whereas in the
past the lecture and laboratory courses might have proceeded almost
independently.

Of all these changes, perhaps the most pervasive innovation across
Coalitions was that of providing meaningful engineering experiences for
freshmen, ranging from 'professional level' laboratory facilities to
realistic design projects. (Traditionally, most engineering programs
have postponed this experience until the junior or senior year.) Some of
the new courses received national recognition for their quality.
However, although changes in the freshman year inevitably have some
impact on the teaching of upper-level courses, the Coalitions' new
courses were only beginning -- as of the time of the assessment -- to
reflect larger strategic shifts and the evolution of large-scale
upper-level and cross-disciplinary curriculum changes. Most of the new
pedagogical practices were 'within campus' innovations, not involving
substantial collaboration at the Coalition level. Except for the changes
in the introductory courses, most also were limited to one or two
departments in engineering (especially mechanical engineering), not the
whole school. To this extent, the innovations in curriculum and
instructional delivery practices to date may still be considered
incremental. Individual faculty have invested a lot of time in improving
specific courses and employing Coalition-like principles in doing so;
however, the institutional reward system for undertaking such efforts
does not appear to have changed, so that pressures for a faculty member
to spend more time in doing research do not appear to have diminished.
(See Section \[VI\] for a discussion of the impact of Coalitions on
academic culture.)

## B. Coalition-building Efforts

The study considered five types of Coalition-building strategies that
were identified in the study's initial meeting with key Coalition
participants and NSF staff: the extent and manner of communications
among Coalition members; use of principles of experimental design and
guidance by concrete operational plans; the dynamics of institutional
commitment; reassessment of goals in light of Coalitions' experiences;
and the dimensions and complexity of tasks.

### Communication Across Coalitions

Across Coalitions, the diversity of processes and circumstances
precludes easy generalizations. Some of the Coalitions worked hard to
develop and operationalize such a vision (e.g., Gateway and SUCCEED);
others appear to have interpreted the communication goal mainly as an
attempt to build bridges across institutions. Even intense
cross-institutional communication, in the absence of such a vision, is
not likely to advance the interests of the Coalition as a whole. The
desired operationalization of the vision usually took the form of some
plan for changing the undergraduate curriculum, based on increasing
agreement over the key concepts within the vision.

### Concrete Operational Plans

Explicit operational strategies for dealing with the spread and
institutionalization of change within individual universities were
markedly absent from the Coalition-building process. In theory, the
scope of the desired undergraduate reform would suggest that changes
cannot be limited to single or a few engineering disciplines
(departments); that nearly all engineering faculty must become part of a
changed way of doing business; and that some critical changes must occur
in schools outside of engineering as well as for an institution as a
whole. The field visits and other sources revealed little in the way of
explicit plans for this type of spread or institutionalization process,
other than as implied by a Coalition's curriculum planning (e.g.,
lower-level courses first, then upper-level courses). Only a minority of
the engineering faculty at any given institution seem to have been aware
of their Coalition's breadth and vision, fewer still participated in the
Coalition's activities.

### Institutional Commitment

Coalition-building also needs to be fueled by sufficient institutional
commitment. There were instances in which one or more deans had made
early and sustained commitments to a Coalition (e.g., ECSEL and
SUCCEED). Several Coalitions had instances of external funding that
supplemented NSF's funding. However, it was difficult to ascertain the
extent of "true" commitment by an institution or even an individual.
Thus, there were occasions when the field visit team was told that
faculty were not encouraged to participate in a Coalition's activities
because it detracted from their research productivity. Whether member
institutions will continue with the Coalition's work (full reform, going
beyond the revisions of individual courses) after NSF funding ends seems
questionable.

Industry has definitely been involved and appreciative of Coalition
efforts, but its level of commitment and involvement is limited. In-kind
contributions of equipment and some funds were noted during field
interviews, but 42% of faculty survey respondents had worked with
individuals from industry in teaching courses and laboratories.
Greenfield's direct ties to the automobile industry in the Detroit area
was the strongest industrial connection evident, but, as indicated
elsewhere in this report, Greenfield has a number of characteristics
unique among the Coalitions.

### Reassessment of Goals in Light of Experiences

Feedback about the Coalition-building process did not occur early
through formal means, such as evaluation or assessment. In general,
evaluation and assessment activities appeared to be isolated (occurring
within a single course, department, or institution, but not uniformly
across an entire Coalition in any systematic manner). More detailed
comments about the Coalitions' assessment experiences are contained in
Section C, below.

### Dimensions and Complexity of Tasks

With regard to the dimensions and complexity of the tasks confronting
the Coalitions, it should be noted that the Coalitions were confronted
with three difficult processes simultaneously: scaling up (if an
innovation appeared promising); institutionalizing (within a single
institution); and coalescing plans and actions across institutions.
Other conditions such as geographic distance and the disbursing of funds
through a single institution on behalf of an entire Coalition may have
made it harder to carry out these processes well. Further, all of these
processes are extremely time-consuming, and it is to the Coalitions'
credit that, given extreme dedication on the part of certain groups of
faculty and participants, the Coalitions have made the progress
exhibited thus far.

## C. Assessment Practices

NSF's intent in requiring Coalitions to institute assessment practices
was to ascertain the quality and outcomes of any new curricula and
institutional changes initiated by the Coalitions. Such assessment does
not appear to have been either an explicit goal from the Belmont
Conference or of the Coalitions Program as originally designed by NSF,
although it had become an explicit requirement by the second
competition. Because of lack of previous exposure to education or social
science *outcome* assessment, some of the Coalitions initially thought
that the desired "assessment" was actually quality control as the term
is used in industry. This misconception had to be overcome, and, in
recent years, both the Coalitions and the broader engineering education
community have come to understand what NSF originally intended. The
desired type of student outcome assessment is now articulated and
emphasized in the new *Engineering Criteria 2000* by the Accreditation
Board for Engineering and Technology, Inc. (ABET), which will be in use
for all accreditation actions by the year 2001. This includes assessment
of student learning and employment outcomes. However, NSF also intended
that Coalitions conduct assessment of new curricular materials and
instructional delivery systems while in the pilot stage and use the
assessment results for continuous improvement prior to dissemination
beyond the originating faculty developers.

The data show that assessment has been slow in starting and a spotty
activity among the Coalitions. Nearly all of the available assessments
have been conducted within institutions or even within single courses,
with little overall strategy for synthesizing lessons learned across
institutions or courses to get a broader perspective on a Coalition's
work. The paucity of assessment information suggests that it is
minimally used for continuous improvement.

Further, whether within institutions or courses, the assessments have
not been well implemented because the member institutions have generally
been unable to overcome such conditions as:

-   Difficulty in defining common data collection instruments across
    courses or institutions;

-   Unacceptably low response rates, whether from faculty members or
    students;

-   Infrequent use of or lack of access to archival sources (e.g.,
    student records) that might overcome the previous two conditions
    better than newly-defined surveys or focus groups;

-   Inability to define non-Coalition groups of students or faculty for
    comparison purposes; and

-   Lack of useful databases to capture more easily Coalition and
    non-Coalition students' demographic and course performance
    information.

Even when such conditions have been overcome, the assessment results
cannot be easily interpreted. For instance, one of the more advanced
assessment teams, at Arizona State University (Foundation Coalition),
found that more freshmen enrolled in the Coalition program than freshmen
enrolled in the traditional program in Fall 1994 were still in
engineering two years later (Spring 1996); however, the results were
reversed for the subsequent cohort of students (freshmen enrolling in
the Fall 1995). When GPAs were compared, those of the comparison group
exceeded those of the Coalition group[^3]. The evaluators have been
perplexed by these findings and have not yet been able to explain them.
Among other possibilities, the baseline equivalence of the two groups
needs to be carefully scrutinized. Similarly, some assessment of
specific efforts, such as SUCCEED's STEPUP program, cannot be readily
interpreted. The local evaluation found that minority students'
retention rates and GPAs were similar to those of non-minority students;
however, whether this represented a positive or disappointing result was
not clearly stated.

Possibly typical of the general challenges confronting assessment teams
have been the experiences reported by another of the more advanced
assessment teams, working with the ECSEL Coalition. The team, led by an
excellent group of investigators from one of the Coalition universities
(Penn State), admitted to being unable, during the first five years of
their work, to collect the type of quantitative data that had been
desired across the Coalition's institutions. As a result, an adaptation
being tried in the second five years is to pursue a *meta-analytic*
strategy, whereby different instruments and measures may have been used
but a common statistic, such as *effect size* is calculated to aggregate
quantitative findings across campuses. However, even within an
institution, the ECSEL team encountered difficulty in collecting
quantitative data on outcomes, with most of the information coming from
a handful of open-ended interviews or from focus groups of students.

Whether even good qualitative data (e.g., case studies of the
Coalitions) are being collected also may be questioned, however. For
instance, only rarely did the study team encounter examples of efforts
to track policy changes in the institutions and the role of Coalitions
in stimulating such change. For example, one such policy change was
reported by the assessment team at Arizona State University without
noting specific Coalition influence. This exemplifies the need for more
Coalition goal-related qualitative data that could have been tracked
more thoroughly by all of the assessment teams. The change was a new
policy of 'post-tenure review' adopted by the Arizona Board of Regents,
whereby all faculty, even those with tenure, will be reviewed annually
for their contributions to teaching, research, and service.[^4] A
similar change took place at the University of Florida, where the state
established a Teaching Improvement Program (TIP) incentive for
engineering faculty. Faculty receive a \$5,000 increase over base salary
for excellence in teaching. During the past three years, 66 engineering
faculty at the University of Florida have received TIP awards. If such
policies are faithfully implemented and become more common across
institutions, a more general culture change favoring the values of the
Coalitions Program may occur.

#  At the same time, an important caveat is that the difficulties encountered by those doing assessments of all kinds should not necessarily be considered specific to the Coalitions Program. Assessments have been successful, but only under special circumstances, such as a focus on a specific course or innovation and where all participants agree to be part of a 'study.' for university-wide assessments, going beyond commonly tracked grade, enrollment, and graduation data is a notoriously difficult task, whether within or outside of engineering. Efforts such as the collaboration between Gateway and SUCCEED since 1996 to work together to develop assessment measures for new educational innovations may therefore represent moves in a promising direction.

## D. Impact on Students

"Outcomes" represent the first meaningful effects of Coalitions, and
these have been examined in terms of three categories: the impact on
students, the impact on faculty, and changed academic culture. This
section discusses the impact on students -- the impact on faculty and
academic culture follow in the next section.

Student outcomes show many examples of the learning behaviors valued by
the Coalitions program -- e.g., students gaining exposure to practical
problems related to industry, students learning to work as part of
teams, students gaining hands-on experiences with computers and other
equipment, and students gaining exposure to engineering design during
their freshman and sophomore years. At the same time, the listing of
outcomes oversimplifies some of the operational challenges in putting
the desired processes into practice. For instance, for cooperative
learning among students, teaming was done in many institutions but
followed different approaches regarding how individuals are assigned to
teams; how individuals are coached to work on teams; whether all team
members will get the same grade; and how to overcome scheduling problems
for teamwork, especially in environments where some students are also
maintaining outside jobs.

Among the student outcomes of high priority, the Coalitions do not
appear to have tracked, systematically, the level of minority and women
students and their level of satisfaction or likely success in
progressing through the engineering curriculum. Most of the Coalitions
have only begun to emphasize new outreach efforts, aimed at recruiting
such students in the first place, but are not yet ready to deal with
issues of progression and retention. Information about student outcomes
tended to come from single institutions or single courses, covering a
variety of outcomes potentially related to but not directly covering
retention and progression. Figure II-1 lists these outcomes and their
illustrative examples.

Among the six Coalitions, Greenfield and any outcomes attributed to this
Coalition may need to be considered apart from those of the other five
Coalitions. Completing the Greenfield program can lead to any of three
degrees: an associate degree, a bachelor of engineering degree, or a
bachelor of science in manufacturing engineering technology. The first
11 students in the bachelor's programs completed their degrees in the
summer of 1998. Early feedback to the evaluation team indicated that the
initial graduates were offered exceptional positions and valued highly
by the corporate partners, because they had engineering degrees and also
had attained machinist level qualifications. While its *modus operandi*
assures a high rate of women and minority enrollment, the attrition rate
from the three programs has been about 50 percent. The Coalition claims,
however, that nearly all candidates leaving before graduation have been
enticed by attractive job offers from industry and not confronted by
academic difficulties.

**Figure II-1**

**Illustrative Student Outcomes from**

**Coalition Courses, Modules, and Labs**

![](media/image1.wmf){width="6.311805555555556in"
height="6.308333333333334in"}

## E. Faculty and Academic Culture Outcomes

In addition to student outcomes, the study design posited that the
Program should result in outcomes reflected in changed faculty
approaches to teaching and in the overall academic culture of the member
institutions. In the study, the faculty outcomes of interest were
observations of any shifts from traditional, passive, lecture-based
instruction to newer models of active, collaborative learning (e.g.,
coaches rather than all-wise dispensers of knowledge). The desired
academic culture upon which the Coalitions Program is based calls for a
shift away from an exclusive focus on individual faculty achievement,
disciplinary specialization, and wholly research-based reward
structures, broadening the focus to include team contributions,
integration and not just specialization of knowledge, and educational
(not just research) scholarship and innovation.

Evidence of such changes in culture was spotty and possibly premature to
expect at this stage. No assessments yet involve case studies or other
qualitative examination of such shifts. Further, complete study of
cultural shifts also involves collecting data from non-engineering or
non-university perspectives (such as industry representatives).
Nevertheless, although evidence of desired cultural changes at the time
of the study was limited, the field visits and surveys did provide some
evidence of changes taking place. These could hardly be characterized as
strong trends, but there seemed to be desirable, if relatively recent,
developments in four areas.

The first and possibly most important pattern across Coalitions was the
*increased valuing of contributions to the practice of engineering*.
SUCCEED's Quality Improvement Partnership project called for individual
university members each to form a partnership with a local firm, with a
preparatory total quality systems course followed by summer internships
at the sponsoring industrial site. Students worked in multi-functional
and multidisciplinary teams on real production and development problems
and created a record of producing significant cost savings for the
plants. Similar benefits are reported in SUCCEED's Integrated Product
and Process Design course, with feedback from industrial recruiters
suggesting that the students have the equivalent of two years' job
experience. ECSEL's upper level product engineering and manufacturing
course also has resulted in students having their redesigns actually
adopted by the partnering firm.

A second consistent theme across all Coalitions was the *development of
cross-institutional and not just cross-disciplinary perspectives and
interactions*. These include the Synthesis Coalition's mechatronics
initiative, as well as the benefits derived from sharing equipment and
facilities at other institutions, as in Gateway's scanning tunneling
microscope and rapid prototyping machine. SUCCEED's integration of
biology and engineering promoted cross-institutional collaboration by
deliberately calling for joint proposals from multiple institutions to
develop new courses and avoid duplication across campuses. Such
collaboration also included cross-disciplinary team-teaching and
laboratory development. The articulated transfer of engineering courses
from a community college to a university in the Foundation Coalition
required inter-institutional collaboration. In a similar manner, the
Synthesis Coalition helped one of its members, a historically black
college or university, to gain ABET accreditation for its newly
established engineering program. Given the overall pattern of the
relatively narrow and largely intra-institutional scope of interaction
during the Coalitions' early years, these are useful developments.

A third potential cultural shift involves *outreach to non-engineering
students and departments*. These include: recruitment of women and
minorities into engineering, as in Foundation's mathematics,
engineering, and science achievement project; a mix of physics and
engineering students enrolled in an upper-level course in the Gateway
Coalition; and SUCCEED's biology and engineering collaboration.
Foundation's integrated engineering program also involves interaction
with the university's English department, and some laboratories,
although created mainly for engineering, are being used by a large
number of students in the liberal arts, with a number of examples of
applications in fields other than engineering. Despite these instances,
however, outreach beyond engineering schools has not been one of the
stronger elements of Coalition efforts.

A fourth shift is evidenced by isolated examples (and, indeed,
counter-examples) of *shifts away from a research-only culture of
promotion and tenure*. In ECSEL, one university has adopted a long-range
plan moving the engineering school away from a program emphasizing
engineering science to one that is more oriented toward the practice of
engineering in a broad context. At another ECSEL university, promotion
and tenure processes are being changed to include recognition of
teaching and involvement in undergraduate engineering. To balance
excellence in research with excellence in education, a Synthesis
Coalition institution plans to use "excellence in teaching" awards as
part of the promotion and tenure process. On the other hand, there also
were clear reports of untenured faculty being given the message that
tenure decisions will be based primarily on research rather than
including education contributions

# III. FINDINGS FROM CAMPUS-LEVEL

# Principal Investigator INTERVIEWS

For the purpose of broadening the study's understanding of the impact of
Coalition participation on campuses that were not included in the field
visits, the perceptions of Coalition leaders at schools not visited were
sought by telephone interviews with the campus PI or leader. Where
changes in leadership had occurred, the individual with the broadest
"institutional memory" was sought out for this purpose. In order to
limit the demands of the interview, efforts were made to focus the
discussion on:

1)  important aspects or special developments in the ***history*** of
    that institution's involvement in the Coalition;

2)  ***interactions*** within the institution, the Coalition, with other
    Coalitions, and outside the Coalition community;

3)  ***dissemination*** efforts;

4)  important ***challenges*** encountered in the past and those seen in
    the future.

The historical elements having been used in conjunction with other parts
of this report, only the last three and some issues that broadly
affected nearly all Coalitions are discussed here.

## A. Interactions

Responses concerning the types and focus of interactions engendered by
the Coalitions Program usually represented personal observations and
experiences, but there were some consistent themes -- both positive and
negative -- and they were generally consistent with the picture that
emerged from the field visits.

On the positive side, a number of respondents pointed to the utility of
travel funds to actually observe what was going on elsewhere. This might
be to observe a course, courseware and its classroom use, or computer
laboratories prior to attempting to adapt and/or adopt these outside the
developing institution. It was also useful for Coalition faculty to
observe where cultures of change had emerged at other institutions. This
was especially important when the member institutions were widely
dispersed. It was reported, for example, that there appears to be more
interaction among SUCCEED institutions and the institutional members of
other Coalitions in the southeast than with more distant members of
other Coalitions: i.e., SUCCEED\'s geographic focus is the basis for
interaction -- not the Coalition program.

Campus-level funds also permitted Coalition members to share facilities
-- Cooper Union used Columbia's Coalition-driven computer laboratory,
other schools have used the University of South Carolina's
Communications Center, New Jersey Institute of Technology, and Ohio
State provided access to rapid prototyping facilities to Gateway
members. (However, in the last case, the practice was abandoned due to
its high costs.)

On the down side, it was clear that both intra-Coalition and
inter-Coalition interaction was difficult to achieve outside of meetings
arranged by NSF, or professional meetings such as the ASEE or FIE. Most
that did occur appeared to represent two-way relationships, although
opportunity to build a network of acquaintances with similar interests
at other schools was often cited as something of great value -- another
observation consonant with field visit interviews. However, a few campus
leaders conceded that their interaction had been largely passive. In
particular, one HBCU PI indicated that his campus had done little in the
way of carrying out projects under the Coalition, largely evaluating
and, in some cases, modifying and adopting Coalition-developed
materials. Campus PIs reported that interaction with institutions
outside the Coalitions Program was nearly non-existent.

## B. Dissemination

Interviewees were asked about dissemination activities and results. The
issue of dissemination was raised in the PI interviews because other
sources of information made it clear that dissemination is a serious
problem and that it is not simply a problem of gaining broader "buy-in"
beyond the originating institution.

The telephone interviews replicated observations from the field visits
with repeated admissions that dissemination was slow to come, behind
schedule, and generally disappointing. The most obvious problem noted
during the field visits was the difficulty of providing documentation
that permitted transfer. Faculty enthusiasts enjoyed projects developing
new courses, projects, and other materials. The problem of getting these
documented derived from many factors. The conflicting pressures on
faculty, the small amounts of money provided individual projects at the
end of the pipeline, and faculty skills at writing and presenting
materials outside the classroom all contributed to dissemination
problems. Despite the frequent mention of publications and conference
presentations in field visits and PI interviews, a paper at a conference
does not provide coursework in and of itself. CD-ROMs and computer
modules may be informally distributable, but without documentation are
neither easily implemented as part of courses nor "commercializeable."
The results of the survey of non-Coalition engineering schools seems to
confirm that, aside from a couple of textbooks and a few CD-ROMs,
Coalition materials seem idiosyncratic and hard to adapt. One problem
identified in the interviews is that the Coalitions Program has produced
its own Catch-22: it has legitimized scholarly publication concerning
curriculum reform, but this takes precedence in the reward structure and
tends to preempt any efforts that most faculty are prepared to undertake
to work up Coalition products for dissemination.

The phone interviews confirmed these problems, as well as suggesting a
broader range of issues concerning the problems involved in
dissemination. Perhaps the broadest, most important, and frequently
noted point is that dissemination must be seen as a process -- not just
the distribution of a product. One campus PI formulated the process as
one of generating awareness first, followed by a need for adaptation,
and only then hope for adoption by other institutions. An allied comment
was that there was a lot more communication going on than dissemination.
Several emphasized the need -- and difficulty, even on their own
campuses -- to get people to recognize the benefits realized by students
to just incremental changes, much less "systemic reform."

Reform efforts can suffer because of the need, even on a single campus,
to mesh iterative efforts to improve courses. The constant "tweaking" of
one first-year course was cited as an example of why it was difficult to
move on to the next level. What was conceived as a sequential
developmental process was actually a parallel one. The team that kept
trying to improve the freshman course frustrated faculty and led to an
unwillingness to adopt the efforts of the team working on the sophomore
course. Continuous improvement can hamper efforts to overcome academic
inertia. There is also simple parochialism---the "Not Invented Here"
factor. It is hard enough to gain buy-in in one's own institution, much
less to get an outside institution to enter into the process of adopting
things developed elsewhere.

Cost inhibits dissemination. Coalition courses and other materials
require additional faculty time, new classroom configurations,
computers, and other equipment, often placing additional burdens on
students, as well. Greenfield indicated that some of its products
(sequences of linked instructional modules) require nearly \$1 million
per year to maintain, given the need to coordinate, professionally
present, and update units.

## C. Challenges

The challenge most commonly identified was that significant, widespread
reform of engineering education is harder than originally thought,
especially in the time-frame provided by the Coalitions Program. It was
difficult to get the Coalitions up and running, particularly the early
ones, hard to get buy-in, and hard to scale things up and
institutionalize them. This is closely related to the theme concerning
dissemination as a process that involved incremental change, not mass
adoption of new approaches.

Coalition courses are typically expensive to maintain. This makes them
hard to transfer, as noted under dissemination, and they may prove hard
to sustain after NSF Coalition funding ceases. Administrators are not
eager to pick up these expenses. None of the PIs interviewed mentioned
the new Action Agenda initiative as a potential vehicle for this,
possibly because their continued funding by the Coalitions Program meant
that they had not turned their attention in that direction. Beyond cost,
there was a broader unease about the prospect of a breakdown of the
Coalition culture and a falling away of people with the loss of focus
that the Program gave.

PI telephone interviews reinforced frequent complaints heard in field
visits concerning the problems of administering funds. Projects were
proposed to a Coalition and accepted. They often then had to be started
up "on spec" since the funds were slow to arrive -- and sometimes did
not show up until the course or project was over.

There was broad agreement that the Coalitions were slow to move on the
assessment requirement. Many got off to a bad start and lacked the
internal resources of ECSEL's Penn State Center for the Study of Higher
Education, or the participation of the Columbia Teachers College
analysts. The prospect of ABET 2000 helped as both a spur and a
challenge, with frequent comments concerning how helpful Coalition
participation had been in preparing for that assessment process.

Most Coalitions encountered difficulties in persuading an "old guard" of
the necessity of change in the undergraduate curriculum, and most had
activities concentrated in a very limited number of departments: the
emphasis on design usually meant that the Department of Mechanical
Engineering was the key participant. This led to another cost barrier
when trying to scale-up or institutionalize new courses, and to move up
to more advanced courses, which often involved other departments.

The phone interviewees also noted the difficulty in dealing with the
Program's goal of increasing the number of degrees awarded to women and
underrepresented minorities. For some it was a matter of student
constituencies or state populations that made this difficult. For
others, the problem was one of developing the skills needed for outreach
and retention. For HBCUs, already focused on a minority constituency, it
was irrelevant. Complaints were frequent concerning the NSF pressure at
a time when affirmative action programs are losing public support, and
some state laws putting some schools in a legal bind. The remarkable
number of undergraduate women at MIT -- about 40% -- was primarily
attributed to an initiative of the Admissions Office that predated the
Coalition. Outreach to K-12 was similarly difficult in terms of skills
and other pressures on faculty, although some schools viewed Coalition
summer programs as being their best recruitment devices. Some local K-12
involvement was feasible, but broad efforts were precluded by other
pressures, especially in research universities.

## D. Broad Issues and Special Concerns

Both phone interviews and field visits pointed to the variety of
converging influences that made it difficult to isolate the influence of
the Coalitions. ABET was cited as an especially strong influence, but
there had been a general sense of the need for reform at the time the
Coalitions Program emerged and that sense was embodied in other programs
funded by a variety of organizations. However, given the complex
problems facing undergraduate engineering education when the Program
began and the changes that have occurred since then, the Coalitions
Program was widely viewed by the campus PIs as the right thing to do at
the time, providing the critical incremental element to get things
moving and legitimizing change and pedagogical scholarship. There is, in
fact, widespread concern about how the momentum initiated by the Program
will be sustained when Coalition funding is phased out

The telephone interviews were consistent with survey and field research
information concerning issues involving younger, untenured faculty.
Experiences ranged from an institution's deliberately discouraging
participation by untenured faculty to being able to cite cases in which
individuals were specifically helped in obtaining tenure through their
Coalition work. Initially, one Coalition as a whole was reported to have
deliberately limited participation to senior faculty, largely with the
goal of establishing an early track record. In another, initial
discussions about joining the Coalition left only two faculty members,
both junior, interested enough to move the school into the Coalition,
and they were discouraged from participating. One prospered as a result,
the other at minimum did not benefit. Pressures are particularly strong
at schools with ambitious administrations that place emphasis on
advancing the school's research reputation. At one such institution,
however, Coalition membership was cited as an important influence in
holding back an administration-sponsored rush from the classroom to the
laboratory.

The need for enthusiastic, energetic, and charismatic leadership was
constantly mentioned as critical to getting a Coalition started and
maintaining its momentum. However, such leadership is not sufficient
without support, both in terms of policy, encouragement, and
supplementary funds from both engineering school and university
administrators.

# IV. Findings from Student Focus Groups

The response of students to their experiences in courses influenced by
Coalition activities was investigated by conducting student focus groups
at each field visit campus. These meetings were not "focus groups" in
the formal sense of the term because the student volunteers could not
always be assembled in a manner that permitted a consistent duration of
structured topical discussion. A number of consistent themes emerged,
however, among the student groups interviewed at the eleven campuses
visited .

## A. Coalition Courses

The most common observations were that Coalition-related courses were
considered to require more work, could be hard on Grade Point Averages,
and could put a separate Coalition cohort at a perceived disadvantage in
comparison with their classmates. Such observations about workloads
require some type of baseline and were thus limited to campuses where
the local operational approach made courses or cohorts identifiable. At
the same time, it was common for students who made such observations to
insist also that, hard as the effort had been, they recognized what it
had done for them and that it often provided them with educational
advantages over their classmates that outweighed the costs in terms of
time, effort, and GPAs.

Students became aware that Coalitions had learning curves. The
inferiority of early efforts to later developments was noted. Those
making such remarks had often suffered the initial experimentation,
moved on through the curriculum, and were observing the improvements
with a degree of envy. Nor were students shy about noting that some
faculty were more adept than others in their efforts to deliver improved
courses -- something especially evident when more than one section of a
course was being taught. The student network was well informed and
highly judgmental.

The students found that variations among the participating faculty were
important. Their reactions to courses varied, depending, for example, on
whether a student had taken the course in a section with professor A or
professor B. Usually, this reflected differences in personality or the
degree to which the faculty member's commitment and/or teaching style
were consistent with the Coalition goals. It was not a matter of
individual professors being assigned to teach courses not having bought
in on the Coalition values: even committed volunteers could vary greatly
in their ability to translate their interest into classroom enthusiasm
and results. Particularly at research-intensive universities, there was
a conflict between the faculty member's interest in curricular change
and a university culture that was at odds with the reality that
developing and using Coalition materials and teaching Coalition courses
required extra time and effort in the classroom, especially in
computer-based courses.

## B. Computer-Based Instruction

The use of computers in instruction in general, as well as reactions to
specific commercial programs, was frequently the subject of criticism by
students, but Coalition-designed modules were seldom a negative topic.
Perhaps one of most interesting themes was the students' concerns about
the effort that it took "learning to learn programs." This took various
forms. At minimum, they felt that the faculty were inattentive to the
time and effort needed to learn a new application. In other cases, the
problem was that time was taken to "learn" an application like Excel
early in a course or course sequence, but that the content of the
training was trivial or irrelevant to its potential uses. When they
really needed to use the program, they did not have an adequate
background to tackle the problems posed.

Specific programs were liked, disliked, critiqued, or thought to be out
of synch with industry. The math instruction program, MAPLE, seemed to
be very commonly used and much disliked. Some CAD/CAM programs common in
the universities were viewed with contempt by students who had co-op,
summer job, or intern experiences: they often felt that what was being
used in their courses did not represent the direction in which industry
was moving. Students would point to changes in versions of software that
the university seemed unable or unwilling to upgrade. Coalition money
often represented the means for substantial investment in computer
equipment and software, but the pressures of obsolescence are constant.
The students showed great acuity in their software reviews and expressed
frequent frustration with the inability of new curricula to keep up with
changes or new developments in commercial programs.

## C. Student Recruitment And Retention In Coalition Programs

Students were recruited into Coalition programs, courses, and other
activities in a variety of ways. For example, Drexel had spent some
years developing its E^4^ lower division curriculum, which was to serve
as the main basis for Gateway's efforts to reform and institutionalize
new curricula, first at the lower division level, then the upper. Since
matriculating Drexel students were automatically enrolled in the E^4^
courses, the school had a head start on other Gateway institutions that
were often less willing or able to move quickly to put the curriculum in
place.

Foundation and SUCCEED worked on the basis of initially taking a limited
number of students into their changing lower division courses, so that
there were parallel cohorts within a given class, some identifiably in
the Coalition curriculum, others in the traditional one. Faculty
interviews at field visits usually described screened invitational
processes, where at least some entering freshmen were invited to
presentations and asked if they would be interested in the alternative
curriculum. Some students considered there to be a coercive element in
the process that almost obligated them to go the Coalition route.
Several subsequently found greater satisfaction in more traditional
courses. One orientation program for entering freshmen produced oddly
symmetric results: one student entered at his parents urging on the
grounds that the hands-on, team-oriented approach described would be
good for him, while another opted for the program over his parents
objections to its experimental nature.

The Coalitions have varied in the degree to which they pursued and
achieved a specified "Coalition" curriculum, as well as the levels to
which reforms had penetrated (lower division more likely than upper,
senior project or design courses more likely than other upper division
courses). Some students were recognizable on campus as "Coalition"
students and able to compare their experiences (generally favorably)
with classmates who were not. In other cases specific courses were
recognizably Coalition-derived by the degree to which they had been
changed to include Coalition elements like design, teaming, and hands-on
experiences. Synthesis was almost invisible at Berkeley: the amount of
influence it had on "Synthesized" courses depended upon which faculty
member taught the course and how much Coalition developed material was
included. At California Polytechnic, San Luis Obispo, on the other hand,
the Coalition was generally a visible entity, but its objectives were so
compatible with the school's tradition and culture as an undergraduate
teaching environment that Coalition project implementation blended into
the scenery.

An important issue related to student retention and to
institutionalization emerged at several schools -- the need for
documentation, or a "road map" of related programs. Few schools have
institutionalized Coalition-based reform beyond a number of common lower
division courses. This represents a particular difficulty where
articulation problems arise when an engineering school receives a
substantial influx of its enrollment from community or junior colleges.
It seemed clear that if the Program's goal was "systemic reform,"
students who liked the reforms wanted a map of the "system" (i.e.,
curriculum sequence) -- just as those who preferred more traditional
approaches felt they needed to know how to exercise their preferences
and still meet degree requirements.

## E. Design Projects

The themes of design and "engineering up front" were of great appeal to
most of the students. Many of the group participants were quite
articulate about the role that these had played in their recruitment and
retention in engineering. For example, one drop-out who had returned to
engineering after several years of military service, thought he would
have been less likely to have dropped out had he had more of the
Coalition-type courses to begin with that he experienced on his return.

Some cynicism was evident about the apparently trivial character of some
of the freshman design experiments used to implement these concepts. One
group pointed to the "game-like" character of design projects involving
bagel and hot dog "launchers," involving materials unlikely to be
encountered in the real world. Yet the same group expressed enthusiasm
about this freshman design course, regarded as the only real engineering
that they would see for some time, and found it a useful entry on their
resumes when seeking summer employment. Many saw the need for "real
world" and "hands-on" opportunities early in the curriculum in order to
hold their interest through the numerous tedious, more theoretical
courses required later in the curriculum.

## F. Teaming Efforts

Teaming was widely recognized by the students as an essential part of
their education and something that was a real asset in their encounters
with industry. It was also one of the more difficult aspects of
Coalition goals to implement, and it was implemented in a variety of
ways that often evoked spirited discussion of common complaints. These
were often offset by one or more of the group pointing out that the real
world was a place in which work-mates were not always congenial team
partners.

Coalition faculty put teams together in a variety of ways. Some used
personality tests, GPA distributions, or, especially on campuses with a
large heterogeneous commuter population, age groups. Some picked teams
randomly, while others tried to ensure a mix of skills. Gender often
played at least an initial role. Many faculty were leery of having but
one woman on an otherwise male team and assigned two per team as far as
possible, often abandoning this concern once students had had some
teaming experience. Within limits, self selection was sometimes
permitted. The students did not seem to have any sense of which method
worked best, although they liked least those in which they viewed
themselves as having been manipulated.

Given the problems that most had encountered in their teaming efforts,
students appreciated efforts to prepare them for it. Within one
Coalition, a field visit observed a class being braced for the social
nuances of team interaction in one of a series of class periods devoted
to this. The student group at the other campus in the same Coalition
complained about the lack of *any* preparation. There were frequent
complaints of the difficulty of gaining faculty intervention when
serious problems within a team surfaced.

The teaming problems most subject to complaint were consistent.
Establishing leadership and how to deal with slackers dominated student
comments. Teams appear to have usually been left on their own to deal
with their internal organization, division of labor, and scheduling
within the project deadline. In that sense, the experience lacked the
real world hierarchical assignments that would usually be made in
industry.

The problem of slackers was a major irritant, and most of the groups
cited examples of team members who essentially disappeared as the
project was supposed to be moving forward. Special problems could arise
in the context of commuter campuses or community colleges. Students
often had job and/or family obligations that made scheduling team
meetings difficult. Heightening the tension was the problem that most
faculty appear to have adopted a policy of collective grading: i.e., the
project grade was given collectively to all team members. Greater
satisfaction with the grading process was evident when some sort of
internal team evaluation could influence the grade distribution in a way
that reflected perceptions of relative contributions.

From the student groups, one got an impression that teaming was readily
accepted into Coalition courses, but that faculty were loath to deal
with most of the problems that arise in the educational environment.
Most took a laissez-faire attitude once the rules on grading were
established and their personal method of establishing the teams set.
Faculty input during field visits seemed quite complacent in terms of
awareness of problems compared to student reactions to the ways in which
teaming was handled.

Despite the problems involved in teaming, student responses were
overwhelmingly favorable. The overall thrust of the groups was one of
great consciousness of the importance of the experience for their
long-term career interests -- their "preparation for the real world"
being a phrase often used. Also, a number of the student groups noted
that the team approach often spread among their fellow students. The
value of team efforts was becoming apparent in the sense that not only
students involved in engineering courses requiring teaming were picking
up on the cooperative approach. Coalition engineering students continued
to engage in informal teams for a variety of courses and problems, and
the practice spread to non-engineering students in some cases.

## F. Industry Response

The student groups were positive about the ways in which their
institution's involvement with industry had increased, as well as the
response of industry to the impact of new Coalition curricular elements
on their skills. The groups were enthusiastic endorsers of the impacts
that their Coalition work had had on their efforts to find jobs and
internships. In job interviews, industrial recruiters were almost always
impressed by students displaying a variety or some combination of
factors reflecting the Program's goals -- working in teams,
communications skills, and the like.

Communication skills were an important factor: students reported that
their experience in making presentations in Coalition courses was
recognized by industry in their job-seeking activities. The down side in
terms of curriculum development was a clear discontent with the ways in
which Coalitions attempted to provide those skills by linking technical
courses to review by humanities instructors, linking grades to judgments
of faculty members trained in writing and other communications skills,
but lacking understanding of the technical content of the material
involved. The implementation of training in communications was not
always sensitive to the tension between the standards normally applied
by humanities teachers and the technical content involved, especially in
grading.

## G. Students' Overall Coalitions Experience

An effort was usually made to ask the students to assign a quantitative
ranking to their "Coalition experience." That phrase was used because of
the variety of experiences in Coalition's context that students had. The
comings and goings of students during focus group sessions sometimes
made this difficult, and the results, should be taken as a non-rigorous
set of ratings. Despite the candid critiques voiced in the student
groups, when asked to rank their experience on a scale of one to five
(five being the top), they gave the Coalitions high marks. From the top
down, one received a rating of 5, two 4.4s, one 4, and two 3.9s.

# V. FACULTY SURVEY

In order to broaden the study's perspective on the activities of the
many faculty who had participated in the variety of Coalition activities
that had taken place at member institutions, a sample survey was
conducted of faculty participants across the six Coalitions. The random
sample of 302, drawn from lists of participating faculty provided to SRI
by the Coalitions (a total of 1151 faculty names, the sample
representing 26%), was given a choice of responding via the web or
printing out the survey to preview it and/or submit it in hard copy, or,
through an SRI contact number, to receive a hard copy by mail. The
overall response rate was slightly over 50% when the survey was closed.
From the 152 responses received to the survey, 18 (12%) indicated that
they had never participated in Coalition activities. Eight of those (5%
of those responding) were not even aware of the Coalition Program's
existence. A sample of non-respondents was contacted, and this revealed
no significant differences in their replies from the returned responses,
so no weighting of responses was required. The survey covered five
general areas about each respondent:

-   Personal demographics;

-   Involvement in the Coalition;

-   Involvement in activities involving course design and educational
    product development;

-   Involvement in instructional implementation of Coalition materials;

-   Assessment of the impact of the Coalition on teaching, students, and
    institutional culture.

## A. Coalition Inputs

Of the 132 valid responses from individuals involved in the Coalition
program at their respective institution, 117 (89%) held professorial
rank at some level, while the remainder held administrative or other
non-teaching positions. With more than 75% of the respondents holding
normally tenured positions, the data suggest that tenure-track junior
faculty do not appear to be significant participants in Coalition
activities.

In terms of time involved in the Coalition, 17% had spent one year or
less on Coalition related activities, 45% had spent between two and four
years, and 37% had spent five or more years. Over half of all
respondents reported that they remained active in the Coalition. Based
on duration in the program, the data suggest that Coalition participants
have exhibited no small degree of commitment to Coalition activities.

Respondents were asked to describe the types of activities in which they
participated as part of their involvement in the Coalition. The vast
majority (79%) described participating in the design or development of
courses or educational-related materials. There was also a great amount
of participation in activities concerning information-exchange with
fellow faculty members; roughly three-fourths had attended meetings
involving other schools from their Coalition, two thirds had
collaborated with faculty from both their own Coalition and other
Coalitions, and half had presented papers on Coalition-related topics at
professional meetings (see Table V-1). Only about a fourth of the
respondents had participated in student-related activities, such as
recruitment and retention. About 60% of faculty were teaching courses
that were either new to the curriculum or had been modified as a result
of the Coalitions program. Less than 5% had been involved in
assessment-related activities.

Respondents were asked to compare their involvement in activities on the
basis of participating either with other faculty from their own
Coalition or with faculty from different Coalitions. Almost two-thirds
(64%) of faculty members reported redesigning or modifying courses or
educational materials with faculty from within their Coalition, while
only about a fourth (24%) had done so with individuals from other
Coalitions. There was a somewhat smaller difference relative to faculty
who reported teaching or learning new instructional techniques: 58%
participated in this activity within their Coalition, while about a
third (30%) did so with faculty from other Coalitions. Aside from more
informal exchanges, most collaboration took place within each Coalition.

![](media/image2.wmf){width="6.253472222222222in"
height="3.816666666666667in"}

Industry involvement in Coalition curricula most often took the form of
a visiting lecturer in courses or labs. Among those teaching courses,
42% of the faculty reported having a guest speaker from industry, but
only 13% had included industry guests in Coalition-related labs. Other
participation by industry representatives included reviews of student
work or presentations (31% courses, 13% labs), generating or
participating in design projects (28%, 11%), and providing internships
or co-op experiences (12%, 7%).

## B. Curriculum And Instructional Delivery

An important objective of the Coalitions Program was to affect the
engineering curriculum at member institutions. Faculty were asked a
series of questions to assess the impact that their Coalition has had on
developing, designing and revising courses, courseware and other
educational materials at their institutions. A total of 79% of
respondents had been involved in these activities through their
Coalition. Half of these individuals helped modify or develop labs,
while almost all had done so for courses (many participated in both).
Collectively, these individuals were involved in reshaping 234 courses
and 99 labs in total. Combined, surveyed faculty taught 146
single-discipline courses and labs, 94 involving multidisciplinary
engineering classes, and 93 cross-disciplinary courses and labs
involving both engineering and non-engineering areas in conjunction with
their Coalition participation.

Project teams were the most common component found in the courses and
labs that respondents reported designing or developing (see Table V-2).
More than two-thirds (70%) of those involved in course design and
development reported that project teams had been part of courses, while
about half (53%) said they were elements in labs.

![](media/image3.wmf){width="6.49375in" height="3.7215277777777778in"}

Faculty reported that the people they dealt with most often in their
work on Coalition courses and labs were faculty from their own
institution (80% in courses and 68% in labs). Significantly less
collaboration occurred with faculty from other institutions within the
respondents' Coalition (47% and 37%), as well as with faculty from other
Coalitions (15% and 18%). Faculty reporting interaction with industry
personnel was relatively low at 34% for courses and 23% for labs. Less
than half had interacted with evaluation and assessment experts.

Respondents were asked to identify the different types of products they
helped develop for these courses and labs (see Table V-3). Two-thirds
said they helped develop instructional modules for courses, while nearly
half did so for labs. About half indicated they participated in
developing a curriculum for a whole course.

![](media/image4.wmf){width="6.120138888888889in"
height="2.2597222222222224in"}

Curricular materials produced by faculty in the survey employed an array
of instructional media. While nearly two-thirds had worked on products
that took hard copy form (e.g., a textbook or a paper module),
substantial numbers had been involved in developing computer-related
materials. Nearly half had worked on developing Web-based materials, and
roughly a fourth had incorporated e-mail, computer diskettes, and C-ROMs
in instructional materials they had produced.

Components of courses and labs taught by Coalition faculty followed the
same pattern as components of curricula designed or modified by faculty.
Open-ended problem solving was cited most often, with 75% of faculty who
taught Coalition courses reporting its use in their course, and 50% of
faculty who taught Coalition labs having used it in their labs. Project
teams (72% and 57%, respectively) and design projects (70%, 46%) were
the other most frequently reported components of Coalition courses and
labs the faculty were involved in teaching. Less commonly-cited elements
of Coalition courses included discovery-based learning (34%, 30%), using
a business context in problem-solving (34%, 15%), and student
recruitment and retention efforts (20%, 4%).

Hardcopy (e.g. textbooks or paper modules) remains the dominant type of
media used in Coalition labs and courses, with 77% of faculty reporting
their use in Coalition courses, and 63% using them in their labs. E-mail
is also used by more than half of the faculty in courses (60%), as are
World Wide Web sites (52%). Multimedia modules, computer diskettes, and
video are all used by over 20% of the faculty teaching Coalition
courses.

Dissemination of materials across Coalition institutions was relatively
uncommon. Only 17% of faculty utilized World Wide Web sites developed at
another Coalition school in the courses they taught, with 15% reporting
their use in labs. Course curricula developed at another Coalition
institution were used by 16% of the faculty in their courses and by 9%
in their labs.

Respondents were asked to elaborate on their use of the World Wide Web
in the courses and labs they taught as part of their involvement in the
Coalition. The most frequent response was the use by faculty to gather
information on courses or labs, with 53% and 28% reporting this
respectively. Faculty also made relatively frequent use of the Web in
obtaining materials for student use, with 48% reporting this use in
courses and 24% in labs. Student use of the Web for courseware occurred
slightly less often at 40% and 24%. Less frequent usage was found in
interdisciplinary exposure (23%, 15%) and assessment-type activities
(15%, 9%).

## C. Student Outcomes

The most systematic information on student outcomes produced by the
study derives from the survey of participating faculty. About half of
the faculty (49%) had taught more than 100 students in their Coalition
courses and labs, while 13% taught under 30 students.

Faculty were asked to compare the students in their Coalition classes or
labs at the beginning term with their performance after a month to six
weeks of exposure to the Coalition materials. Generally, once given time
to get used to the media, the content, and the learning environment, the
students appeared to gain a firm grasp of the courses.

Clearly, the students had a fair amount of acclimating to do: a minority
of faculty (40%) reported that, at the beginning of the course, most or
all of the students knew what was expected of them. Four to six weeks
into the course, nearly three quarters of the faculty reported that most
students understood the course expectations. The number of faculty who
said that most or all of their students understood the material easily
doubled after four to six weeks in the course, going from 15% to 30%.
The Coalition curriculum also had some impact on students' comfort with
team work: faculty who reported that most or all of their students were
comfortable working in groups rose from 29% at the beginning of a course
to 51% four to six weeks later.

Coalition students also compared very favorably to their non-Coalition
counterparts in terms of achievement. Among faculty reporting (Table
V-4), 82% said that the Coalition students performed better in
interdisciplinary projects. Coalition students were also rated higher at
working in groups (79%) and understanding and pursuing hands-on learning
(61%). Overall the faculty rated the Coalition students higher than the
non-Coalition students on all seven criteria listed.

![](media/image5.wmf){width="6.49375in" height="3.81875in"}

The student workload for Coalition courses and labs was considered by
faculty to be significantly higher compared with non-Coalition classes.
The majority (69%) thought that students had either much more work or
somewhat more work in the new classes. However, faculty generally
believed that Coalition courses had had a positive impact on students.
More than three-fourths (79%) stated that Coalition courses had had a
positive impact on students working in teams. Roughly two-thirds
reported a positive impact of these courses on students' preparation for
employment (69%), communication skills (68%), and breadth of
understanding of subject matter (63%). Half or more also thought that
the Coalition courses had had a positive impact on students' depth of
understanding of subject matter (58%) and retention in engineering
majors (50%). A considerably lower percentage (38%) thought that
Coalition courses had had a positive impact on students' academic
performance. Very few identified negative impacts.

Faculty were asked to discuss the aspects of Coalition courses or labs
that students enjoyed the most as compared with more traditional
non-Coalition classes. A few themes dominated their responses. Students
really enjoyed the opportunity to work in teams. They learned how to
solve problems collaboratively, something they didn't get to do as often
in non-Coalition classes. It also allowed them to interact with people
they normally wouldn't, including students and professors from not only
outside of their class but from other disciplines as well. Students also
preferred the hands-on/active learning structure in Coalition courses.
Passive listening to lectures gave way to active involvement in
projects, which clarified some difficult ideas. Included in this
hands-on approach is the chance to incorporate cutting-edge technology
into the coursework. Finally, students appreciated, in the view of
faculty, the real-world focus provided by projects that had practical
applications, as opposed to involving more abstract concepts, especially
while still in their first year.

When asked what students liked the least about Coalition courses or
labs, the over-whelming response from the faculty was the amount of
work. Clearly the courses and labs were much more demanding of the
students' time than traditional classes. Another major complaint of the
students was the same as what they identified as liking most about
Coalition classes: working in groups. Students got frustrated when
conflicts inevitably arose over which direction the group should take
towards completing a project. According to faculty, another student
complaint about working in teams was that some members didn't do their
share of the work, leaving it up to others instead, and working with
students from other disciplines could also initiate problems.
Ironically, aside from the workload, faculty reported that what students
enjoyed most about Coalition courses and labs were the same things with
which others had difficulties.

## D. Outcomes: Faculty and Academic Culture 

The survey of participating faculty asked respondents whether
participation in the Coalitions had affected their involvement with
undergraduate students in terms of teaching, advising/mentoring,
research, and outreach/recruiting. In each case, more than half of the
faculty reported no change from prior to their involvement with the
Coalition. The area in which increased involvement with undergraduates
was reported by the highest number of respondents was teaching (43%),
followed by advising/mentoring (35%), research (28%), and
outreach/recruiting (19%). One out of ten (11%) faculty members reported
*decreased* involvement with undergraduates in their research
activities.

Faculty were asked in an open-ended question to describe other types of
cultural changes they identified as having taken place in their
institutions or departments during the past five years. Increased
faculty interest in teaching was the most frequently listed change (17
respondents), followed by increased industry interaction (14),
interdisciplinary collaboration (12), student mentoring (10), and
student teamwork (7).

Three-fourths of the participating faculty (75%) reported that the
Coalition had led to improved teaching. A small majority of faculty
(53%), thought that the value of teaching compared with research in
their departments had remained relatively the same since prior to the
Coalition. However, over a third (35%), thought that the value of
teaching had increased somewhat relative to research, and an additional
one out of ten (11%) thought that it had increased a lot. Most
participating faculty (83%) reported that research has always been and
remains more highly valued than teaching in promotion and tenure
decisions. A handful (6%) reported, however, that Coalition activity had
led to teaching being valued as highly as research.

Faculty were also asked whether the weighting of their participation in
education reform activities had changed in promotion and tenure
decisions since before the Coalition. While most reported that
innovative teaching and curricular reform activities are currently
valued about the same as before the Coalition, about a third thought
that innovative teaching is currently valued more than prior to the
Coalition in promotion decisions (36%) and in tenure decisions (28%).
Roughly one out of four or five thought that curricular reform
activities are currently valued more than prior to the Coalition in
promotion (27%) and tenure decisions (18%).

# VI. Survey of Deans of Schools of Engineering

The Coalitions have collectively produced many examples of outcomes
reflecting Coalitions program goals. The field visit evidence suggests
that the impact of these outcomes falls far short of the desired
systemic reform. Of the various types of additional information that
were collected, the most systematic comes from a survey of engineering
school deans, focused on the types of reforms that have been instituted
at their institutions and their awareness of the Coalitions' efforts and
the use that has been made of Coalitions' materials at their schools.

The survey was administered to deans of the schools or colleges of
engineering at all U.S. colleges and universities that have
undergraduate engineering programs except those that are part of an
Engineering Education Coalition. Of the 193 Deans to whom questionnaires
were distributed by a combination of e-mail and web-based
administration, 126 valid responses were received, for a response rate
of 65%. The questionnaires were addressed by name to the deans of the
schools or colleges of engineering, noting that the dean might wish to
direct the questionnaire to another individual in the university
administration with special knowledge of engineering curriculum issues,
such as the academic affairs officer. Two-thirds of the responses (65%)
were completed by the deans of engineering themselves, and another
fourth (25%) were completed by the associate/assistant deans for
academic affairs. The remaining ten percent were completed by vice
presidents/vice provosts for academic affairs, associate/assistant Deans
for engineering, Deans of the undergraduate college or academic
programs, and in one case by the provost.

Respondents were asked to identify the types of undergraduate education
reforms that had been initiated in their school/college of engineering
during the last five years. About a third (36%) of these schools of
engineering had initiated the development and management of *large
scale* curricular changes. In terms of less sweeping changes, virtually
all of the schools of engineering (98%) had undertaken course design,
development, or revision, with two-thirds or more having initiated most
of the range of other activities is shown in Table VI-1.

![](media/image6.wmf)

Of the 25 that had introduced changes aimed at the adoption of an
engineered systems approach, more than half were institutions in which
an NSF Engineering Research Center has been established. Most of the
above reforms had been initiated at the level of the school/college of
engineering (89%) or department (87%), but a substantial number (44%)
had been university-wide.

One of the primary factors that has motivated reforms at engineering
schools over the last five years has been ABET 2000; roughly nine out of
ten (91%) survey respondents indicated that this had been a factor
motivating change. Faculty interest has also been a strong factor
motivating reform, indicated by about 8 out of 10 (81%) survey
respondents. Other factors are listed in Table VI-2.

![](media/image7.wmf){width="6.240277777777778in"
height="2.6631944444444446in"}

With wide scale reform of engineering education the basic goal of the
Coalitions Program, institutional commitments to teaching in promotion
and tenure decisions are an important measure of the extent to which
such reforms are being established. Innovative teaching is given greater
weight in promotion and tenure decisions than five years previously at
roughly forty percent of colleges and schools of engineering. While in
many cases it is not receiving any *greater* weight than previously,
almost no cases were reported in which it is being weighted *less* than
five years ago. Roughly twenty-five to thirty percent of the schools
also reported that curricular reform activities and efforts are being
weighted more heavily in promotion and tenure decisions than five years
ago. While again, the vast majority reported no differences from five
years previously, almost none reported that such activities are weighted
*less* than five years earlier.

Evidence of important change might include interdisciplinary interaction
arising in schools outside the Coalitions Program. Respondents to the
Deans survey were asked what proportion of their faculty engage
significantly in interactions beyond their home departments. Faculty
interaction with other departments is not uncommon, with 61% of the
schools of engineering reporting that about a third of their faculty
interact across departmental lines and another 27% reporting that more
than half of their faculty engage in cross-departmental activities.
Interaction with non-engineering colleges is much less common, with 50%
of respondents reporting that less than one out of ten of their faculty
engage in cross-college activities, another 43% reporting that only
about a third of their faculty are significantly engaged in these
activities, and only 7% reporting that more than half of their faculty
interact with other colleges.

Faculty interaction with industry, another part of the Coalitions
Program agenda, is more common in colleges of engineering than is
interaction across departmental lines. Close to four out of ten of the
schools surveyed (38%) reported that more than half of their faculty
have significant interactions with industry and another five out of ten
(51%) reported that about a third of their faculty are involved in such
interactions.

About half of the colleges of engineering that responded to the survey
(48%) reported that their college had collaborated with other
universities or colleges in the development of courses, material, or
student assessments during the past five years. More than a third (38%)
of these other universities and colleges with which education-related
collaboration had taken place were said to have been members of an NSF
Engineering Education Coalition.

By far the most commonly reported sources for learning of new
developments in curricular materials were ASEE publications,
conferences, and meetings, listed by over 50 respondents in response to
an open-ended question. Other frequently cited sources of information
included journals, articles, and other publications (19); the Internet
and the Web (17); national conferences (17); professional meetings (16);
ABET meetings and publications (16); NSF Engineering Education
Coalitions materials and publications (13); NSF reports and publications
(8); and publishers and vendors (7). The field research noted that
Coalitions reported a large amount of ASEE media as means of
dissemination. With ASEE sources listed as the top source of learning of
new developments, it is hard to assess the reason that Coalition
materials, themselves, ranked so low in respondents' awareness of
Coalition efforts and materials, but it is possible that the respondents
read about Coalition-related efforts in ASEE publications without being
aware of the relationship to the Program.

This is especially true because roughly eight out of ten of the Deans'
survey respondents reported that they were familiar with NSF's
Engineering Education Coalitions Program. SUCCEED was known of by 56% of
respondents, ECSEL by 47%, Gateway by 44%, Synthesis by 37%, Foundation
by 36%, and Greenfield by 21%. When asked whether they were familiar
with any products and/or materials (e.g., textbooks, CD-ROMs, World-Wide
Web sites) developed by an Engineering Education Coalition, however,
less than half (40%) answered in the affirmative. Specific products with
which survey respondents were familiar are listed in Table VI-3.

![](media/image8.wmf){width="6.041666666666667in"
height="6.499305555555556in"}

About one-third of respondents were familiar with specific Coalitions
products. As shown in Table VI-4, 46% of these respondents considered
the Coalitions products with which they were familiar very accessible,
40% considered them very user friendly, and 34% considered them very
useful.

![](media/image9.wmf)

Respondents were also asked whether, based on their understanding of
those in their school or college of engineering who had reviewed
Coalitions products or materials, any shortcomings in these materials
had been found. Highly pertinent to understanding some of the
difficulties involved in disseminating Coalition products is that while
roughly a third of the respondents indicated that no serious
shortcomings in Coalitions products had been found, about a fourth
indicated that the materials require too much faculty time to use
properly and a fourth also indicated that the materials are not ready
for transfer beyond the originating institution. Several comments about
transferability implied that the materials had limited utility outside
the Coalition group of institutions because they were too
campus-specific, too internally focused, or limited by differences in
size and culture of the institutions involved.

Of the 50 respondents who indicated that they were familiar with
Coalition materials and products, only 22 indicated that any of these
materials or products had been used in their school or college of
engineering. This is about 18% of all institutions that responded to the
survey. Eleven institutions are using methods for incorporating new
technology in courses or labs (e.g., video or distance learning), 9
institutions are using instructional modules for courses or labs, and 6
are using other methodological materials. Only a few institutions are
using a curriculum for an entire degree, a complete curriculum for an
individual course or lab, or a curriculum for a set or sequence of
courses or labs. The most common media through which these Coalition
materials and products have been obtained are Web sites, hardcopy
textbooks or paper modules, and CD-ROMs. Eleven institutions reported
that Coalition materials have been enthusiastically embraced by a small
number of faculty, 6 reported that the materials are considered to be no
different from those already available, and 10 reported that faculty are
not sufficiently aware of the materials to have an opinion.

Only fifteen respondents reported awareness that individuals from their
institution had looked at the National Engineering Education Delivery
System (NEEDS). Most of those who had not looked at the system reported
that they were not aware of it. Of those who had examined the system,
most found it somewhat useful but none found it very useful, a roughly
equal number found it somewhat or very accessible, and a roughly equal
number found it somewhat or very user friendly. Eleven respondents
reported knowledge of someone in their institution having downloaded
courseware or other materials from NEEDS, 5 reported someone having
tested such materials in a course or lab, and 4 reported someone having
evaluated such materials in a formal sense (e.g., individual or
committee review). In fact, relatively few of the respondents in the
Faculty Participant Survey (9%) had used NEEDS postings to disseminate
products, as opposed, for example to conference presentations (57%).

The Deans survey respondents, especially in questions indicating
detailed awareness and/or attempts to use Coalition outputs, indicated
that the impact of the Coalitions and their outputs has been mixed and
limited. The results reinforce information from field research pointing
to a serious problem with dissemination of Coalition materials to the
wider engineering education community, as distinct from actually
adopting Coalition curricula and materials in non-Coalition
institutions.

# VII. CONCLUSIONS AND RECOMMENDATIONS

## Overall Findings

### Progress towards Reform Thus Far

The Engineering Education Coalitions Program was established to create
broad-based institutional collaborations aimed at establishing "bold,
innovative, comprehensive, and systemic new models for undergraduate
engineering education." *"Comprehensive Curriculum Restructuring"* was
emphasized, along with widespread dissemination and adoption of
Coalition products at both Coalition and non-Coalition schools of
engineering. The objective of this study was to assess the progress that
the Coalitions had made during the first five years of the program
toward these goals at roughly the midpoint of the Program's prospective
life span.

A large number of Coalition-supported "products" have been generated,
both in terms of curricular materials and instructional practice. All
are responsive, typically, to more than one of NSF's desired
characteristics: a teaming approach, open-ended learning, new
instructional technologies, etc. The most pervasive of these was found
to be the presence of "engineering up front": the exposure of freshmen
to hands-on, real-world engineering practice early in their
undergraduate education. Both dissemination and program evaluation
activities were found to have been much slower to develop than the
Program had intended.

Senior design projects with industrial partners were highly valued and
efforts are being made to expand them. Both faculty and students saw
Coalition emphasis on a team approach as a difficult experience, replete
with problems associated with team composition and organization, ways of
dealing with slackers, and grading methods. Students nonetheless rated
it highly as a valuable "real world" experience that gave them an
advantage in the job market. Because too few students had graduated from
degree programs heavily influenced by Coalition efforts when the study
was conducted, no component of following their experience in industry
was included in the study. However, available student feedback indicated
that Coalition experience was entirely advantageous in terms of the
reactions of industry personnel during internships, coop or summer jobs,
and senior job interviews. The teaming experience, hands-on
understanding, and communications skills derived from
Coalition-influenced courses were recognized and valued by industry as
reported by students interviewed at the Coalitions.

Preexisting characteristics of the institutions involved had an
important impact on the degree to which "buy-in" and the consequent
inception, breadth, and institutionalization of Coalition efforts was
received. Some disciplines in a given institution had long since
developed a team culture, others resisted it. Support, both in terms of
leadership and resources from administrators in both the engineering
school and the larger institution, had a significant impact. Some
institutions were active developers of materials, others passive
recipients and/or adaptors. Coalition efforts had to compete with other
priorities over and above research agendas within schools and
departments. Coalition acceptance was aided by and, in turn, aided
schools in dealing with ABET 2000 efforts. In fact, it is difficult to
assess the progress of the Coalitions because there are so many other
convergent forces pressing for the reform of engineering education.

A clear limitation of Coalition initiatives has been their breadth in
relation to the full range of engineering disciplines. Even at a single
institution, much less across institutions, much of the progress thus
far has been within certain disciplines such as mechanical engineering
or at lower undergraduate levels, such as freshman coursework. Most of
the Coalitions have had only a limited impact on the other major
engineering disciplines or at the upper levels of the undergraduate
curriculum. Although many examples of desirable change have been
described in this report, the examples are largely isolated observations
of the desired behavior, covering only a small minority of courses,
faculty, or students. Some possible reasons for the lack of stronger
signs of systemic reform to date follow.

### How Long Should Reform Take?

One obvious response to the limitations on reform observed thus far is
that the Coalitions have been functioning for a relatively short time,
and the focus of the first stages of reform has tended to concentrate to
date on lower division courses. Such a response may be entirely
reasonable, because the time needed for broad-based reform to occur is
not well understood. System-wide reform of undergraduate engineering
education in the United States certainly takes more than five or even
ten years. For example, for some types of systems change, an entire
generation of people (in this case, faculty) may need to turn over
before institutions are fully reformed. From this broader time
perspective, the fact that the Coalitions have been able to make
significant changes in some disciplines, and, at the lower undergraduate
levels in particular, can be taken as a positive sign of progress.

### Conditions Impeding Reform

In terms of developing broad-based reform of the U.S. undergraduate
engineering education system, study findings suggest that several
conditions exist that may be impeding reform. First would be that the
Coalitions Program strategy lacks sufficient breadth. Based on 1996
data, the Coalitions collectively cover only 32% of the "teaching"
faculty in undergraduate engineering in the United States, and only 24%
of the undergraduates enrolled in engineering. Even though the
Coalitions include many of the leading engineering institutions, change
across the entire country will still require substantial dissemination
and diffusion into practice of the Coalitions' successful experiences.

A second potential impediment is that some of the factors underlying the
concerns leading the original Belmont Conference have not changed
greatly. For example, large-scale funding for academic engineering
research, whether from industrial or governmental sources, does not
appear to have diminished substantially. At the same time, while the
funding of the Coalitions collectively seems substantial, the small
amounts allocated to individual projects could make them appear to be
more trouble than they were worth. Faculty who are oriented toward
research can still pursue their interests and either resist change in
undergraduate teaching practices or be distracted from it. In the face
of such conditions, institutions as a whole have less pressure to
change.

### Conditions Accelerating Reform

Despite possible impediments, one force strongly favoring reform is the
gradual implementation of ABET's *Engineering Criteria 2000*. Beginning
in the year 2001, the criteria will apply to all accreditation actions
in the United States. ABET reviews about 25% of the approximately 330
schools in the U.S. with undergraduate engineering programs (about 305
schools currently have accredited programs). Thus, by 2005, nearly all
schools desiring ABET accreditation will have had to comply with the
criteria. As this report has noted, some Coalition schools were helped
in meeting the ABET criteria by their Coalition involvement. In
addition, some Coalitions provided assistance to non-Coalition schools
preparing for their own accreditation reviews. Their example suggests
that adopting some of the practices and innovations already tested by
the Coalitions should be an attractive strategy for other schools in
efforts to meet the new criteria. In this sense, the work of the
Coalitions will then have provided a valuable contribution toward the
overall movement for engineering education reform.

## Building on the Progress of the Coalitions

The study suggests that there has been limited success in achieving the
Coalitions Program's perhaps overly ambitious goal of comprehensive
changes and reform in such a short time-frame of five years. There is
much still to be accomplished along such lines as changing academic
cultures to integrate engineering education and research, enhance
interdisciplinary and team-oriented teaching, and provide greater
institutionalization of reforms that provide early experience with
design and engineering practice.

Nevertheless, much has been accomplished. As the Program moved into its
second and final phase, it had:

-   developed a significant base of enthusiastic human resources,
    dedicated to and experienced in the effort to provide innovative
    courses and instructional materials;

-   produced a wide range of pedagogical outputs that range from
    computer-based instructional modules through complete new courses to
    integrated curricula aimed at meeting NSF's goals as well as ABET
    2000 criteria.

Moreover, while ABET 2000 has been and will continue to be a driving
force that influences schools of engineering to adopt programs and
materials consistent with the goals of the Coalitions Program, the
Program has meanwhile played an important role in preparing schools for
ABET review. It also has legitimized to some degree pedagogical research
as a part of the reward system of a number of universities.

The intent here is to suggest ways in which the Coalitions Program can
most effectively leverage these outputs during its final years of
investment in the reform of engineering education. To do so is a form of
innovation. Exemplary program outputs need to be disseminated, adapted,
and adopted widely for systemic reform to be achieved. What we know
about innovation says that dissemination, at least in the sense of
communication about the availability of something, does not necessarily
result in technology transfer (adaptation and adoption), and this is the
critical strategic issue for the Coalitions Program to confront. How can
it most effectively promote the transfer and adoption of the best work
that it has sponsored since 1990?

Diffusion of educational innovations faces a number of barriers.
Dissemination was shown by the study to be a significant problem for
several reasons related to it being an intermediate step, not an end
point.

1)  Most innovative developments were limited to intra-institutional
    adoption among Coalition members, and even within institutions,
    expanding "the choir" has not been easy.

2)  Documentation is a serious problem. The most enthusiastic faculty
    participant is still not inclined by disposition, available project
    money, and other aspects of the reward structure -- including his or
    her regular research agenda -- to invest substantial time in
    providing the documentation necessary to translate the originating
    classroom experience into something immediately transferable to
    other teachers, curricula, and institutions.

3)  Dissemination methods used thus far have failed to distinguish
    between knowledge transfer and technology transfer. The Program has
    facilitated travel to meetings, and its legitimization of
    pedagogical research has resulted in the publication of numerous
    papers in ASEE and other related education journals. However, just
    as few patents are ever licensed for application, survey evidence
    from the study suggested that few presentations or publications had
    resulted in true transfer -- adaptation and adoption -- of Coalition
    outputs beyond the boundaries of their developing institution or, in
    some cases, the Coalition more broadly.

These problems with dissemination suggest that the Coalitions Program
needs to establish a ***process*** that will bring together the products
that have been produced and, beyond where they are already in use, the
human resources who are committed to needed reforms and seeking
innovations to adopt. The process needs to start with a strategic
assessment of the products for their quality, adaptability, and
marketability in the domain of engineering education. Without
demonstrative empirical proof of the effectiveness of an innovation, it
is unlikely to be accepted in the market place.

This is not enough to bring about adoption, however. Instruction modules
and courses need to documented sufficiently in order to be usable in
different types of institutions. They must also be sufficiently generic
in context to be usable in other settings: if products aren't
immediately usable in other institutions they must be adaptable. Broader
multi-course curricula face more difficult barriers: they need buy-in
from larger and multi-department constituencies. Courses need to be
evaluated for their potential adaptability and for their potential for
institutionalization, not brief experimentation. Finally, marketability
would be enhanced if products were potentially commercializable through
publishers, software vendors, and related sources. One of the most
serious transfer barriers for Coalition courses and products is that
they are typically more costly to implement than traditional teaching
methods, both in time and monetary investment: efficiencies of scale or
other economies are needed.

At least two mechanisms might be considered in order to commence this
process.

1)  Some type of primary screening of both available materials and the
    related human resources -- their developers, or faculty who have
    taught courses developed by the Coalition or using
    Coalition-developed materials. Even the best "products" will have
    difficulty finding acceptance without "champions."

2)  Selectivity Workshops: Priorities need to be established for efforts
    to push the best of the best across and out of the Coalitions. Such
    workshops can be topically focused. They need to include the
    developers and prospective champions of the materials, engineering
    deans or other academic representatives from outside the Coalitions,
    representatives from industry who are not Coalition insiders, and
    "commercializers" (publishers, etc.).

Following the establishment of selective priorities, a concerted effort
at transfer to other institutions can follow to complete the process.
This could include:

1)  The development and marketing of commercial products (textbooks,
    documented modules and computer programs, etc.).

2)  Training workshops -- perhaps summer sessions of one or more weeks
    in which faculty from non-Coalition schools are brought together
    with Coalition champions of the selected innovations to provide an
    overview of their effectiveness and training in their use or
    adaptation.

3)  Exchange programs intended to leverage the human resources developed
    by the Coalitions by bringing outside faculty into a Coalition
    environment containing innovative instructional techniques and
    cultural shifts, as well as putting product champions into other
    schools to teach and show others how to implement Coalition courses
    and materials in non-Coalition contexts.

The latter two are aimed at moving beyond forms of secondary
interaction, such as conference presentations and the reading of
published papers to the most assured approach to innovation: primary
human interaction on an extended basis that is intended to equip others
to use Coalition products successfully.

Finally, if such activities are undertaken, each Coalition should be
given the responsibility of providing NSF with a tracking of its
activities in the process and an assessment of the effectiveness of
those activities in contributing to the broadening of reform toward the
goal of more systematic and pervasive reform of undergraduate
engineering education.

The National Science Foundation (NSF) funds research and education in
most fields of science and engineering. Grantees are wholly responsible
for conducting their project activities and preparing the results for
publication. Thus, the Foundation does not assume responsibility for
such findings or their interpretation.

NSF welcomes proposals from all qualified scientists, engineers and
educators. The Foundation strongly encourages women, minorities, and
persons with disabilities to compete fully in its programs. In
accordance with federal statutes, regulations, and NSF policies, no
person on grounds of race, color, age, sex, national origin, or
disability shall be excluded from participation in, be denied the
benefits of, or be subjected to discrimination under any program or
activity receiving financial assistance from NSF (unless otherwise
specified in the eligibility requirements for a particular program).

Facilitation Awards for Scientists and Engineers with Disabilities
(FASED) provide funding for special assistance or equipment to enable
persons with disabilities (investigators and other staff, including
student research assistants) to work on NSF-supported projects. See the
program announcement or contact the program coordinator at (703)
306-1636.

The National Science Foundation has Telephonic Device for the Deaf (TDD)
and Federal Relay Service (FRS) capabilities that enable individuals
with hearing impairments to communicate with the Foundation regarding
NSF programs, employment, or general information. TDD may be accessed at
(703) 306-0090 or through FRS on 1-800-877-8339.

The National Science Foundation is committed to making all of the
information we publish easy to understand. If you have a suggestion
about how to improve the clarity of this document or other NSF-published
materials, please contact us at plainlanguage@nsf.gov.

Publication Number: NSF 00-116

[^1]: National Science Foundation, *Engineering Education Coalitions:
    Program Solicitation*, NSF 89-107, Directorate for Engineering and
    Directorate for Science and Engineering Education, p.1.

[^2]: R.K. Yin, "The Case Study Method as a Tool for Doing Evaluation,"
    *Current Sociology*, **40**: 121-137 (Spring 1992).

[^3]: \"Year 4 Report of the Foundation Coalition Program at Arizona
    State University\" (October 26, 1997), p.24.

[^4]: *Op. cit.*, p.34.