Update to the 2004 Committee of Visitors Report

NSF Division of Engineering Education and Centers

**MEMORANDUM**

As noted below, the following are observations, recommendations, and
responses contained in the 2004 COV Report for the Division of
Engineering Education and Centers (EEC), and the 2004 Response, the 2005
Update, and the 2006 Update to that COV report.

1.  **COV Finding/Recommendation:** The COV notes that, while most merit
    review procedures were found to be effective, a wide variance in the
    overall quality and level of detail of the reviews was observed. The
    report supports the continued use of the review templates that are
    currently used in EEC programs and suggests that these templates
    should be further improved to increase their effectiveness.

**2004 NSF Response:** EEC wholeheartedly endorses this recommendation
and notes that review quality has improved significantly since
program-specific review templates were instituted division-wide at the
urging of the FY 2001 COV. We will implement the specific
recommendations to appoint a chair for each review panel and to insert
language in each template urging the panelists to submit their reviews
before the panel meeting. We will also provide a reference to the NSF
document that provides guidance for and examples of activities that
address the broader impact criterion
(<http://www.nsf.gov/pubs/2002/nsf022/bicexamples.pdf>).

**2005 NSF Response:** We have continued to implement the
recommendations outlined in our 2004 response, continuing review
templates, appointing a chair for each review panel, and providing
information on NSF guidelines for review criteria. While some variance
in review quality likely will always exist, we believe that these steps
are improving the overall quality of the reviews we are receiving in our
panels.

> **2006 NSF Response:** The three specific actions indicated in the
> 2004 EEC Response have been continued. With these actions and the
> continued use of program-specific review templates, the review
> community has been continually educated as to what comprises a high
> quality review, with a commensurate increase in average review quality
> and decrease in the variance of that quality.

2.  **COV Finding/Recommendation:** The COV suggests that the diversity
    of the reviewer pool should be increased and, while recognizing that
    current policies make the collection of comprehensive demographic
    data difficult, indicates its frustration with the poor quality of
    the demographic information provided to evaluate the diversity of
    the pool. In addition, the COV urges EEC to set diversity goals for
    all programs that are aggressive, but realistic.

**2004 NSF Response:** Although there is room for improvement, EEC
review panels currently include a significant number of women,
underrepresented minorities, and industrial reviewers. Regarding data
availability, there is talk of expanding to all of ENG a successful
pilot in SBIR and CMS of a system that prompts reviewers until they
self-identify their demographic information. EEC will participate
enthusiastically in such a system.

**2005 NSF Response:** EEC review panels have continued to be diverse,
with significant numbers of women, underrepresented minorities, and
industrial reviewers participating in panel reviews. A Directorate-wide
reviewer database, however is not available as of this writing, but an
effort is underway to complete its development. In the meantime, we are
working to maintain better records on reviewer demographic information
for future COV evaluations.

> **2006 NSF Response:** EEC is always striving to improve diversity and
> its review panels reflect among the highest, and probably the highest,
> level of diversity within the Engineering Directorate. As pointed out
> in the recommendation, if reviewers choose not to self-identify their
> ethnic or racial characteristics, NSF staff are not allowed to assess
> them. The hoped for rollout of the Directorate-wide reviewer database
> has not yet occurred.

3.  **COV Finding/Recommendation:** The COV found that the EEC portfolio
    of awards is consistent with program guidelines and reviewer
    recommendations. While praising the ERC program for its innovative
    awards, integration of research and education, and identification
    and support of new investigators, the COV recommends that smaller,
    interdisciplinary teams be funded in preference to increasing the
    size of individual ERC awards.

**2004 NSF Response:** Developing a viable mechanism for funding "small,
interdisciplinary groups" in ENG has been under discussion for at least
10 years. The establishment of the Nanoscale Science and Engineering
Initiative enabled the funding of a significant number of Nanoscale
Interdisciplinary Research Teams (NIRT), but did not address the broader
issue of how to foster efforts in other disciplines of a scale between
individual investigator awards and ERCs. In the current budget
environment, even keeping ERC funding flat will only allow very limited
funding for such efforts. This may be a good time to begin a planning
process for the time when budgets begin to increase again.

**2005 NSF Response:** As noted in our 2004 response, developing a
viable mechanism for funding small, interdisciplinary groups would
require an improved budget environment than we are current facing. As
part of the ENG strategic planning process, new ideas for funding future
ERC programs are being explored that could include a provision for
funding of smaller groups pursuing higher risk research. However, no
decisions have been made as of this writing.

> **2006 NSF Response:** The Directorate for Engineering will be
> reorganized in FY 2007, to include an Office for Emerging Frontiers in
> Research and Innovation (EFRI). EFRI will recommend, prioritize, and
> fund interdisciplinary initiatives at the emerging frontier of
> engineering research and education. These investments will represent
> transformative opportunities, potentially leading to new research
> areas for NSF, ENG, and other agencies, and the inception of new
> industries providing opportunities for US leadership in, and/or
> significant progress on recognized national needs. Per project funding
> of up to \$500,000 per year for up to 4 years should be sufficient to
> fund the types of small, interdisciplinary teams suggested by the COV.

4.  **COV Finding/Recommendation:** The COV observes that EEC programs
    have been highly successful in meeting the PEOPLE goal and that its
    programs have had a dramatic impact on diversity, curricula, and
    pre-college outreach. Further, EEC programs have provided the
    impetus for entirely new degree programs, produced breakthrough
    results that are redefining performance limits in critical
    technology areas, and are making significant contributions to
    economic development.

**2004 NSF Response:** We agree.

**2005 NSF Response:** EEC continues to actively support the NSF PEOPLE
strategic goal.

> **2006 NSF Response:** This is still the case. We're particularly
> proud of the 2006 Class of Engineering Research Centers (ERC).

5.  **COV Finding/Recommendation:** The COV observes that the
    overwhelming majority of EEC awards are to research-intensive
    institutions and that more capacity needs to be built at other
    institutions.

**2004 NSF Response:** There is a bit of a "chicken and egg" issue here,
in that many smaller engineering institutions don't have the
infrastructure to generate competitive proposals in our mainline
programs. We are encouraging partnering and have asked ERCs to reach out
to non-research-intensive institutions. We have also asked REU sites to
recruit from smaller schools and have provided a number of
Department-Level Reform planning grants to small departments and an
implementation award in FY 2003 to Sweet Briar College to set up a new
engineering program.

**2005 NSF Response:** The size of ERC awards still demands that a
competitive proposal show that a significant research infrastructure is
available to support the research. While this still favors the
research-intensive schools, we are beginning to see ERC proposals from
new schools that would not be classified as of this writing as research
intensive. They are not yet competitive, but we believe that the
proposal experience could help them in understanding how to become more
competitive in these large programs. The ERC program continues to
require that Centers develop partnerships with non-research intensive
schools, and these have provided an excellent means of involving faculty
and students in cutting edge research. In other EEC programs,
particularly the I/UCRC program and the engineering education programs,
there is a broad range of institutions being supported.

> **2006 NSF Response:** The steps outlined above are beginning to show
> progress. The ERC for Reconfigurable Manufacturing Systems at the
> University of Michigan has recruited Morgan State University, an HBCU,
> as a core partner and is working with Morgan State to upgrade their
> manufacturing programs. Several ERCs are establishing 4+1 programs to
> recruit students from predominantly undergraduate institutions to
> pursue masters degrees, with the hope that many will continue to
> doctoral studies. Interestingly, the Sweet Briar College engineering
> program was studied as a possible model for a proposed new engineering
> degree program (that would be the first 4-year engineering degree
> program at a Tribal College) at Salish Kootenai College, in Montana.

6.  **COV Finding/Recommendation:** The COV recommends that EEC
    undertake a comprehensive study to answer the following questions:
    What will ERCs look like in 5-10 years? What are the overarching
    goals of the EEC Educations and Human Resource Development Programs?

**2004 NSF Response:** EEC is intensifying its efforts to address these
issues and the appropriate metrics for evaluating progress as part of
the ENG-wide strategic planning process that is currently underway.

**2005 NSF Response:** As part of the ENG-wide strategic planning
process, EEC has been actively addressing these two questions. The ERC
program is being examined in light of 20 years of success, and proposals
for new ways of continuing the program for the next 10 years are being
explored. As of this writing, these plans are still in development but
we expect to have any changes in the program implemented before the next
round of ERC proposal preparation. Overarching goals for both the
engineering education and human resource development programs have been
developed and are included as part of the EEC and ENG strategic plans.

> **2006 NSF Response:** The first phase of the ENG-wide strategic
> planning process is complete and has resulted in a major
> reorganization of the Directorate for Engineering, effective October
> 1, 2006. EEC was an active participant in the planning process. Among
> the changes for EEC is the inception of a "Third Generation" of the
> ERC Program in the FY 2006 solicitation. The new generation will focus
> on transformational research and education, innovation, partnership
> with small business, and international collaboration. In addition,
> EEC's education programs have increased their emphasis on the
> fundamental study of how engineers learn engineering and deemphasized
> the development of engineering curricula without the benefit of such
> understanding.

7.  **COV Finding/Recommendation:** The COV requests that increased
    attention be paid to planning and assessment of the education and
    human resource assessment programs, including cross-project
    evaluation.

> **2004 NSF Response:** We are well on our way to realizing this
> suggestion, having funded a formal evaluation study of the
> department-level reform program from its inception in 2002, initiated
> an evaluation of the RET program in 2003, and completed a study of the
> PFI program in 2004.

**2005 NSF Response:** The initial study of the RET program has been
completed and a more expanded study will begin in FY06. The other
studies are still underway. In addition, we are beginning the planning
for an evaluation study of the REU sites program.

**2006 NSF Response:**

EEC co-funded the National Academy of Engineering studies, *The Engineer
of 2020: Visions of Engineering in the New Century*, published in 2004,
Educating the Engineer of 2020: *Adapting Engineering Education to the
New Century*, published in 2005, and *Enhancing the Community College
Pathway to Engineering Careers*, published in 2005. In addition, the SRI
International studies, *The Impact of Industry Interaction with ERCs*
and *Evaluation of the Research Experiences for Teachers (RET) Program:
2001-2005* were completed in 2004 and 2006, respectively, and the study
of the planning awards for the Department-level Reform and Bridges for
Engineering Education programs, *Educational Innovation Through
Interdisciplinary Collaboration: A Study of NSF Planning Grants,
2002-2003*, was completed in 2006. Evaluations of the Bioengineering and
Bioinformatics Summer Internship (BBSI) Program, the Research
Experiences for Undergraduates (REU) Site Program, and on the economic
impacts of the 3 Engineering Research Centers that graduated in 2006
were initiated in 2006.

8.  **COV Finding/Recommendation:** Finally, the COV made
    recommendations with regard to the COV process. They requested that
    their instructions be modified to better specify the expectations of
    the NSF with respect to the deliverables from the COV process, that
    they be provided with easy electronic access to jackets, and that a
    method be developed for the random selection of an adequate number
    of jackets to provide a valid sample across all programs.

**2004 NSF Response:** The COV instructions are modified every year and
seem to be improving. This recommendation should assist that process.
The jacket selection process is more difficult. For the FY 2001 COV, we
selected every 10^th^ jacket by jacket number, which should give a
random sample, and in FY 2004, the chair of the COV selected the jackets
for examination to assure broad program coverage. Both methods were
found to be inadequate by the respective committees. As the FY 2004 COV
observes, EEC programs are so diverse that it may be impossible for the
COV to examine enough jackets during their time at NSF to make them feel
comfortable that they haven't missed something. The answer may be to
give COV members easy remote access to all of the jackets before the
COV. We attempted this in the FY 2004 COV with mixed success, since we
were the first to try it and we encountered a number of bugs.
Nonetheless, the COV members took full advantage of the access they got
and strongly recommended that NSF make the necessary modifications to
the e-Jacket system to make early electronic access possible for future
COVs.

**2005 NSF Response:** Access for the COV to eJacket is available now.
Using what we have learned in the past two COVs, we will attempt a
better sampling strategy for the next COV.

> **2006 NSF Response:** We are pleased that the granting of electronic
> access to COV members prior to their meeting is now the standard
> procedure at the NSF and believe that the pioneering experiment of the
> 2004 EEC COV had some influence on the development of this capability.