November 29, 2001

Dr. Harry Armen

Director of Research and Development

Advanced Systems and Technology

Northrup Grumman Corporation

Mailstop A01-26, South Oyster Bay Road

Bethpage, NY 11714

Dr. Thomas D. O\'Rourke

School of Civil and Environmental Engineering

273 Hollister Hall

Cornell University

Ithaca, NY 14853-3501

Dear Harry:

This is in response to the report of the Committee of Visitors (COV) for
the Division of Civil and Mechanical Systems of the Directorate for
Engineering, which was presented to the Engineering Advisory Committee
on October 24, 2001, and which was adopted at that meeting.

At the outset, however, I must thank you and your Committee for a
thorough and thoughtful job well done. The report stimulated a lively
discussion at the Advisory Committee meeting. Dr. Priscilla Nelson,
Director of the Division of Civil and Mechanical Systems, gave a
detailed response at the meeting. I know that she is giving full
attention to your recommendations, many of which she has already acted
upon.

First, I am pleased that you find that the Division has been well
managed and that the review procedures are fair, efficient, and
effective. I also note your recommendations that the Division include
more representatives from government and industry on the review panels
and that reviewer qualifications should be more visible in project
folders. I share your concern that proposals should be processed as
promptly as possible and, in particular, that the dwell time for
declinations be minimized.

With regard to the review processes, the fact that the majority of
reviewers did not address the broader impacts criterion is a matter of
some concern. This is a pervasive problem and much attention is being
given to it across the Foundation. However, I note that the period of FY
99-FY 01was a transition period in merit criteria implementation, and
that current assessments indicate a strong improvement in the process
for all engineering divisions.

I agree with your suggestion that the size and duration of awards be
increased. However, there is pressure to use our limited funds to
support as many proposals as possible, and a significant increase in
award

size will be very difficult to achieve under flat or effectively
decreasing disciplinary program budgets. We have wrestled with this for
some time. Your recommendation will help move our actions in the
direction you support.

NSF is very active in promoting diversity in all our programs. It is
pleasing to see that you find that CMS has made a concerted effort to
involve underrepresented groups in the reviewing process. Your comment

about using a metric to judge performance is one that I plan to pass on
to other divisions.

Secondly, with regard to your assessment of the Division\'s outputs and
outcomes, I am pleased that you find that the Division has successfully
met its goals with regard to People, Ideas, and Tools. Your
recommendation that internal strategic planning for the future should
continue is timely. The strategic planning is one of the important
ingredients for the Division to continue to be strong and relevant.

Your COV is the first to be conducted in two stages. While this requires
more travel and time for COV members, your positive assessment of this
arrangement will no doubt lead other COVs to consider this schedule. The
usual three-day meeting is very compressed and we have resisted
extending this to four, considering that this would be an unreasonable
amount of time to request of members. I hope now that other COVs will
try your two-meeting format. Your detailed recommendations on the
structure and content of the meeting are very helpful.

Comments and recommendations such as yours aid us in two ways. First,
they help us improve the performance of our programs. Secondly, as we
are now using the COVs to include an assessment of the outcomes of our
programs, your comments enable us to fine-tune our COV procedures and to
provide valuable information in preparing its response to the Government
Performance and Results Act (GPRA).

On behalf of the Foundation, let me again thank you and the members of
the COV for your efforts in helping us maintain a high standard of
service to the engineering research and education community. Through
activities such as the work of the COVs, we receive the feedback we
require to be more

responsive to the needs of that community and the nation.

Sincerely,

Esin Gulari

Acting Assistant Director for Engineering

Enclosure

Copy to:

Director, NSF

Inspector General, NSF

Chief Operating Officer, NSF

Chief Financial Officer, NSF

Chief Information Officer, NSF

Director, OIA

Members, Committee of Visitors

NSF Committee Management Officer, HRM

Priscilla Nelson, CMS/ENG

(Identical letter sent to Dr. Thomas D. O'Rourke)

**Review of the Division of Civil and Mechanical Systems\
of the Engineering Directorate**

##### Submitted to:

Dr. Louis Martin-Vega\
Acting Assistant Director of Engineering, National Science Foundation

##### Dates of Visits by the Committee of Visitors:

##### April 26-27, 2001 to evaluate Section A - Integrity and Efficiency of the Program's Processes and Management;

##### June 18-19, 2001 to evaluate Section B - Outputs and Outcomes of NSF Investments in CMS, and Section C - Supplementary Areas: Improvements, Performance, and the COV Process 

**Members of the Committee of Visitors:**

-   Harry Armen (Co-chair)\
    Northrop Grumman Corporation

-   James G. Batterson\
    NASA Langley Research Center

-   Steven B. Chase\
    Federal Highway Administration, Turner-Fairbank Highway Research
    Center

-   Shirley J. Dyke\
    Washington University, Department of Civil Engineering

-   Horacio Espinosa\
    Northwestern University, Department of Mechanical Engineering

-   James E. Hubbard\
    IproVica Corporation

-   Anne S. Kiremidjian\
    Stanford University, Department of Civil and Environmental
    Engineering

-   Howard Kunreuther\
    University of Pennsylvania, Wharton School

-   Henry L. Michel\
    Parsons Brinckerhoff

-   Carol Miller\
    Wayne State University, Department of Civil and Environmental
    Engineering

-   Horace K. Moo-Young\
    Lehigh University, Department of Civil and Environmental Engineering

-   Douglas J. Nyman\
    D. J. Nyman & Associates

-   Thomas D. O\'Rourke (Co-chair)\
    Cornell University, School of Civil and Environmental Engineering

-   Jocelyn M. Seng\
    Owens Corning Science & Technology Center

**Programs of the Division:**

-   Solid Mechanics and Materials Engineering (SMME)

-   Infrastructure and Information Systems (IIS)

-   Dynamic System Modeling, Sensing and Control (DSMSC)

-   Geotechnical and Geohazards Systems (GHS)

-   Structural Systems and Engineering (SSE)

-   Network for Earthquake Engineering Simulation (NEES)

Number of Actions Reviewed: 153

**Executive Summary**

The Committee of Visitors (COV) for the Civil Mechanical Systems
Division (CMS) met at the National Science Foundation on April 26-27 and
June 18-19, 2001. The results of the COV deliberations are contained in
the attached report, which is divided into three parts, addressing A)
integrity and efficiency of processes and management, B) outputs and
outcomes of NSF investments, C) improvements in the Division, program
performance, and the COV review process. Included in the third part
(Section 12) is a report on the George E. Brown, Jr. Network for
Earthquake Engineering Simulation, herein referred to as NEES.

# A. Integrity and Efficiency of the Program\'s Processes and Management

The principal review mechanism for CMS has been the review panel. The
panels were well organized, with a good representation of people from
different institutions and parts of the country. In general, the review
procedures were fair, efficient, and effective. The Division should
continue its use of review panels. The advantages of this process are
significant, and include timely assessment of proposals, opportunity for
junior researchers to network with more senior members of the community,
and an open forum for the exchange of ideas, thereby diminishing the
potential for mistakes.

The review process can be improved by involving more representatives
from industry and government in the review panels. Furthermore, there
should be sufficient information in project folders for a future COV to
judge reviewer qualifications. In general, the documentation in sampled
folders was found to be inadequate for assessing such qualifications.
Several measures can be taken to link proposals with reviewers having
appropriate expertise, including requesting a technical self-assessment
from prospective reviewers and maintaining a systematic database on the
background and qualifications of reviewers.

The average time from receipt of unsolicited proposals to notification
of award was approximately six to seven months, which is consistent with
NSF goals. The dwell time for declinations reviewed by the COV was 11 to
12 months. In the CMS overall Divisional report data are provided that
show the Divisional average to be significantly better than this
(slightly greater than seven months). The time taken to process declined
awards should be reduced. The goal should be to achieve a dwell time of
six months for both awards and declinations.

It is clear that CMS has made a concerted effort to involve
underrepresented groups in its review process. It is not possible,
however, to judge how this involvement compares with the actual
participation of such groups in the US scientific and engineering
community. There are a series of congressionally mandated biennial
reports on the status of women and minorities in science and
engineering. The latest document (NSF 00-327), \"Women, Minorities and
Persons with Disabilities in Science and Engineering: 2000,\" contains a
203-page appendix of statistics. It would be advisable to glean
pertinent information associated with engineering from these statistics
and use them as a metric for assessing the Division's success in
engaging underrepresented groups.

The reviewers and Program Directors were effective in addressing the
intellectual merit criterion in the review of unsolicited proposals. The
majority of reviewers of unsolicited proposals, however, did not address
the broader impacts criterion. Program Directors should make it clear to
reviewers that this criterion is important. The Division is encouraged
to place greater emphasis on the broader impacts criterion. If future
proposals will be judged more rigorously with respect to this criterion,
then the program officers need to ensure that its importance is properly
communicated to potential principal investigators

With respect to the portfolio of awards, the COV concluded that awards
are consistent with program guidelines and reviewer recommendations.
Particularly noteworthy are the number of CAREER awards. The Division
has been committed to supporting young researchers, and is commended for
its dedication to the CAREER program.

The Division should continue to increase the size and duration of
awards. Increased size and duration are necessary to encourage
multidisciplinary proposals that involve multiple investigators. They
are also needed to promote higher quality.

More emphasis should be given to high risk/high return projects. Steps
in the development and implementation of such a policy include
establishing a working definition within the Division of "high risk/high
return", setting aside a specific percentage of the yearly CMS funds to
support such proposals, and clear communication of this policy to
prospective principal investigators and review panels.

# B. Outputs and Outcomes of NSF Investments

Programs administered by CMS have been successful in aggregate in
meeting the People, Tools, and Ideas strategic outcome goals. There is
ample evidence of successful outcomes for all three goals to confirm
that CMS has fulfilled its mission with respect to NSF outputs. Numerous
examples to support this conclusion are provided in the report.

Several aspects of the strategic outcome goals related to People will
benefit from additional attention. Specifically, the goals for improved
K-12 math and technology skills and a workforce that reflects diversity
were not addressed in the preponderance of annual reports reviewed by
the COV. It will be advantageous for the Division to encourage K-12
educational activities and the involvement of minorities through its
program announcements and interactions with the research community.

# C. Supplementary Areas - Improvements, Performance, and the COV Process

The Division should play a stronger role in the environmental area and
utilize the expertise of geoenvironmental researchers supported through
GHS in this effort. More research initiatives should originate in CMS
that result in co-sponsorship with other Divisions and directorates. In
particular, CMS should take the lead in developing initiatives with the
SBE Directorate.

Internal strategic planning should continue to be undertaken to address
the question of what CMS will look like five years from now. This effort
should include the identification of fundamental themes and outcomes
that CMS wishes to pursue.

Serious consideration should be given to reducing the proposal load of
Program Directors. A key issue for CMS is the balance between process
and stewardship. A heavy proposal load affects the time available for
Program Directors to develop initiatives and to interact with proposers
and funded researchers. Increased stewardship of individual researchers
requires a more moderate level of processing than currently exists.

There is ample evidence that CMS is meeting its non-GPRA goals through
its restructuring, participation in research initiatives, and leadership
of NEES. Moreover, there appears to be an enthusiasm and focus within
the Division that will support innovative and meaningful activities in
the future.

There are several measures that can be taken improve the COV review
process. For example, the annual reports of principal investigators are
used as a primary COV resource. These reports are not organized in a way
that responds to the GPRA-related questions. Asking principal
investigators for information that explicitly addresses the COV
performance assessment questions will make the evaluation process more
efficient and develop a better understanding among researchers, NSF
personnel, and COV members about performance goals and metrics.
Moreover, there should be more guidance for principal investigators in
addressing GPRA-related issues in their annual reports. Some examples of
appropriately prepared reports could be posted on the NSF web site to
help in the annual report preparation.

During the first day of the COV review, there should be a top-down
assessment of each program by the relevant Program Directors. The
Program Directors should present their views and objectives, and provide
guidance about how the COV can best help the Division with its advice.
It would also be beneficial for the Division Director to give a top-down
briefing on how the Division has responded to the recommendations in the
previous COV report, and clearly identify the objectives and
significance of the COV process to all members of the COV, particularly
first-time members. The significance of the COV report should be
stressed with respect to improving the investment decisions and
effectiveness of programs for the Division, the Directorate, and the
NSF. In addition, it should be made clear that these reports are
reviewed, approved, and used by the Advisory Committee to carry out
their responsibilities, including those associated with assessing the
Directorate\'s performance in meeting GPRA-related goals.

A brief summary of the activities of the earthquake engineering research
centers should be presented at future COV reviews. Significant CMS
resources have been invested in the centers, and an overview of their
principal accomplishments would be useful.

Future COV reviews should consider as viable options the use of
co-chairs, involvement of the previous COV chair, and scheduling the COV
review as two separate meetings. The use of co-chairs helps to
distribute the load, and the involvement of the previous COV chair
provides valuable experience in organizing the meetings and preparing
the report.

Section 12 - [Network for Earthquake Engineering Simulation
(NEES)]{.underline}

Lessons learned in developing NEES will have ramifications on the way
research is conducted throughout NSF. This program should receive
attention, resources, and advice from the Engineering Directorate. It
should be a priority for the Engineering Directorate to document the
experience gained and process that evolves in organizing and operating
NEES so that the appropriate knowledge can be used in other Divisions
and directorates.

**The organizational chart for NEES includes a Management and Oversight
Board that is linked directly to the NEES Program Director. This board,
however, consists of personnel from various Divisions and directorates
within NSF. It is equally important to obtain the advice and guidance of
earthquake engineering and information technology experts external to
NSF. CMS should consider forming an independent Scientific Advisory
Committee or Expert Panel to provide technical as well as
community-building advice to CMS for this complex project.**

Although there are no research outcomes of NEES at this early stage in
its development, there will have been sufficient time for outcomes when
the next COV review occurs three years from now. Performance measures
should be developed that will be effective in gauging the success of the
program at that time. An independent Expert Panel, as discussed above,
could help with the identification and development of the appropriate
metrics.

The impact of NEES will be broad and profound. The substantial
experimental and computer-networking infrastructure created by NEES will
require significant maintenance costs after the NEES MRE appropriation
has been spent. CMS needs to develop a vision for its earthquake
engineering research program that encompasses the influence of NEES. The
target years for this vision and planning scenario are 2005-2010. The
planning should start soon and include an assessment of how NEES will
relate to and affect the projects and budgets within the five other
programs of the Division.

**COV MEMBER SIGNATURES**

Review of the Division of Civil and Mechanical Systems

**Dedication:**

On May 23rd, 2001, Henry L. Michel, dear colleague, friend and mentor to
scores of engineers in the field of Civil and Mechanical Systems, and
most recently a member of this Committee of Visitors, died in his home
in New York City. Henry was chairman emeritus and former president of
Parsons Brinckerhoff. He enjoyed more than a half-century career in
which he assumed a multitude of roles and responsibilities that
ultimately culminated in his oversight of large-scale transportation
projects that spanned the globe. In addition to his many technical and
administrative talents, Henry possessed the quality of passion, a
quality he applied to each and every endeavor in which he was engaged,
and to each and every professional and personal relationship he
fostered. In the minds of almost everyone who ever met him, Henry\'s
talents and passion for his work, together with his genuine love of
people have made him the quintessential representative of the
time-honored moniker, \"a gentleman and scholar.\"

He enhanced the lives of all of us who were fortunate enough to have
been touched by his presence, as he graced our profession with his
talents. As a tribute to his contributions to this effort, the Committee
Of Visitors dedicates this report to Henry.

**Introductory Comments:**

The Committee of Visitors (COV) met at NSF headquarters on 26-27 April
2001 to evaluate the integrity and efficiency of the processes and
management of the Civil Mechanical Systems (CMS) Division. A second
meeting was held on 18-19 June 2001 to evaluate the outputs and outcomes
of NSF investments in the CMS Division, and also to address questions
pertaining to program areas needing improvement, performance in meeting
program specific goals, and the COV review process.

At the end of FY 1999, CMS was reorganized from a system of program
elements organized under three program clusters into six distinct
programs focused on Solid Mechanics and Materials Engineering (SMME),
Infrastructure and Information Systems (IIS), Dynamic System Modeling,
Sensing and Control (DSMSC), Geotechnical and Geohazards Systems (GHS),
Structural Systems and Engineering (SSE), and the Network for Earthquake
Engineering Simulation (NEES). The COV review process was organized in
accordance with the current organizational structure of CMS. There were
14 members of the COV, including the two co-chairs. Two COV members were
assigned to each program, with the co-chairs assisting in the evaluation
of several programs.

Table 1 provides a summary of the characteristics of the jackets, or
files, that were reviewed for CMS. Each jacket contained documentation
about a specific proposal, including dates of processing, reviewer
comments, and action taken. A total of 153 jackets, each providing
documentation for specific actions, were reviewed. Listed in the table
are the number of awards and declinations reviewed for each CMS program.
Also listed for each program, with the exception of the IIS and NEES
programs, are the number of unsolicited proposals and proposals
associated with special programs (such as CAREER, POWRE, HBCU, etc.)
that were reviewed.

Table 1 Summary of Jackets Reviewed for the Civil and Mechanical Systems
(CMS) Division

+-------------+-------------+-------------+-------------+------------+
| CMS         | Awards      | D           | Unsolicited | Special    |
|             |             | eclinations |             | P          |
| Programs    |             |             | Proposals   | rograms^1^ |
+-------------+-------------+-------------+-------------+------------+
|             |             |             |             |            |
+-------------+-------------+-------------+-------------+------------+
| SMME        | 25          | 10          | 32          | 3          |
+-------------+-------------+-------------+-------------+------------+
|             |             |             |             |            |
+-------------+-------------+-------------+-------------+------------+
| DSMSC       | 12          | 13          | 16          | 9          |
+-------------+-------------+-------------+-------------+------------+
|             |             |             |             |            |
+-------------+-------------+-------------+-------------+------------+
| SSE         | 15          | 5           | 15          | 5          |
+-------------+-------------+-------------+-------------+------------+
|             |             |             |             |            |
+-------------+-------------+-------------+-------------+------------+
| GHS         | 25          | 9           | 24          | 10         |
+-------------+-------------+-------------+-------------+------------+
|             |             |             |             |            |
+-------------+-------------+-------------+-------------+------------+
| IIS         | 19          | 7           | NA^2^       | NA^2^      |
+-------------+-------------+-------------+-------------+------------+
|             |             |             |             |            |
+-------------+-------------+-------------+-------------+------------+
| NEES        | 8           | 5           | \--         | 13         |
+-------------+-------------+-------------+-------------+------------+
|             |             |             |             |            |
+-------------+-------------+-------------+-------------+------------+
| Subtotals   | **104**     | **49**      | **87+**     | **40+**    |
+-------------+-------------+-------------+-------------+------------+
|             |             |             |             |            |
+-------------+-------------+-------------+-------------+------------+
| Total       |             |             |             |            |
| Number of   |             |             |             |            |
| Actions     |             |             |             |            |
| Reviewed =  |             |             |             |            |
| 153         |             |             |             |            |
+-------------+-------------+-------------+-------------+------------+

^1^Special Programs refers to CAREER, POWRE, GOALI, HBCU, PATH, I/UCRC
programs and NEES.

^2^Not available

The CMS program also includes one program element for the Major Research
Equipment (MRE) project - - the George E. Brown, Jr. Network for
Earthquake Engineering Simulation (NEES). The NEES program is the first
MRE project to be supported in the Engineering Directorate. It is a
unique effort to develop a model for collaborative research through
advanced information technologies. Because of its strategic importance
and differences with respect to the other five CMS programs, this report
is organized to provide specific comments about NEES.

The report is divided into three parts, A through C, dealing with
subsets of the core questions presented by NSF to the COV. Comments
pertaining to process efficiency and integrity of the NEES program are
provided in Part A. Section 12 of Part C presents recommendations and
observations concerning the NEES program.

**A. INTEGRITY AND EFFICIENCY OF THE PROGRAM'S PROCESSES & MANAGEMENT**

1.  **Effectiveness of the program's use of merit review procedures:**

```{=html}
<!-- -->
```
a.  **Overall design, including appropriateness of review mechanism
    (panels, ad hoc reviews, site visits):**

> The principal review mechanism for CMS is the review panel. In some
> cases, evaluations were made by a combination of panel and mail
> reviews or by mail reviews only. The panels were generally well
> organized. There appears to be a good representation of people with
> appropriate expertise from different institutions and parts of the
> country, although some improvements could be achieved as noted under
> Question 3 below.
>
> The panel review process has several advantages. It allows for a
> relatively rapid evaluation of proposals in that meeting deadlines and
> proposal evaluations are set according to CMS schedule and milestones.
> A panel provides the opportunity to bring junior members of the
> research community into contact with more senior, established
> researchers and practitioners. In this way, networking can be promoted
> and junior researchers can gain valuable insights about the
> preparation of proposals and the review standards. By providing a
> forum for open discussion about proposals, the panelists are able to
> change their views and improve their evaluations on the basis of
> interactions with other panelists. In this way, the potential for
> misconceptions and mistakes is diminished and the panelists' own views
> are subject to scrutiny and modification. On the other hand the COV
> expressed some concern that the panel review process, that is
> inherently consensus-based decision making, will not be particularly
> conducive to selecting high-risk, high-potential payoff projects.
>
> Whereas most COV members support the panel review process, it was
> recognized that some panels might lack specific expertise in some of
> the technologies and subjects involved in the proposals. It may be
> advantageous, therefore, to continue to seek mail reviews from experts
> to supplement the reviews and discussions of the panelists. The
> objective would be to identify a limited number of experts to provide
> written reviews of specific proposals in advance of the panel meeting.
> Since authorities in a particular discipline or technology would
> provide these written reviews, they would help provide a technically
> more robust review and help assure that the appropriate expertise is
> available for the diverse subjects covered in the proposals.
>
> With respect to NEES, NSF should consider requiring PIs, selected for
> the short list of proposals, to visit NSF and present their proposal,
> defend the technical basis of their proposal, and respond to questions
> and concerns. The requirement for reviewer confidentiality interferes
> with this process, but possibly an effective means to do this may be
> found. One possibility would be to link the reviewers to the PI's by
> computer teleconferencing. This would allow the reviewers' identities
> to remain confidential, and yet give them the ability to ask
> questions.

b.  **Effectiveness of program's review process:**

> The panel review process is effective. The jacket reviews show that
> there were an adequate number of panelists for the number of proposals
> to be processed. There seems to be significant consistency in the
> review comments particularly when the review process was based on a
> panel review. The reviewers were geographically well balanced. In
> addition, there was diversity in institutional representation and
> experience. There has been a marked increase in the number of women
> review panelists and external reviewers in recent years. The use of a
> panel having the benefit of a set of prepared reviews by a subset of
> the panel appears to be highly effective since there is an opportunity
> for thoughtful reviews in advance, followed by open discussion with
> all panelists.
>
> The panel review process appears to be working well because it allows
> reviewers to discuss the potential impact of a specific proposal.
> Having more experts on the panel to review the proposals would be
> beneficial. The panels were quite thorough in their reviews,
> addressing both merit review criteria sufficiently.
>
> **c. Efficiency; time to decision:**
>
> For most programs, the average time from receipt of unsolicited
> proposals to notification of award was approximately 6 to 7 months.
> This dwell time is consistent with the CMS average reported in the
> Divisional Summary Report for the COV and consistent with NSF goals.
> The time from receipt to notification for CAREER awards appeared to be
> longer. It was about 11 to 12 months for the limited number of CAREER
> award jackets reviewed. With respect to IIS, COV members found 8 out
> of the 26 jackets required 17 months, or longer for notification of
> award, while 3 jackets had delays between 24 and 39 months. Since
> CAREER is an NSF-wide program, the COV has determined that the delays
> observed here were not the result of actions by the CMS Program
> Directors, but are more likely attributable to NSF administrative
> practices
>
> The dwell time from proposal receipt to notification of the
> declinations that were reviewed ranged from eleven to twelve months.
> In the CMS overall Divisional report data are provided that show the
> Divisional average to be significantly better than this (slightly
> greater than seven months). The goal should be to achieve a dwell time
> of six months for both awards and declinations.
>
> With respect to NEES, the time required to make a decision varied
> considerably. The System Integrator Scoping Study award and a handful
> of equipment awards were determined fairly quickly and others took
> somewhat longer. In many cases this was because NSF had additional
> questions for the PI to clarify concerns and issues that the panel had
> brought up. This additional step is considered to be appropriate for a
> cooperative agreement on the average of \$4 million. However, none of
> the decisions were delayed an unreasonable amount of time.

d.  **Completeness of documentation making recommendations:**

> In general, there was appropriate documentation in the jackets that
> was well-maintained and relatively easy to track. One exception
> appears to be associated with declinations. In several instances,
> there were no dates on letters notifying proposers of declined awards.
> Better documentation of declination dates is recommended.
>
> The jackets reviewed contained records of conflict of interest
> discussions. In several jackets, there was documentation pertaining to
> declared conflicts that involved reviewers recusing themselves and
> leaving the discussion when home institutions were involved.
>
> With respect to NEES, the jackets were quite detailed and complete.
> Since the Program Director found it necessary to go back to the
> investigators several times with questions about the proposal, there
> is a great deal of information in the jackets describing the
> intentions of the PI in further detail than was provided in the
> proposals themselves. The information is clearly documented.

e.  **Consistency with priorities and criteria stated in the program's
    solicitations, announcements, and guidelines:**

> The jackets reviewed were mainly associated with unsolicited, CAREER,
> and NEES proposals. The criteria for evaluating such proposals
> involve 1) technical merit and 2) broader, societal implications of
> the research. Although these criteria were communicated to proposers
> and reviewers, there is evidence with respect to unsolicited proposals
> that much greater emphasis is placed on technical merit during both
> proposal preparation and subsequent review. This issue is discussed in
> greater detail in the next section.

**Recommendations:**

> 1.1 The use of review panels for evaluating proposals should be
> continued. For the case of the NEES program for which there are
> substantial cost proposals (multi-million dollar), linking panel
> reviewers with PI's via computer teleconferencing should be considered
> to generate a greater degree of interaction in the evaluation of
> proposals.
>
> 1.2 Mail reviews should be solicited from experts when review panels
> lack specific expertise in some of the technologies and subjects
> involved in the proposals. Such input should reflect the views of
> recognized authorities in a particular discipline or technology so
> that a technically robust review can be accomplished.
>
> 1.3 Streamline, to the extent that is practicable, the processing of
> proposals to facilitate timely and constructive feedback to PIs,
> particularly young investigators. The time taken to notify proposers
> of declinations should be reduced to no more than six months.
>
> 1.4 Complete documentation, including date of decision notification,
> should be included with each declination and award jacket.
>
> 1.5 Require Reviewers to identify, with some degree of specificity,
> how well a proposal addresses the goals, priorities, and mission of
> the Program, the Directorate, and/or the NSF.

2.  **The program's use of the NSF Merit Review Criteria (intellectual
    merit and broader impacts):**

    a.  **Performance Goal:** **Implementation of Merit Review Criteria
        by Reviewers:**

> Intellectual merit criteria were weighed heavily in the review of
> proposals. The majority of the COV members felt that the reviewers do
> a very good job in judging the proposals based on NSF criterion #1.
> The COV members who reviewed the SSE program pointed out that Panel
> summary responses varied greatly, ranging from succinct two to three
> bullets to several pages of detailed deliberations. It would be
> advantageous, therefore, for Program Directors to provide more
> guidance to panel members to achieve better consistency in the
> summaries.
>
> The majority of reviewers of unsolicited proposals did not address
> adequately criterion #2. Earlier proposals did not seem to address
> this criterion at all. Reviews of several of the more recent proposals
> did include the broader impact. Program Directors should make it clear
> to the reviewers that this criterion is important, why it is
> important, and that it needs to be considered in the review process.
> Furthermore, if future proposals will be penalized for not meeting
> this criterion, then the program officers need to ensure that the
> criterion and its importance are properly communicated to potential
> PIs.

CAREER proposal reviews typically scrutinize criteria 2 more closely
than is the case for the unsolicited proposals. In general, criteria 1
and 2 were addressed appropriately during CAREER proposal reviews.

With respect to NEES, the panel reviewers did adequately address
criterion 1 in their reviews. In the NEES program, most of the funding
is designated for equipment, and there is little basic research in the
proposals. Thus, the broader impact criterion 2 should be the more
important one in the reviews. The reviewers used this approach.

The NEES panel reviewers were very thorough in addressing criterion 2.
The reviewers often suggested that NSF request clarification on an issue
mentioned in the proposal (e.g., how will this equipment be used for
education, for outreach, etc). The NEES program, if it is to be
successful, must focus on the broader impacts of any equipment grant to
ensure that the community of researchers will make appropriate and
significant use of the sites. For instance the PI's must have a vision
for their site. Additionally, educational efforts must be integrated
into the proposal. The reviewers addressed both these issues.

b.  **Performance Goal: Implementation of Merit Review Criteria by
    Program Officers:**

> Overall, the Program Directors emphasize the intellectual merit
> criterion in the selection of unsolicited proposals for award. In
> general, the intellectual merit criterion was applied with rigor and
> balance. The Program Directors have typically acted on the
> recommendations of the panel. Where the panel emphasized the technical
> merit criterion, the Program Directors have followed suit.
>
> Criterion #2, was given minimum emphasis in the panel review process
> for unsolicited proposals. Accordingly, it received minor emphasis
> from Program Directors.
>
> As indicated above, both criteria 1 and 2 were considered in the
> evaluation of NEES and CAREER proposals by the review panels. The
> Program Directors, acting on the recommendations of the panels, took
> account therefore of both the technical merit and broader impacts
> criteria.
>
> In a few cases, proposals receiving excellent and very good reviews
> and that were rated as highly recommended by panels did not receive
> support. The COV assumes that the lack of support in these instances
> was because of budgetary constraints and the concomitant judgment of
> the Program Director not to fund. The reasons for declination in these
> instances are not well documented. The COV recommends that proposers
> of highly recommended proposals be encouraged to resubmit and that
> appropriate supporting documentation/ justification of the declination
> reasons be maintained in the jacket.
>
> **c. Discuss any concerns the COV has with respect to NSF's merit
> review system:**
>
> NSF's merit criteria system has been well publicized, but poorly
> implemented with respect to criterion #2 in the review process. In
> general, reviewers do not have a good understanding of the terminology
> "broader impacts". NSF may benefit from a formal explanation and
> discussion of the importance of criterion #2 at the beginning of each
> panel meeting, and by formal emphasis of the same criterion when
> communicating with those who are asked to provide mail reviews.
> Without a strong degree of specificity of the term \"broader impact,\"
> reviewers will address this criterion using their own interpretation,
> or the criterion will be ignored.

**Recommendations:**

> 2.1 Program Directors should provide more guidance to review panels to
> achieve a more consistent product and an appropriately comprehensive
> summary of panel deliberations and decisions.
>
> 2.2 Program priorities that are clearly linked to broader Directorate
> and/or NSF priorities must be communicated in program announcements,
> and to Reviewers.
>
> 2.3 Criterion #2, which addresses the broader impacts of the proposed
> research, needs to be clearly defined, emphasized and used more
> vigorously at all levels.
>
> 2.4 The importance of Criterion #2 should be communicated clearly to
> potential principal investigators, and then used in a serious and
> deliberate way to rank proposals. The need for both communicating the
> importance of and utilizing criterion #2 in proposal assessment is
> most acute for unsolicited proposals.

**3. Reviewer selection:**

> **a.** **Use of adequate number for balanced review;**
>
> Most proposals reviewed by a panel had from four to six reviewers,
> some (in the case of IIS programs) have used as few as three
> reviewers. The COV believes that four to six reviewers represent an
> adequate number to provide a balanced review.
>
> A total of sixty different reviewers were used for the fifteen NEES
> proposals considered by the COV. Four to six reviewers were used for
> each proposal. This is appropriate for the type of funding awarded
> within the NEES program.
>
> **b. Use of reviewers having appropriate expertise/qualifications;**
>
> Since the COV determined that the proposal reviews were of good
> quality, then it can be concluded that the reviewers were properly
> chosen. However, if the COV is to provide a thorough evaluation of the
> reviewers expertise and qualifications, then an extensive data base
> should be developed and made available to the COV. In general, the
> documentation included in the sampled project folders did not contain
> specific information about the expertise and qualifications of the
> reviewers to support a thorough evaluation by the COV.
>
> Although the IIS group did observe a preponderance of reviewers from
> the academic community, the IIS report indicates an overall
> distribution of reviewers that appears to have a decent balance (10%
> government and 13% industry). They believe the situation they observed
> underscores a need to solicit reviewers having practical experience.
> Furthermore, they observed that some panels were inappropriate for
> certain proposals. For example, one declined proposal that had a
> social scientist as the PI was reviewed by a panel composed entirely
> of members of the academic community from civil engineering
> departments.
>
> It also appears that many of the reviewers for a particular program
> had similar scientific and technical backgrounds. There was a lack of
> a crosscutting, or multidisciplinary mix of reviewers for programs
> that were of a multidisciplinary nature.
>
> The proposals for the NEES program were mainly from the top
> engineering schools in the nation (as well as the three national
> earthquake centers), where a large portion of US earthquake
> engineering research is being conducted. The need for avoiding
> conflicts of interest, therefore, made it challenging to find a large
> number of highly recognized reviewers for the panel. In some cases
> international reviewers were used to ensure the quality of the panel.
> This was appropriate because these international reviewers are well
> informed about the NSF review criteria and the panel process.
>
> **c. Use of reviewers reflecting balance among characteristics such as
> geography, type of institution, and underrepresented groups;**
>
> Several of the program groups did observe that there was a
> preponderance of representation from academia, with limited industrial
> or government lab representation. The participation of industry is
> increasingly important in addressing merit review criterion #2.
>
> The IIS group determined that there was a lack of international
> expertise in fields (e.g., construction) where much of the
> state-of-the-art research takes place. For example, one proposal
> dealing with mitigation strategies in Asian countries had only
> reviewers from the US, despite the fact that there is considerable
> expertise in this area from other countries.
>
> Information associated with underrepresented groups was contained in
> each of the Program reports. The data indicate that the groups
> identified (other than male) still do not reflect their representation
> in the general population. There are a series of congressionally
> mandated biennial reports on the status of women and minorities in
> science and engineering. The latest document (NSF 00-327) \"Women,
> Minorities and Persons with Disabilities in Science and Engineering:
> 2000,\" contains a 203-page appendix of statistics. It would be
> advisable to glean pertinent information associated with engineering
> from these statistics and present them to the COV to assist them in
> addressing this performance measure. Unfortunately, there were no data
> given to the COV to indicate what percentage these groups represent in
> the engineering community. Additionally there were no historical data
> provided to the COV to track the progress, or lack of progress, in
> this area
>
> The use of reviewers did reflect a balance among the characteristics
> of geography and gender. The GHS group found that Research II and
> non-Ph.D. granting institutions were not well represented, thus they
> concluded that there was less of a balance for institutions.
> Conversely, the SSE group determined the SSE program did an excellent
> job of promoting institutional diversity.
>
> In the case of the NEES program the reviewers were selected from a
> wide range of backgrounds, considering geographic location and
> diversity.
>
> Overall, the statistics associated with women and minorities used in
> the review process are an improvement over those associated with
> previous years. They are equal to, or greater than the representation
> of these groups in the general earthquake engineering researcher
> community.
>
> **d. As appropriate, recognition and resolution of conflicts of
> interest by NSF staff and adequacy of documentation justifying actions
> taken.**
>
> There were only a few instances of conflicts of interest documented in
> the projects reviewed. The NSF staff did a commendable job of
> recognizing potential conflicts, and adequately documenting the
> actions taken and the justification for them.

**Recommendations:**

> 3.1 Request that prospective reviewers provide a self-assessment of
> their level of expertise in the technical area for which they are
> being asked to review. This can reduce the likelihood of having a
> proposal being reviewed by individuals having little expertise.
>
> 3.2 Maintain a common database of potential reviewers from a
> consolidation of various Program Directors' individual databases.
> Consideration should also be given to developing a database from the
> vitae of PIs as they submit proposals.
>
> 3.3 Establish a procedure to collect data associated with the
> background, qualifications, and expertise of potential reviewers.
> Since many reviewers are from academia, and have probably submitted
> proposals to NSF, it is recommended that NSF consider using the
> FastLane system to capture the required data as an ancillary function.
>
> 3.4 Provide, to the COV, pertinent statistics associated with
> underrepresented groups in the scientific, engineering and technical
> community. Only then can an assessment be made concerning the
> Division\'s progress in utilizing these groups in their review
> process.

**Resulting portfolio of awards:**

a.  **Overall quality of science and engineering;**

> Most of the awards reviewed appear to be those that can be
> characterized as high quality. The IIS group observed that there needs
> to be an increase in the number of proposals that link science and
> engineering to social science and public policy issues. The GHS group
> observed that a small percentage of the declinations represented
> significantly low levels of science and engineering.
>
> The NEES group determined that the awards in their program appear to
> be supporting high quality efforts. There were, however, a significant
> number of high quality proposals that were not funded. Thus,
> indicating the highly competitive nature of the proposals submitted
> for award.

a.  **Appropriateness of award scope, size, and duration;**

> Most of the programs awards have size and duration levels that are
> consistent with the respective averages associated with the CMS
> Division. However, the current size and duration of the awards are
> generally determined to be appropriate for exploratory research and
> for developing expertise and research experience for faculty and
> graduate students. These levels for size and duration are perceived to
> be smaller than that required to fund breakthrough, quantum change
> research efforts, without having to request subsequent supplemental
> funding.
>
> The GHS group found that complex, multidisciplinary studies, whose
> duration should be three-to-four years, are being scaled back to
> two-year efforts to satisfy budgetary constraints. This group also
> determined that there was an appropriate increase in size and duration
> of CAREER awards.
>
> The SMME group commented that increasing the size and duration of
> awards cannot be accomplished within the constraints of a flat budget
> projection without reducing the success rate. The current success rate
> is perceived to be, for most of the programs, relatively low for the
> large volume of high quality proposals received.
>
> For the NEES program, the average size of the awards (\$3M to \$5M)
> was determined to be appropriate for the equipment requested. A few
> projects were awarded greater sums than the average, and appropriate
> procedures for authorizing large grants were undertaken by NSF staff.

b.  **Effective identification and support for emerging opportunities;**

> There is a clear dichotomy of responses to this question. The SMME
> group observed that, as a result of the close participation of the
> Program Directors in NSF initiatives, such as nanotechnology and
> information technologies, there has been a successful shift of
> emphasis in the unsolicited proposals received to address these
> emerging areas. The SMME group found that approximately half of the
> research projects they had reviewed were supporting emerging
> opportunities.
>
> The DSMSC group, on the other hand, observed that most unsolicited
> proposals appear to be representing projects that are evolutionary in
> nature. This group determined that emerging opportunities are being
> identified by NSF-wide initiatives, rather than from unsolicited
> proposals. Furthermore, the GHS group observed that the unsolicited
> proposals do not appear to be effectively addressing emerging
> opportunities identified by the NSF. This group further observed that
> small-scale research projects addressing emerging areas are typically
> funded through the SGER program.
>
> The IIS group had difficulty determining the source of the definition
> of "emerging opportunities" (NSF, the PI, the panel). They found
> several proposal declinations that did not refer to the fact that the
> proposals under review were individually addressing the basic NSF
> mission to develop and deploy new information technologies, an
> activity that could be viewed as an \"emerging opportunity. \"
>
> The NEES group viewed the entire NEES program as an effort in
> developing and implementing emerging technology. Such a program has
> never been undertaken on such a widespread basis.

d.  **Appropriate attention to maintaining openness in the system, for
    example through the support of new investigators;**

> The Division is found to be responsive to maintaining openness in the
> system through the support of new investigators. New investigators are
> supported primarily through the CAREER program. This program appears
> to be an effective mechanism for meeting the objective of maintaining
> openness.
>
> The GHS and SSE groups found that new investigators have less success
> in obtaining awards via unsolicited proposals. Many of the reviews
> used the track record of the PIs as a strong determining factor for
> awards.
>
> Due to the nature of the NEES program, teams of investigators are the
> most appropriate use of scientific and engineering talent.

e.  **Evidence that proposers have addressed the integration of research
    and education in proposals;**

> With the exception of the NEES program, there was little evidence of
> the integration of research and education in those projects reviewed
> by the other groups. There were examples of this integration through
> standard models that use graduate students, and to a lesser extent,
> undergraduate students (the REU program). The GHS group found that
> reference to the integration of research and education is often vague,
> with statements such as, "\...a strong effort will be made to recruit
> from the qualified minority pool\..." (CMS 9872618). The CAREER
> program appears to be the primary mechanism that addresses the
> integration of research and education. However, it should be noted
> that the DSMSC group observed that a CAREER HBCU proposal (CMS
> 9984769) containing a plethora of ideas associated with curriculum
> development and other educational activities was declined because it
> was not considered to be strong with respect to merit review criterion
> #1. The NEES program was determined to have a great deal of potential
> for the integration of research and education. From the jackets
> reviewed by this group it was clear that many of the PIs did address
> this integration issue in their proposals.

f.  **Evidence of increased numbers of applications from
    underrepresented groups;**

> In some programs (DSMSC and SMME) the data were not available to
> address this issue. The data provided to the GHS group indicate that
> the representation of women has remained relatively static over the
> past three years. The data for other underrepresented groups show no
> noticeable increase during the reporting period. Similarly, the SSE
> group found no evidence of increased numbers of applications from
> underrepresented groups in the projects they reviewed. The IIS group
> did determine that there were many women, but few minorities, funded
> in the social sciences. The NEES group determined that one of the
> fifteen jackets they reviewed was awarded to an individual from an
> underrepresented group. They also determined a number of co-PIs were
> from underrepresented groups.

g.  **Balance of projects characterized as:**

-   **High-risk**

-   **Multidisciplinary**

-   **Innovative**

    The GHS group found a reasonably balanced mix of multidisciplinary
    and innovative projects. However, there was some evidence that
    high-risk proposals suffer from expectations by members of panel
    reviews that the proposed methodology and outputs should (must) be
    known prior to grant award. Therefore, panelists may be unwilling to
    fund "risky" proposals during periods of tightening budgets. The
    sample of projects reviewed by the SMME group indicates that less
    than half of the projects can be characterized as high-risk and
    innovative, while less than a quarter of these projects are
    multidisciplinary. The IIS group found relatively few proposals were
    innovative in both the social sciences and in engineering Most of
    the projects were attempting to improve existing approaches, or were
    designed to extend findings from earlier research. Although the
    DSMSC group found a significant percentage of the projects they
    reviewed to be multidisciplinary, there was a noticeable lack of
    high-risk projects.

    The NEES group found that awards were given to a select number of
    universities having outstanding records of achievement. In many
    cases the co-PIs consisted of a multidisciplinary team, including
    structural engineers, geoscientists, and computer scientists.
    Although the proposals were for the purchase of equipment, there
    were a few proposals that introduced concepts for new large-scale
    testing systems, or mobile sensor arrays. Certain aspects of NEES
    awards are risky until the plans of the consortium are developed and
    the network is implemented. The question of how the various
    equipment sites will be networked seamlessly is unclear at this
    time.

    **Recommendations:**

4.1 Emphasize research projects that examine and use lessons learned
from past activities.

4.2 Support those projects that provide insight into and address future
research requirements and priorities.

4.3 Continue to increase the size and duration of awards, in general. In
particular, increased size and duration is necessary to increase the
number of multidisciplinary proposals that require the involvement of
multiple investigators.

4.4 Encourage Program Directors to have a stronger participation in the
identification and support of emerging opportunities.

> 4.5 Provide specific guidance to improve the success rate for new
> investigators, particularly those who have recently received a
> declination.

4.6 The use of new investigators on review panels is a commendable
practice in their development process. However, an appropriate balance
must be maintained between "seasoned" engineers and scientists and the
less experienced investigators to ensure that proposals are
appropriately reviewed.

> 4.7 Continue to promote programs, in addition to the CAREER program,
> that emphasize the integration of research and education.
>
> 4.8 Provide the COV with data associated with the status and trends of
> underrepresented groups in the science, engineering and technology
> areas.
>
> 4.9 Explicitly include the characteristics of high-risk,
> multidisciplinary, and innovation in the solicitation and evaluation
> of new initiatives and projects.
>
> 4.11 Continue collaborations with international researchers, and
> encourage cross-industry collaborations.
>
> 4.10 Establish a new type of award, Mid-career Re-education Award
> program that is designed to assist the transition of engineers from
> traditional areas of investigation to non-traditional areas, such as
> biology, and theoretical physics. This program will facilitate the
> development of a multidisciplinary community of engineering
> scientists**.**

##### 

##### B. RESULTS: OUTPUTS AND OUTCOMES OF NSF INVESTMENTS

# 

**5. PEOPLE Strategic Outcome Goal: Development of a diverse,
internationally-competitive and globally-engaged workforce of
scientists, engineers, and well-prepared citizens.**

> The COV determined that the CMS programs, in the aggregate, are
> successful in achieving this Performance Goal. A summary of the
> findings for each of the abovementioned indicators, from each of the
> program areas within the CMS Division, is presented in the following
> portion of this section of the report, and was used as a basis for the
> successful rating. The CMS Division is to be commended for the
> attention and significance they have given to the REU program through
> supplements. These supplements represent an effective mechanism
> towards the development of the next generation of globally-engaged
> scientists and engineers.

a.  **Improved mathematics, science, and technology skills for U.S.
    students at the K-12 level;**

> The COV groups found minimal explicit evidence for success in
> addressing this indicator. The following are some observations and
> examples noted from three of the COV groups.
>
> The DSMSC group found, for example, project CMS 9501470 that supported
> the development of an active noise cancellation system used on display
> at the North Carolina Museum of Life and Science. This interactive
> display promoted potential noise pollution control technology for all
> visitors but particularly the legions of K-12 students who visit the
> museum as part of their science curriculum.
>
> The GHS group found the overwhelming majority of the reviewed project
> reports did not include any reference to the K-12 education program.
> However, they did find one example (CMS 9733064) that documents the
> participation of high school students in the research through the
> Polytechnic Youth in Engineering & Science (YES) Program. However,
> there was no description of the nature of the student involvement or
> the benefit to either the students or to the research for this
> project.
>
> The SSE group found that it was not possible to determine the extent
> of influence of the programs they reviewed on K-12 education. However,
> in one instance, the PI reported on a collaborative and outreach
> activity with a high school educational program.
>
> **b. Improved mathematics, science, and technology skills for citizens
> of all ages, so that they can be competitive in a technological
> society;**
>
> The COV determined that the CMS programs are successful when viewed in
> a broad sense. That is, the projects sponsored by this CMS Division,
> by their very nature, will lead to improved mathematics, science, and
> technology skills through the training and the execution of tasks
> required of the project participants.
>
> **c. A science and technology and instructional workforce that
> reflects America\'s diversity;**

The COV groups found limited success associated with addressing this
indicator. There were, however, highly successful workshops that CMS has
held annually, alternating from meetings for faculty from
underrepresented groups, and for CAREER awardees. These workshops have
been effective and have led to the inclusion of a large number of
students from under-represented groups in REU projects and CAREER
proposal submittals.

> The following is an IIS example of a research project that has been
> successful in addressing this indicator:

-   A completed program brought professionals who helped plan and manage
    reconstruction after major foreign earthquakes to a symposium to
    describe their experiences to the planning directors of major U.S.
    cities subject to earthquakes. Before the symposium occurred, both
    the Whittier Narrows and Loma Prieta earthquakes struck. Planners
    from the affected cities were also invited to the symposium both to
    share their experiences with early recovery and to learn from their
    foreign counterparts.

    This was an early attempt to bring information about post-earthquake
    reconstruction directly to the global planning community. NSF
    funding enabled the development of a training exercise for planners
    with respect to the major reconstruction decisions, problems and
    opportunities typically faced in the first year or two after an
    earthquake, including preparing a reconstruction plan and defining
    mitigation actions that could be taken ahead of time to make each of
    the tasks easier to carry out after an earthquake. The set of three
    exercises has been distributed by FEMA to state mitigation officers
    and state flood, earthquake and hurricane program managers. The
    exercises are designed for easy facilitating and are now available
    from FEMA and the states to all jurisdictions in the U.S.

    The DSMSC area reviewed two programs that they believed addressed
    this indicator.

```{=html}
<!-- -->
```
-   In project CMS 9625161 the participants were engaged in the INROADS
    outreach program for minority high school students in St. Louis.
    Also under this grant, research results were transformed into
    undergraduate and graduate laboratory demonstrations to provide
    students with "outside the textbook" experience.

-   Project CMS 9900248, provided an opportunity for women and minority
    authors to attend an international meeting.

    In the projects reviewed by the GHS group there appeared to be
    minimal reference to this indicator. However, they did find in a
    final report for project, CMS 9800291, that small groups of minority
    students were introduced to graduate education and encouraged to
    pursue graduate degrees.

> **d. Globally engaged science and engineering professionals who are
> among the best in the world;**
>
> The COV determined that the CMS programs are successful in addressing
> this indicator. Several examples from some of the programs are listed
> below.
>
> GHS examples of research projects that have been successful in meeting
> this goal include:

-   Geotechnical reconnaissance report on the effects of the
    Hyogoken-Nanbu earthquake that devastated Kobe, Japan (CMS 9520204).
    Eighteen US, Japanese, and Canadian researchers who were rapidly
    deployed in the earthquake-stricken area to prepare a document of
    substantial value to US practitioners and researchers prepared this
    report. Shortly after the earthquake, the report was available by
    web. Not only did this work represent exceptional international
    collaboration, but provided invaluable reconnaissance training for
    young researchers who have subsequently become leaders of
    reconnaissance efforts after major earthquakes in Turkey, Taiwan,
    and Washington State.

```{=html}
<!-- -->
```
-   Several of the reports documented activities between collaborators
    from several countries and international presentation of research
    findings. In CMS 9531782, collaboration with researchers in The
    Netherlands and Norway allowed full development and field testing of
    the miniature piezocone penetrometer used in the project.

-   GHS program has done an excellent job of engaging the science and
    engineering professionals throughout the world to respond to
    international crises. GHS funds the United States Strong Motion
    Program Advisory Committee (CMS 9526340), which aids USGS in
    developing the research priorities for earthquake research.

-   GHS also funds numerous workshops and symposium with global impact
    such as CMS 9872945.

> IIS examples of research projects that have been successful in
> addressing this indicator include:

-   Several projects were funded under the US-Japan Initiative to
    facilitate interchange between researchers in both countries. In one
    project (CMS 9812503) researchers undertook a joint literature
    review and developed common case studies. Researchers from Japan
    assisted US researchers in obtaining data on Japanese case studies
    and vice versa. The project has brought together two disciplines
    (earthquake and transportation engineering) and is developing a
    common vocabulary between the two fields. Transportation engineering
    has a systems perspective, while earthquake engineering is grounded
    on a strength-of-materials perspective.

-   In another project (CMS 9896104) the PI undertook five trips to
    Japan to collect considerable data through publications and meetings
    with 25 agencies and many researchers on the regional economic
    impact of the Kobe earthquake to gain perspective on how to estimate
    impact of a future catastrophic urban earthquake in the US.

> The SMME and SSE groups found that the primary mechanism used to
> address this indicator was through conference participation,
> publications in international journals, and through international
> collaborations.
>
> The SMME group did review programs that were sponsored jointly with
> other U.S. organizations (e.g. EPRI, DOE/Sandia, FHA, AFOSR, and NIST)
> that engaged the global science and engineering community. These
> programs include:

-   Tribology Review with AFOSR and ONR (e.g., Steve Danyluk, Georgia
    Tech, CMS 0123556)

-   Durability Review with AFOSR (e.g., on life of thermal barrier
    coatings, Klod Kokini, Purdue University, CMS 9908187; on structural
    polymers, Wolfgang Knauss, Caltech, CMS 9800672; and grantees review
    meeting at U.C. at Berkeley, CMS 9812757)

> **e. A public that is provided access to the processes and benefits of
> science and engineering research and education.**
>
> The CMS programs are successful with respect to addressing this
> indicator.
>
> IIS examples of research projects that have been successful in
> addressing this Indicator include:

-   A project (CMS 9705644) between a public interest group and a US
    university involved an extensive effort to disseminate information
    to the public on mitigation and recovery efforts for earthquakes.
    This was carried out through written reports, a video production
    concerned with reducing losses from earthquake, use of an internet
    site to post information on the research effort, and a 10^th^
    Anniversary conference on the Loma Prieta earthquake that was widely
    attended and was influential in drafting legislation.

```{=html}
<!-- -->
```
-   Another research effort provided much needed information to decision
    makers on developing appropriate public policies concerning
    disasters. One idea embraced by a diverse groups of researchers and
    practitioners alike is that disaster policy is not a mass political
    issue but rather one of "specialized politics" involving a policy
    network. FEMA's Project Impact and part of the Cascadia earthquake
    policy group in the Pacific Northwest reflect this notion of
    hierarchy in their disaster design strategies.

```{=html}
<!-- -->
```
-   **A major research project on "2^nd^ National Assessment of Research
    and Applications on Natural Hazards" (CMS 9541969, CMS 9544211, and
    CMS 9641063) involved over 130 national experts in hazards and
    disasters from all fields of science, engineering, policy, and
    practice. The results of this effort greatly influenced policies for
    and approach to hazard loss mitigation in FEMA's Project Impact. The
    objectives of Project Impact are to establish a national risk
    assessment, meet the need for computer-aided systems to inform local
    hazard decision-making (e.g., FEMA's HAZUS), and to assist local
    efforts to design safer communities, such as efforts underway in
    Berkeley, California and Tulsa, Oklahoma. A summary brochure of the
    assessment was distributed to every member of the U.S. Congressional
    House of Representatives. The document was subsequently used
    internationally. It provided the basis for redrafting New Zealand's
    environmental and hazards legislation to link sustainable
    development with environmental management and hazards mitigation.**

> The GHS group determined that the public is provided access to the
> benefits of this research through various forms of dissemination,
> ranging from web site postings to integration in the classroom
> curriculum to journal and conference publications. There was little
> evidence of novel methods employed to increase public access.
>
> The SSE and DSMSC groups determined that public access to the benefits
> of the programs sponsored in their area was through publications in
> technical journals. In several cases web sites were created to
> facilitate access to the results of the research and the reports
> generated from the projects. They did not find examples where
> publications or other means were used to convey the benefits of the
> research results to the general public.

6.  **IDEAS Strategic Outcome Goal: Enabling discovery across the
    frontier of science and engineering, connected to learning,
    innovation and service to society.**

> The COV determined that the CMS programs, in the aggregate, are
> successful in achieving this Performance Goal. A summary of the
> findings for each of the abovementioned indicators, from each of the
> program areas within the CMS Division, is presented in the following
> portion of this section of the report, and was used as a basis for the
> successful rating.
>
> **a. A robust and growing fundamental knowledge base that enhances
> progress in all science and engineering areas including the science of
> learning;**
>
> The GHS group determined that most of the project reports showed some
> evidence of increasing fundamental knowledge, although for many of the
> projects the primary emphasis was application of fundamental knowledge
> rather than its development or enhancement. The following is an
> example of a research project that has been successful in addressing
> this indicator:

-   A novel application of neural networks by researchers at the
    University of Illinois (CMS 0084556) has led to the development of a
    method that back-calculates the most appropriate characteristics for
    modeling soil behavior based on actual field measurements of deep
    excavation performance. This approach provides for auto-adaptive
    modeling to estimate subsequent construction behavior. The resulting
    real-time simulations allow engineer to predict ground deformation
    and instabilities in advance and take appropriate measures to avert
    life safety and economic problems.

> The IIS group found several examples that demonstrated achievements in
> the development of a fundamental knowledge base that enhances
> scientific and engineering progress. These examples include:

-   The 1987 Whittier Narrows earthquake revealed a significant gap in
    disaster research literature: the need to improve understanding of
    the impact of major natural disasters on businesses. Most of the
    existing research had focused on the economic impacts of disasters
    on households and families. The PIs were able to pursue this
    interest as well as explore factors influencing the provision of
    government loan assistance to businesses that suffered
    earthquake-related losses. With NSF funding from the National Center
    for Earthquake Engineering Research (NCEER), the PI's carried out a
    large-scale survey of businesses in Memphis/Shelby County,
    Tennessee, on business owners' perceptions of earthquake hazard, on
    their judgment on the vulnerability to and effects of lifeline
    disruptions, and on their preparedness measures. With funding from
    NCEER, they collected similar data from the Northridge earthquake.

-   In another NSF grant (CMS 9632779) the PIs investigated the
    long-term effects of disaster on business viability by collecting
    data through mail surveys of businesses (933 in Santa Cruz County,
    California and 1078 in Dade County, Florida) and face-to-face
    interviews with 53 community officials in two study areas. Overall
    these studies have produced data on nearly 5,000 businesses of all
    sizes and types, comprising the largest database on business
    disaster readiness and effects in the world.

> From the annual reports reviewed by the SMME group, they have
> identified programs that have demonstrated success with respect to
> this indicator. The following is an example of one such program:

-   John Hutchinson (Harvard University) initiated research on the
    mechanics of interfaces in 1984 with funding from the SMME program
    office. His early work in the topic applied to composite materials
    used in the aerospace industry resulted in numerous activities at
    the national level. Interactions with colleagues resulted in seminal
    contributions involving a large number of graduate students that
    became faculty members at prestigious universities. For instance,
    one of Hutchinson's students, Z. Suo, wrote his dissertation on the
    Fracture Mechanics of Interfaces. He later used his background to
    study fracture problems associated to the microelectronics industry
    and is currently investigating self-assembling dynamics of
    nanostructures with SMME support (CMS 9820713)

> The SMME program has also been aggressive in supporting fundamental
> knowledge with a high societal payoff. An example of one such program
> is:

-   Project CMS 9617452 developed a process for the conversion of SiC to
    crystalline diamond at ambient pressure. This has direct application
    to the surface science of microelectromechanical systems (MEMS). The
    entire technological field of MEMS represents an area that is
    gaining momentum with the commercialization of these devices to
    applications in communications, transportation and aerospace. An
    article, describing this work was published in *Nature* and received
    great attention from the news media.

> All of the projects reviewed under the SSE program resulted in
> research findings that contributed to the advancement of the
> fundamental knowledge base in the field of structural systems and
> control. Several appeared to have made analytical advances resulting
> in publications in reputable journals and keynote lectures at major
> international meetings. The majority of these, however, represented
> incremental advancement of science and engineering.
>
> **b. Discoveries that advance the frontiers of science, engineering,
> and technology;**
>
> There were several examples of projects at the forefront of research
> ideas in the area of DSMSC. Examples include:

-   Project CMS 9625161 transformed ideas to physical experiment of a
    method to anticipate pathological nonlinear vibrations through
    precursor sensor measurements. This new development was applied to
    the detection of cardiac arrhythmia in collaboration with Cardiac
    Pacemakers, Inc. Additional applications of the theory were made
    with the Boeing Company in the detection of abnormal behaviors of
    drilling and milling tools.

```{=html}
<!-- -->
```
-   Project CMS 9501470 developed active noise control methods in
    collaboration with NASA Langley Research Center. The advances
    associated with using piezoelectric actuators to mitigate engine
    noise levels the nacelles of airplanes can be used to provide
    increased passenger comfort and improved public health.

> None of the GHS project reports that were reviewed showed evidence of
> significant theoretical "discoveries". However, several projects
> resulted in products that advance the frontiers of science,
> engineering and technology. One such example is:

-   Project CMS 9531782 resulted in development and application of an
    electronic miniature piezocone penetrometer for a more precise and
    cost effective site characterization methodology.

> Although some other projects also resulted in marketable, publicly
> accessible hardware, most of the product development was in the area
> of software, especially numerical simulation models. For example:

-   In project CMS 9612136, a new type of effective stress model was
    developed for predicting soil deformations around a deep excavation
    in soft clay.

> SMME has several examples of breakthrough discoveries:

-   An innovative proposal supported by the SMME program is entitled
    "Investigating the Use of Flexible Wings for Uninhabited Air
    Vehicles" by P.G. Ifju, University of Florida, CMS 9941148. This
    project deals with the design of palm size airplanes, among the
    world's smallest by using flexible wings made of composite
    materials. The flight characteristics of these airplanes are similar
    to biological counterparts. The aircraft can be used in unmanned
    surveillance and reconnaissance missions.

```{=html}
<!-- -->
```
-   Project CMS 9523276, Steve Granick, University of Illinois, whose
    work resulted in several publications, one of which received 170
    citations and another 230 citations.

> SMME examples of applied research success stories include:

-   Project CMS 9814251 - Steve Danyluk (now Georgia Tech, then of UIC)
    and the Kelvin probe;

-   Project CMS 9713544, Victor Petrenko, Dartmouth, flipping the
    scientific phenomenon into a useful engineering benefit, namely
    de-icing of aircraft.

> The IIS group determined that Daniel Halpin advanced the science and
> engineering of the state of the construction industry by shifting
> research from rudimentary information technology tools, such as bar
> charting, to simulation of construction operations at both the project
> and process level. One result of the project was a game called
> CONSTRUCTO, which allowed the user to interact with a project in a
> gaming format.
>
> As previously mentioned, the majority of the programs reviewed within
> the SSE group represented incremental advancement of science and
> engineering. The portfolio did not contain high-risk, high-payoff
> projects. However, two projects, brought to our attention by the
> Program Director, can be considered to be high-risk and high-payoff.
> These include:

-   Project CMS 9802127 - Application of a microbial immobilization
    Technique in Remediation of Concrete Cracks,

-   Project CMS 9528083 - Magnetorheological Semi-Active Dampers" under
    the Active and Passive Control Initiative.

> Both of the above-mentioned projects have resulted in highly
> innovative products that have been recognized by the research and
> practicing engineering communities.
>
> **Observation**
>
> An observation from the SSE group focused on a need for encouraging
> more innovation and high-risk ideas within this program of CMS. There
> are many examples of innovative research being done elsewhere that
> could and should be supported by this program. For instance, research
> in fiber reinforced reactive powder concrete is being conducted by
> foreign industry resulting in proprietary products. To promote a more
> widespread and competitive national and worldwide implementation, such
> new materials should be developed under NSF supported research. The
> program managers are also encouraged to take advantage of the current
> initiative associated with information technology to radically advance
> the state of the science and technology of structural systems
> research.
>
> **c. Partnerships connecting discovery to innovation, learning, and
> societal advancement**
>
> Review by the GHS group uncovered several examples of very successful
> partnering among several universities, as well as partnering between
> universities and industrial collaborators. Industrial collaborators
> included construction firms, consulting engineering companies, and
> fiber producing companies. The significance and practicality of the
> research results from these collaborative efforts appear to be greater
> than those from a single investigator. In one case (CMS 9414361), the
> aquifer imaging technology developed from the sponsored research was
> applied at no cost to a village in northern Michigan where a fuel
> spill had endangered domestic wells. The program has been successful
> in the development of partnerships that connect discovery to
> innovation, learning, and societal advancement. For example, John
> Braden in project CMS 9613562 developed innovative partnerships with
> the Illinois Department of Natural Resources and the Illinois
> Groundwater Consortium to develop a alternative floodplain management
> schemes. Anne Steinemann also developed an innovative partnership with
> three universities (University of Washington, University of Georgia,
> and Florida Institute of Technology) as well as state and governmental
> environmental agencies to reduce the impact of drought hazards.
>
> One project (CMS 9526094), reported within the IIS group, involved a
> partnership with the City of Houston officials. The project conducted
> surveys and carried out studies directly related to the successful
> execution of an ambitious ten-year Greater Houston Wastewater program
> to rehabilitate old pipes and add new pipelines and facilities.
>
> The SSE group reported that there were no instances where industry was
> a significant partner in the research projects that they reviewed.
> There were examples, however, where industry had donated material and
> partial financial support to the research project. Strong synergistic
> partnerships could not be identified even in those instances.
>
> The DSMSC group observed that partnerships were more the exception and
> generally not the rule. For those projects that involved outside
> partners, it was difficult to determine whether or not these
> partnerships provided an effective link between discovery, innovation,
> learning and societal advancement.
>
> **d. Research and education processes that are synergistic.**
>
> Nearly all of the GHS project reports indicated that the research
> methodology and findings were integrated into the classroom. The GHS
> program found successful projects associated with mentoring graduate
> students for the workforce. For example:

-   Paul Mayne of Georgia Tech supported 12 graduate students over the
    duration of his NYI project (CMS 9257642). One of his graduate
    students, Dr. Susan Burns is currently a faculty member at the
    University of Virginia where she received a CAREER Award in 2000.

-   Dr. John Braden of the University of Illinois Urbana-Champaign
    supported four graduate and six undergraduate students in project
    CMS 9613562.

-   Mary Roth in project CMS 9734899 worked with 3 undergraduate
    students at Lafayette College, and one of the students, Jennifer
    Mackey, is currently pursuing a Ph.D. at the University of
    Massachusetts at Amherst.

> A project within the IIS group (CMS 963386) enabled graduate and
> undergraduate students in various disciplines (e.g. transportation and
> structural engineering, seismology, economics and regional science) to
> work together on the effects of earthquakes on urban highway
> infrastructure productivity. This project is a model of
> interdisciplinary research consistent with NSF's Integrated Graduate
> Education and Research Traineeships (IGERT) grant program.
>
> The DSMSC group believes that those research projects that employ the
> use of graduate and undergraduate students represent a basic form of
> synergistic processes in research and education, and hence they
> believe that this indicator was successfully addressed.
>
> **7. TOOLS Strategic Outcome Goal: Providing broadly accessible,
> state-of-the-art information-bases and shared research and education
> tools.**
>
> The COV determined that the CMS programs, in the aggregate, are
> successful in achieving this Performance Goal. A summary of the
> findings for each of the abovementioned indicators, from each of the
> program areas within the CMS Division, is presented in the following
> portion of this section of the report, and was used as a basis for the
> successful rating.
>
> **a. Shared use platforms, facilities, instruments, and databases that
> enable discovery;**
>
> Almost all of the projects within the GHS group resulted in a database
> of either experimental, field, or simulation data that could be shared
> with other investigators. Some of the researchers made their project
> data easily accessible through a project website. In several cases,
> there was shared use of facilities and instruments to enable project
> completion in a more efficient and cost-effective manner. Usually
> these sharing opportunities were afforded through either collaboration
> among universities or collaboration with industrial partners.
>
> Examples of research projects from the GHS group that have been
> successful in addressing this indicator include:

-   The development of computer-based models by researchers at MIT that
    accurately simulate the construction of various building systems so
    that a designer or project manager can evaluate project duration,
    cost, and worker safety (CMS 9596277). The modeling process was used
    to create a graduate course dealing with innovation in construction.
    In conjunction with the course, a database was compiled of over 300
    construction innovations that have been placed on a special course
    related web page. The graduate course was used to create a course
    for construction companies, and additional curricula were developed
    for two new undergraduate courses. The simulation technology was
    licensed to a private company for commercialization.

> The SMME program has co-sponsored instrumentation initiatives through
> MRI's and individual investigator instrumentation proposals. For
> example, award #0072423 was made to Montana Sate University for the
> purchase of instrumentation for testing and characterization of
> polymeric systems. These tools provide the means to examine long-term
> performance and durability of polymer and polymer-matrix composites,
> which are used in the aerospace, automotive and sensor/actuator
> industry. The instrumentation is also integrated with course
> developments at the graduate and undergraduate levels.
>
> The Program Directors of the SMME program were instrumental in
> launching the long- term durability of materials and structures
> initiative, \"Modeling and Accelerated Techniques.\" One of the main
> goals of the program included the development of facilities and
> databases to address durability of materials and structures.
>
> The DSMSC group observes that any new or significantly improved
> process can be viewed as a useful tool. In this framework, all reports
> and publications have introduced a new tool to the science and
> engineering community. The dissemination of these documents will
> result in new discoveries and improved productivity.
>
> **b. Shared use platforms, facilities, instruments, and databases that
> enhance the productivity and effectiveness of the science and
> engineering workforce;**
>
> On the basis of a review of their annual reports, the COV determined
> that the SMME programs have successfully addressed this indicator.
> Cost-effectiveness is probably the most significant driver in the use
> of shared resources. SMME examples of success stories with respect to
> development of tools include:

-   Several awards, CMS 9724303 to Uzi Landman (at Georgia Tech) that
    supported evolution of molecular modeling simulations over a
    ten-year period

-   Award CMS 9414610 to Ranga Komanduri (at Oklahoma State University)

-   Award CMS 0084632 to Andres Soon (at SUNY Buffalo) for the
    development of computational tools

> GHS has successfully funded projects that provide state of the art
> information that shares educational and research tools. For example,
> the Quake project (CMS 9318163) an NSF Grand Challenge Application
> funded by [NSF](http://www.nsf.gov/) and [DARPA](http://www.arpa.mil/)
> under the [HPCC/GCAG](http://www.hpcc.gov/) (High Performance
> Computing and Communication/Grand Challenge Application Grant)
> program, is a joint effort by the Department of Civil and
> Environmental Engineering and the [School of Computer
> Science](http://www.cs.cmu.edu/) at [Carnegie Mellon
> University](http://www.cmu.edu/), and seismologists at the [University
> of Southern California](http://www.usc.edu/). Their goal is to develop
> the capability for predicting, by computer simulation, the ground
> motion of large basins during strong earthquakes, and to use this
> capability to study the seismic response of the Greater Los Angeles
> Basin.
>
> One project from the IIS group (CMS 9703200) focused on integrating
> three-dimensional (3D) design models with workflow management systems
> to provide real-time performance monitoring, through visualization, of
> construction processes. The performance monitoring is used to generate
> a future path prediction that provides signals of possible failures,
> uncertainties or bottlenecks in the string of activities or tasks that
> are on schedule for completion of a project execution plan. A software
> system integrates workflow management systems with 3-D modeling and
> visualization software for performance monitoring.
>
> **c. Networking and connectivity that takes full advantage of the
> Internet and makes SMET information available to all citizens;**
>
> SMME examples of web tools include:

-   Award CMS 9619527 to Barber at the University of Michigan, and SGER
    grantee, who has developed a model to predict brake heating. This
    model is available on the Internet.

> This is an indicator to which the GHS group would like to the CMS
> Division to place more emphasis in future projects than they have in
> the past. There was little evidence of networking and connectivity for
> information transfer, other than project web site developments.
>
> **d. Information and policy analyses that contribute to the effective
> use of science and engineering resources.**
>
> One project (CMS 9526094) from the IIS group, previously mentioned in
> connection with section 6-c dealing with partnerships, assisted City
> of Houston officials to conduct studies directly related to the
> successful execution of an ambitious ten-year Greater Houston
> Wastewater program to rehabilitate old pipes and add new pipelines and
> facilities. The IIS group believes this project is a very good example
> of the effective use of engineering resources to provide significant
> information to the policy makers of the City of Houston.

Another research project from the IIS group proposed the use of third
party inspections coupled with insurance for designing public-private
partnerships. This research has application for natural hazards policy
(i.e., enforcement of building codes and encouraging mitigation
measures) as well as environmental risk management (i.e., encouraging
firms to adopt risk management plans that are required under Section
112R of the Clean Air Act Amendments). Current research under an EPA
Cooperative Agreement is attempting to implement this program on a
national basis.

This indicator does not appear to be directly applicable to many of the
GHS project reports that were reviewed. However, several of the projects
resulted in developments and findings that are relevant to environmental
policy. For example, the research completed in CMS 9617468 leads to a
better understanding of frost penetration through landfill cover liners
that may impact regulatory guidelines for landfill design and
construction.

##### 

##### C. SUPPLEMENTARY AREAS - IMPROVEMENTS, PERFORMANCE, AND THE COV PROCESS

**8. Areas of Emphasis**

CMS programs demonstrate the likelihood of strong performance in the
future on the basis of available data examined by the COV and the
current research areas slated for investment across the CMS programs.
There is a risk, however, that an environment could develop as a result
of declining budget and restricted availability of funding that will
discourage proposal submissions. CMS operates in a delicate balance of
supporting a significant number of research projects, while providing
sufficient support for each project to encourage innovative and
significant work. Further decreases in the CMS budget will jeopardize
this balance and pose a threat to the productivity and contributions of
the Division.

**9. PROGRAM AREAS NEEDING IMPROVEMENT**

There are many opportunities for CMS in developing, refining, and
expanding its current system of programs. Some of these involve
improvements in existing programs, but others entail new initiatives,
strategic thinking, and decisions regarding Program Director proposal
load and level of interaction with researchers supported by CMS.

One area of opportunity involves the current NSF environmental
initiative. The Engineering Directorate initiated a program through the
BES Division in January 2000 to emphasize high risk/high return
exploratory work in new technologies applied to the environment. Of 200
proposals received, only 14 involved research areas associated with CMS
and only four of those were funded.

It would be advantageous for CMS to play a stronger role in the
environmental area. The geo-environmental research currently supported
through the GHS program could provide a platform from which to organize
this effort. Environmental problems related to contamination of soils,
rock, and groundwater fall with the expertise of researchers supported
through GHS. Greater emphasis on these problems and encouragement of
geo-environmental researchers would be a good starting point for more
vigorous CMS involvement in environmental issues.

Many COV members believe that NSF should be a place for high risk/high
return proposals and attendant support. Several COV members noted a lack
of such proposals and funded projects. It is the committee's
understanding that SGER is often used in CMS to support high-risk
projects. The projects supported by SGER involve small grants, and
therefore may not provide enough opportunity to promote research with
worthy, but unconventional concepts and procedures.

CMS is strongly encouraged to consider a policy that supports more high
risk/high return research. Steps in the development and implementation
of such a policy include establishing a working definition of "high
risk/high return", setting aside a specific percentage of the yearly CMS
investment funds to support such proposals, and clear communication of
this policy to proposers and review panels.

It will be advantageous for CMS to originate more initiatives for
co-sponsorship with other Divisions. Several potential initiatives were
discussed, including a broad initiative centered on the theme of
"sensing the world". Such initiatives capitalize on the creative
thinking of CMS Program Directors and help establish a strong resource
base for CMS-sponsored research.

There are also opportunities for CMS to rally the Engineering
Directorate into a more visible leadership role with other collaborative
inter-directorate activities. One area of opportunity involves joint
initiatives with the SBE Directorate. Discussions have been undertaken
at NSF to emphasize social/ behavioral science and economics as a
Foundation-wide initiative in 2003. CMS has supported projects and
collaborations that emphasize interdisciplinary research between
engineers and applied social scientists, such as the Institute for Civil
Infrastructure Systems. These projects and collaborations could provide
a framework for productive interdisciplinary initiatives with SBE.

CMS would benefit from a continuing their internal strategic planning
activity focused on a vision of five years into the future. Such
planning would help identify major themes that the Division wants to
pursue. Major themes of significance for CMS include developing the
engineering and scientific basis for national and international public
policy; adapting advanced information technologies and nano- and
micro-devices to civil infrastructure and mechanical systems; and
planning and operating energy distribution systems in coordination with
civil infrastructure. The influence of NEES on the Division's earthquake
engineering research projects as well as its impact on the CMS budget
and operations after the MRE appropriation has been spent is also an
important factor to consider. Additional issues associated with planning
for NEES are discussed in Section 12.

The proposal load carried by CMS Program Directors is high. Currently,
each Program Director deals with between 100 and 150 new proposals per
year, in addition to managing a similar number of research projects in
progress. A load that is approximately half this size is appropriate,
and will be more productive. A key issue for CMS is the balance between
process and stewardship. A heavy proposal load affects the time
available for Program Directors to interact with proposers and funded
researchers. Increased stewardship of individual researchers requires a
more moderate level of processing than currently exists.

NSF has promoted broad access to information using the Internet.
Therefore, it is surprising that the annual reports of principal
investigators are not accessible to either interested researchers or the
public through the NSF web site. Because the annual reports are required
to be filed through Fast Lane, it would be easy to make these reports
generally available at the NSF web site.

**Recommendations**

1.  CMS should play a stronger role in the environmental area and
    utilize the expertise of geoenvironmental researchers supported
    through GHS in this effort.

2.  More emphasis should be placed on high risk/high return projects.

3.  More research initiatives should originate in CMS that result in
    co-sponsorship with other Divisions and directorates. In particular,
    CMS should take the lead in developing initiatives with the SBE
    Directorate.

4.  Internal strategic planning should be undertaken to address the
    question of what CMS will look like five years from now. This effort
    should include the identification of fundamental themes and outcomes
    that CMS wishes to pursue.

5.  Serious consideration should be given to reducing the proposal load
    of Program Directors to promote more stewardship of awards, rather
    than processing research proposals.

6.  The annual reports of principal investigators should be accessible
    to all interested parties at the NSF website.

7.  Require Reviewers to identify, with some degree of specificity, how
    well a proposal addresses the goals, priorities, and mission of the
    Program, Directorate, and/or NSF.

    **10. PERFORMANCE IN MEETING PROGRAM-SPECIFIC GOALS (NON-GPRA
    GOALS)\
    **

    The CMS mission is expressed in terms of two major goals to: 1)
    provide fundamental and quantitative underpinning for the
    professions of civil and mechanical engineering in application to
    mechanical systems and the built environment and 2) support the
    rapid development and deployment of technology in service to
    society.

    The Division is meeting these goals. Of special significance is the
    restructuring during FY00 that changed the organization from 12
    disciplinary programs to five research programs and NEES. This
    change resulted in a more integrated approach to research that can
    capitalize on advances in other areas of science and engineering.

    CMS has participated in various initiatives, some of which were
    developed principally in CMS while others were shared with other
    Divisions that assumed lead roles. Initiatives developed principally
    within or with substantial involvement from CMS include those
    focused on model based simulation, long-term durability of materials
    and structures, research related to the 1999 earthquakes in Turkey
    and Taiwan, US-Japan cooperative research in urban earthquake
    disaster mitigation, and advanced technologies for housing.
    Initiatives in which CMS participated in efforts developed by other
    Divisions include scalable enterprise systems, "XYZ" on a chip, and
    engineering for the environment, among others.

    The initiatives developed with strong leadership from CMS involve
    interdisciplinary research and generally emphasize either
    international or institutional cooperation. Initiatives related to
    the Turkey and Taiwan earthquakes and US-Japan cooperative research
    have stimulated research collaborations with academic, governmental,
    and industrial partners in other countries, thereby leveraging NSF
    investments with data sharing, personnel resources, and parallel
    financial investments abroad. Initiatives related to the durability
    of materials and structures, and housing have involved collaboration
    with Federal and state DOTs and HUD, respectively. These types of
    cooperative initiatives leverage NSF funds with support from other
    governmental institutions. They benefit from concept development and
    execution that are both multi-disciplinary and multi-institutional.

    A critically important initiative for CMS is NEES. As discussed in
    Section 12, this initiative is intended to extend the frontiers of
    networking and collaboration through the advanced use of information
    technologies and state-of-the-art experimental equipment distributed
    throughout the US. Lessons learned in developing NEES will have
    ramifications on the way research is conducted at NSF.

    Particularly noteworthy is the CMS policy of supporting CAREER
    awards. CMS receives more than 10% of all CAREER proposals submitted
    to NSF. The Division is commended for its success in supporting
    young researchers through the CAREER program.

    **Observation**

    There is ample evidence that CMS is meeting its non-GPRA goals
    through its restructuring, participation in research initiatives,
    and leadership of NEES. Moreover, there appears to be an enthusiasm
    and focus within the Division that will support innovative and
    meaningful activities in the future.

    **11. COV REVIEW PROCESS**

    In evaluating the outputs and outcomes of NSF investments, the COV
    used the annual reports of the principal investigators as a primary
    resource for its assessment. The annual reports, however, are not
    organized in a way that responds to the GPRA-related questions.
    There appears to be a serious disconnect here. Since the annual
    reports and the COV questions are both menu-driven by electronic
    templates, it is relatively easy in principle to coordinate the two.
    Asking principal investigators for information that explicitly
    addresses the COV performance assessment questions will make the
    evaluation process more efficient and develop a better understanding
    among researchers, NSF personnel, and COV members regarding the
    performance goals and metrics. It would be advantageous to provide
    guidance for principal investigators in addressing GPRA-related
    issues in their annual reports. Some examples of appropriately
    prepared reports could be posted on the NSF web site to help in the
    annual report preparation.

    The process for COV assessment of outputs and outcomes was for COV
    members to ask for a diverse but limited number of annual reports
    submitted within the previous three years. After review and
    discussion of these reports, the COV rates the Division's success in
    achieving outcome goals by means of specific examples from the
    reports. This process is constrained by the choice and limited
    number of reports that are reviewed. Because only a fraction of the
    reports can be reviewed within a relatively short period, the
    assessment cannot be comprehensive. Moreover, the process promotes
    an anecdotal assessment in which specific examples are cited as
    evidence of the Division's success.

    It is highly recommended that the COV set aside sufficient time to
    obtain a top-down assessment of each program by the relevant Program
    Directors. Although offered to each team, it appears that only the
    NEES team took advantage of this offer. These briefings provide the
    Program Directors with an opportunity to present a self-assessment
    of their programs in meeting GPRA goals, and an opportunity to
    provide guidance to COV members concerning the most effective manner
    the COV can assist the Division in meeting its objectives for the
    future. It would also be advisable for the Division Director to
    present a top-down briefing concerning the Division\'s response to
    the recommendations in the previous COV report. Such a briefing
    would provide a sharper focus for the COV and would allow COV
    members to make a more effective choice of annual reports to review.
    It would help the COV acquire an overview of the program and
    Division activities and to understand what the Division regards as
    its most prominent successes and shortcomings.

    A significant portion of CMS-related research is conducted by three
    earthquake engineering research centers (EERCs), these centers are
    officially part of the EEC Division and therefore not explicitly
    included in the CMS review. Nevertheless, the outputs and outcomes
    of these centers are directly related to CMS activities and receive
    significant support from CMS. It would be helpful if some summary of
    EERC activity could be provided for future COVs.

    It is very important to preserve "institutional memory" in the
    review process to promote continuity and consistency of advice from
    prior to current reviewers. In this regard, it is important for the
    COV to contain a limited number of members from the previous COV.
    The current CMS COV contains 12 members of which 3 were part of the
    previous COV. Moreover, the previous COV chair is the co-chair of
    the current COV. The current COV met on two occasions to address the
    integrity and efficiency of process and management during the first
    meeting and to address outcomes and outputs during the second
    meeting.

    The use of co-chairs has the advantage of distributing the
    responsibilities associated with leading the COV. Furthermore, the
    involvement of the previous COV chair has provided valuable
    experience in organizing the meetings and preparing the report.
    Meeting on two separate occasions has promoted a more detailed and
    comprehensive treatment of the issues, and has allowed the COV to
    interact with CMS staff and provide more detailed guidance than
    could have been achieved in one meeting. The involvement of the
    previous COV chair coupled with meeting on two separate occasions
    represents a viable model for future COV reviews.

    **Recommendations**

```{=html}
<!-- -->
```
1.  The annual report of principal investigators should provide
    information that explicitly addresses the GPRA-related questions
    presented to COVs. The importance of answering such questions should
    be stressed with principal investigators. Guidance in answering them
    should be provided by NSF.

2.  As part of the formal COV process, there should be a top-down verbal
    briefing from Program Directors about each program as well as a
    briefing from the Division Director on how the Division has
    responded to the previous COV report.

3.  A brief summary of the activities of the earthquake engineering
    research centers should be presented at future COV reviews.
    Significant CMS resources have been invested in the centers, and an
    overview of their principal accomplishments would be useful.

4.  Future COV reviews should consider as viable options the use of
    co-chairs, involvement of the previous COV chair, and scheduling of
    the COV review as two separate meetings.

5.  Clearly identify the objectives and significance of the COV process
    to all members of the COV, particularly first-time members.

6.  Stress the significance of the COV report with respect to improving
    the investment decisions and effectiveness of programs for the
    Division, the Directorate, and the NSF. In addition, indicate that
    these reports are reviewed, approved, and used by the Advisory
    Committee to carry out their responsibilities, including those
    associated with assessing the Directorate\'s performance in meeting
    GPRA-related goals.

7.  Provide a top-down assessment of the Division\'s response to the
    recommendations presented in the previous COV report.

**12. NETWORK FOR EARTHQUAKE ENGINEERING SIMULATION (NEES)**

> The NEES program is a unique effort to develop a new model for
> collaborative research based on modern (and not yet developed)
> telecommunication technologies. The objective is to create a network
> of equipment sites where numerous researchers will be able to perform
> experiments in a collaborative environment. A repository for
> earthquake engineering related data will also be created.
>
> As envisioned and specified in NSF announcements, this type of program
> has never been undertaken. Existing models for the concept and
> integrated activities of NEES do not currently exist. The program
> therefore is to a significant extent experimental. Lessons learned in
> developing NEES will have ramifications on the way research is
> conducted throughout NSF. This program should receive attention,
> resources, and advice from the Engineering Directorate. It should be a
> priority for the Engineering Directorate to document the experience
> gained and process that evolves in organizing and operating NEES so
> that the appropriate knowledge can be used in other Divisions and
> directorates.
>
> The goals of NEES are ambitious. They involve not just the creation of
> networked resources, but a change in research and collaborative
> culture. The principal challenge of NEES, therefore, is not so much
> the development of experimental facilities and associated networking,
> but the engagement of the earthquake engineering community in the
> adoption of a vision and coordination strategy to transform its
> research and educational procedures.
>
> Everyone, who knows of NEES, has a different vision of how it might
> develop. For this program to succeed, NSF program officers must
> provide sufficient guidance for the program while giving the
> researchers a great deal of latitude to stray from non-traditional
> research approaches. Appropriate monitoring is needed from NSF to
> ensure that each of the teams is making progress. This includes,
> initially, those institutions acquiring equipment, and later, those
> institutions participating in the collaborative research efforts.
>
> The organizational chart for NEES includes a Management and Oversight
> Board that is linked directly to the NEES Program Director. This
> board, however, consists of personnel from various Divisions and
> directorates within NSF. It is equally important to obtain the advice
> and guidance of earthquake engineering and information technology
> experts external to NSF. CMS should consider forming an independent
> Scientific Advisory Committee or Expert Panel to provide technical as
> well as community-building advice to CMS for this complex project.
>
> It is recognized that NEES will involve a Consortium, comprised of
> earthquake engineering community members, who will be tasked with
> providing leadership and developing consensus participation. The
> Consortium, however, will be funded through a cooperative agreement
> with NSF and will itself need to be assessed by an external,
> independent group. The traditional CMS approach to independent review
> is through external reviewers, typically gathered as panels on an \"as
> needed\" basis. Because NEES is nontraditional, broad, and will be
> organized and operated over a long time period, some continuing
> commitment to and familiarity with the program will be highly
> advantageous. Given the complexity of NEES, it will be important for
> CMS to work with external, independent advisors who are able to
> acquire an "institutional memory" and therefore function in an
> integrated way that reflects the integration NSF seeks from NEES.
>
> Program Directors will need experts to assist them in the review and
> evaluation of the system integration decisions and directions. Special
> emphasis should be given to data acquisition capabilities, computer
> networking, and advanced information technologies that support
> geographically separate, but synchronous research activity. One
> example of an operational "collaboratory" is the Great Lakes Regional
> Center for AIDS Research (CFAR), which engaged social scientists in
> assessing the needs and modalities of its system. Experts drawn from
> such functional collaborations could provide valuable guidance, based
> on experience, for CMS Program Directors.
>
> The NEES program should provide the means for the entire earthquake
> engineering community to participate it its programs. This approach
> will result in the largest gain by the scientific community, as well
> as our society. It is critically important to select the appropriate
> group for the Consortium, and monitor its interaction with the
> community of researchers. In the beginning, extensive evaluation of
> the program should be performed to ensure that the Consortium and
> System Integrator are developing a network that is useful to a wide
> variety of earthquake engineering researchers. There must be a
> feedback mechanism in place within the Consortium to ensure that all
> researchers can be a part of this.
>
> There must be mechanisms in place to ensure that the participants
> follow through on their commitments. One of the most challenging
> aspects of a large collaborative project is ensuring that each group
> delivers. There are some mechanisms to do this (e.g., requiring NSF to
> authorize large equipment purchases, ensuring that each group
> purchases compatible equipment and charges a fair price for use of the
> equipment, etc). Continued vigilance, however, is of critical
> importance. Every participating group must meet its deadlines.
> Researchers, who cannot meet reasonable deadlines, will delay not only
> their work, but also the work of everyone involved in single
> experiments as well as other projects waiting to use the facility.
>
> Investigators should have the opportunity to present and defend their
> proposals to review panels, either face-to-face or via conference
> calls. In the second phase of the equipment funding, this approach
> should be considered. If educational impact is expected to be an
> emphasis of the NEES program, the Program Directors should make sure
> this is conveyed clearly to the Consortium. Although this program
> addresses the second NSF criterion on "broader impacts" in several
> ways, it is easy to imagine educational efforts being lost in this
> widespread national research effort. Educational efforts should be
> integrated into the research when possible.
>
> Although there are no research outcomes of NEES at this early stage in
> its development, there will have been sufficient time for outcomes
> when the next COV review occurs three years from now. Performance
> measures should be developed that will be effective in gauging the
> success of the program at that time. Characteristics related to
> performance that should be considered include research quality and
> relevance, accessibility, interaction with the community, quality of
> service, capacity and speed of information exchange, and tools of
> operation. For NEES to be successful, it must develop operational
> tools for data searching and graphical display that are easy to learn
> and use, and therefore broadly utilized. An independent expert panel,
> as discussed above, could help with the identification and development
> of the appropriate metrics.
>
> The impact of NEES will be broad and profound. The substantial
> experimental and computer-networking infrastructure created by NEES
> will require significant maintenance costs after the NEES MRE
> appropriation has been spent. Moreover, researchers working with NEES
> facilities will need to support research projects that are not covered
> by the MRE budget. In recent years, the CMS budget has been declining.
> In a flat or declining budgetary environment, it is especially
> important to plan for the future. CMS needs to develop a vision for
> its earthquake engineering research program that encompasses the
> influence of NEES. The target years for this vision and planning
> scenario are 2005-2010. This vision should also include an assessment
> of how NEES will relate to and affect the projects and budgets within
> the five other programs of the Division.

**Recommendations**

1.  Serious consideration should be given to the formation of an
    independent expert panel to advise CMS on issues related to NEES.
    Special emphasis in the panel should be given to experienced members
    of the earthquake engineering community and experts in data
    acquisition, computer networking, and advanced information
    technologies that support geographically separate, but synchronous
    research.

2.  Vigorous initial monitoring of the NEES Consortium should be pursued
    to ensure that there is proper interaction with the System
    Integrator and community of researchers.

3.  There should be continued vigilance on the part of Program Directors
    to ensure that NEES participants follow through with their
    commitments. Timely completion of work and compliance with deadlines
    must be stressed.

4.  Investigators in the second round of equipment proposals should be
    given the opportunity to defend their proposals to review panels,
    either face-to-face or via conference phone calls and video
    conferencing.

5.  The educational objectives of NEES should be conveyed clearly to the
    Consortium, and performance measures for education should be
    established to gauge successful outcomes.

6.  CMS performance measures for NEES should be developed so that the
    next COV has clear guidance on how to evaluate the program outcomes.
    An independent expert panel would be able to help in the formulation
    of appropriate metrics.

7.  CMS strategic planning should focus on the impact of NEES on the
    Division's resources and activities after the MRE allocation for the
    project has been spent. The target years for this planning are
    2005-2010. The planning should address the resources needed to
    maintain the network and equipment sites, the influence of NEES on
    other earthquake engineering research projects, and NEES impact on
    the resources for the other five Division programs.