April 14, 2004

Kristina Johnson, Dean

Pratt School of Engineering

305 Teer Building

Box 90271

Duke University

Durham, NC 27708

**RE: Committee of Visitors Report for the Division of Civil and
Mechanical Systems (CMS)**

Dear Dr. Johnson:

Enclosed, please find a revised Committee of Visitors (COV) report for
the Division of Civil and Mechanical Systems (CMS) in the Directorate of
Engineering at the National Science Foundation (NSF) and cover letter,
dated 14 April 2004.

This report should replace the one dated 7 April 2004. The initial
report does not benefit from the input and clarifications of some COV
members, which were received after it was mailed to you. The enclosed
report properly reflects the views of all COV members and should be
received and used as the final COV report for CMS.

We regret any inconvenience associated with this change. We are pleased,
however, to submit a document that embodies the consensus assessment of
the COV and provides a solid foundation for CMS evaluation and future
directions.

Sincerely,

T. D. O'Rourke J. E. Bernard

Committee Chair Committee Co-Chair

Thomas R. Briggs Professor of Engineering Anson Marston Distinguished
Professor

Cornell University of Engineering

Iowa State University

CC Galip Ulsoy and John Brighton

April 14, 2004

Kristina Johnson, Dean

Pratt School of Engineering

305 Teer Building

Box 90271

Duke University

Durham, NC 27708

**RE: Committee of Visitors Report for the Division of Civil and
Mechanical Systems (CMS)**

Dear Dr. Johnson:

Attached please find the Committee of Visitors (COV) report for the
Division of Civil and Mechanical Systems (CMS) in the Directorate of
Engineering at the National Science Foundation (NSF).

The report is based on the COV visit that took place on March 22-24,
2004 at NSF. The visit began with oral briefings by the Division
Director of CMS and the Program Directors. The Committee then examined
over 112 randomly selected jackets, reviewed proposal actions and
funding statistics, and discussed in closed and open meetings issues of
importance to CMS.

CMS is home for five research programs: Dynamic System Modeling, Sensing
and Control, Solid Mechanics and Materials Engineering, Structural
Systems and Engineering, Geotechnical and GeoHazards Engineering, and
Infrastructure and Information Systems. CMS also manages one Major
Research Equipment and Facilities Construction project: the George E.
Brown, Jr. Network for Earthquake Engineering Simulation. The COV
developed a separate section of its report to address NEES, which is
located after Part B in the attached document.

Overall, the COV finds that the CMS Division is doing a very good job in
managing its programs. The Division has been successful in helping the
Foundation to achieve desirable outcomes in its investments in people,
ideas, and tools. In the spirit of continuous improvement, the COV
identified several areas for future attention.

On behalf of the COV, we would like to thank the Division Director,
Galip Ulsoy, and the Program Directors: Perumalsamy N. Balaguru, Ken P.
Chong, Yip-Wah Chung, Jesus M. De La Garza, Richard J. Fragaszy, Jorn
Larsen-Basse, Shih-Chi Liu, Steven L. McCabe, Vilas Mujumdar, Joy M.
Pauschke, Juan M. Pestana-Nascimento, Masayoshi Tomizuka, and Dennis
Wenger. We also appreciate and thank the CMS staff. All CMS personnel
were exceptionally helpful and responsive.

Please feel free to contact us if you have questions concerning the
report.

Sincerely,

T. D. O'Rourke J. E. Bernard

Committee Chair Committee Co-Chair

Thomas R. Briggs Professor of Engineering Anson Marston Distinguished
Professor

Cornell University of Engineering

Iowa State University

CC Galip Ulsoy and John Brighton

**Report**

Committee of Visitors

Civil and Mechanical Systems Division

Directorate of Engineering

National Science Foundation

March 22, 23, and 24

Submitted to

Dr. Kristina M. Johnson

Chair

Engineering Advisory Committee

National Science Foundation

######### Committee Members

> Tom O'Rourke, (Chair), Cornell University
>
> Jim Bernard, (Co-chair), Iowa State University
>
> Harry Asada, Massachusetts Institute of Technology
>
> Ted Belytschko, Northwestern University
>
> Shirley Dyke, Washington University
>
> Silvia Ferrari, Duke University
>
> Wilfred Iwan, California Institute of Technology
>
> Marshall Jones, GE Global Research
>
> Edward Kavazanjian, Jr., University of Southern California
>
> Peter May, University of Washington
>
> Sue McNeil, University of Illinois, Chicago
>
> Horace Moo-Young, Lehigh University
>
> Glaucio Paulino, University of Illinois, Urbana
>
> Jocelyn Seng, Owens Corning and Air Force Research Laboratory
>
> Sharon Wood, University of Texas at Austin

**Executive Summary**

The Committee of Visitors of the Civil and Mechanical Systems (CMS)
Division met on March 22-24, 2004 to review programs for fiscal years
2001, 2002, and 2003. The charge to the committee included a review of
the integrity, efficacy and quality of the processes for proposal
actions and monitoring active projects; the quality and significance of
the results of the Division's programmatic investments; the degree to
which the award process supports the long-range goals and core
strategies of NSF; the divisions balance, priorities, and future
directions; and other issues the COV determines to be relevant.

The visit began with oral briefings by the Division Director of CMS and
the Program Directors in the Division. The Committee then examined 112
randomly selected jackets (including 56 awards and 56 declinations),
plus additional jackets that were provided as requested by COV members
during the review. The Committee reviewed proposal actions and funding
statistics, and discussed issues of importance to CMS in closed and open
meetings.

The COV finds the Division to be effective in assuring the integrity and
achieving efficiency in its program processes and management. Proposals
selected for funding are of high quality. In spite of continuing
increases in the number of proposals handled, average dwell time has
decreased to less than six months. It is an average 5.4 months, which is
an excellent record.

The documentation in the jackets is very good. CMS uses the panel review
process, supplemented with mail reviews. This process has been
implemented effectively and fairly, and a good distribution of reviewers
has been achieved in terms of geographic location, gender, and minority
representation. Likewise, the CMS portfolio of funded projects has an
appropriate distribution in terms of geography, gender, and minority
representation.

The use of the broader impacts criterion improved over the three-year
period of COV evaluation. The reviewers now appear to be cognizant of
the importance of broader impacts and use the criterion in their
assessments. The interpretation of the meaning of broad impact varies
significantly among the panels. It is therefore desirable to seek a more
consistent understanding and application of the criterion in future
panel reviews.

In general, it was difficult to assess the expertise and qualifications
of reviewers on the basis of the information provided in the jackets.
The COV recommends that reviewers be asked to provide short biographical
sketches, and that this information be included in the jackets.

The COV judges that CMS has been successful in meeting the outcome goals
in people, ideas, and tools. Specific examples illustrating the
Division's success in each of these areas are given in the report.

The Program Directors are commended for supporting first time
researchers. Approximately 30% of CMS funding has been directed to first
time researchers, thus providing the entrance and experience base for
those seeking careers with a strong component of research. Especially
noteworthy is CMS support of CAREER awards, which constitute about 50%
of the funding for first time researches, or 15% of the research
portfolio.

The COV notes that about half of CMS funding is pre-committed to
research initiatives and other mandated projects, or "fenced". Combined
with budget reductions, the net result is that the success rate for
proposals within the CMS core competencies may fall to less than 10% for
FY 2004. We advise carefully monitoring the ratio of fenced funds to
total funds to ensure enough funds remain available for flexible use. We
recommend that a proper balance be maintained between fenced initiatives
and the funding of core competencies.

To meet the challenge of increasing numbers of proposals, the COV
recommends that additional staff be assigned to CMS at both the PD and
support staff levels. Additional funds are also sorely needed to support
the many worthy projects that are proposed, but unable to be funded. The
COV recognizes significant increases in funding may not be available in
the near term. Therefore, it may be necessary to deal with increasing
proposal loads under the assumption of relatively flat funding. Options
include, but are not limited to, restricting the number of proposals
from a single PI and readjusting the levels of support provided for
various activities.

The George E. Brown, Jr. Network for Earthquake Engineering Research
(NEES) should be a top priority at the division, directorate, and upper
management levels of NSF. This project provides the opportunity to
explore the use of the cyberinfrastructure in its application to
geographically distributed experimental facilities for cost-effective
investments in large scale experimentation through shared-use facilities
and experiments and more efficient utilization of major research
equipment. NEES also provides unique opportunities with respect to
database management and retrieval, advanced computational modeling, and
linkage with the research, academic, industrial, and K-12 communities.
It involves not only significant technical challenges, but entails
social and cultural challenges as members of the civil engineering and
computer science communities work together at an unprecedented level of
collaboration. The potential payoff is very high. Much can be learned
and applied from NEES that is relevant to future projects at NSF. It is
in the interest of all to ensure the success of NEES.

Large-scale research programs such as NEES place a heavy burden on NSF
professional and support staff. It is vital that PDs have adequate
resources to perform their work effectively. In particular, they should
receive the necessary travel assistance to visit equipment and research
sites on a regular basis, and to maintain close contact with key
individuals within the research and user communities.

It appears that resources are not sufficient within CMS and the
Engineering Directorate to realize the full potential of NEES.
Furthermore, funds will be reallocated from other programs at the
division and directorate levels just to support NEES with a resource
base significantly below its capabilities. The COV does not believe that
NEES should drain resources from other programs in CMS and the
Engineering Directorate. Because of the importance of this project for
NSF, the COV strongly recommends exploring with NSF upper management
ways to obtain additional funds for NEES as a supplement to the
Engineering Directorate budget.

There is an excellent opportunity for CMS to take a continuing lead role
in developing and directing NSF research in the area of homeland
security. The Division has distinguished itself to date by undertaking a
major research effort on the effects of September 11, 2001, which
culminated in a special publication and press conference dealing with
the research results. The COV recommends that CMS pursue research on
homeland security issues and continue to pursue leadership position in
this area.

It would be advantageous to have a mechanism for division-level
strategic advice. The COV is not well suited to this mission. Its charge
is to assess program-level technical and managerial matters pertaining
to program decisions. Moreover, the advice provided by the Engineering
Advisory Committee to the Engineering Directorate is generally at a
strategic level that addresses cross-cutting divisional issues and areas
of broader NSF policy. The COV therefore recommends that consideration
be given to establishing a division-level advisory committee composed of
external experts from universities, industry, and government. It is
likely that this recommendation applies to other divisions as well.

**PART A. INTEGRITY AND EFFICIENCY OF THE PROGRAM'S PROCESSES AND
MANAGEMENT**

Briefly discuss and provide comments for *each* relevant aspect of the
program\'s review process and management. Comments should be based on a
review of proposal actions (awards, declinations, and withdrawals) that
were *completed within the past three fiscal years*. Provide comments
for *each* program being reviewed and for those questions that are
relevant to the program under review. Quantitative information may be
required for some questions. Constructive comments noting areas in need
of improvement are encouraged. Please do not take time to answer
questions if they do not apply to the program.

A.1 Questions about the quality and effectiveness of the program's use
of merit review procedures. Provide comments in the space below the
question. Discuss areas of concern in the space provided.

+-----------------------------------------------------------+----------+
| **QUALITY AND EFFECTIVENESS OF MERIT REVIEW PROCEDURES**  | **YES,   |
|                                                           | NO,**    |
|                                                           |          |
|                                                           | **DATA   |
|                                                           | NOT      |
|                                                           | AV       |
|                                                           | AILABLE, |
|                                                           | or NOT   |
|                                                           | APPL     |
|                                                           | ICABLE** |
+-----------------------------------------------------------+----------+
| Is the review mechanism appropriate? (panels, ad hoc      | **YES**  |
| reviews, site visits)                                     |          |
|                                                           |          |
| Comments:                                                 |          |
|                                                           |          |
| The principal review mechanism for CMS is the review      |          |
| panel. Proposal evaluations were made either by a panel   |          |
| or a combination of panel and mail reviews. The panels    |          |
| were generally well organized.                            |          |
|                                                           |          |
| The COV finds that the panel review process is            |          |
| appropriate. It allows for a relatively rapid evaluation  |          |
| of proposals, which is an important attribute given the   |          |
| large numbers of proposals processed by CMS. A panel also |          |
| provides the opportunity to bring junior members of the   |          |
| research community into contact with established          |          |
| researchers and practitioners. Panels promote networking  |          |
| among junior and senior researchers. They help junior     |          |
| members of the community obtain insights about the        |          |
| preparation of proposals and review standards.            |          |
|                                                           |          |
| The panel review process, however, can be biased against  |          |
| unusual proposals and lesser-known researchers. It is     |          |
| therefore imperative for PDs to continue to encourage     |          |
| reviewers to remain receptive to new ideas and            |          |
| technologies.                                             |          |
+-----------------------------------------------------------+----------+
| Is the review process efficient and effective?            | **YES**  |
|                                                           |          |
| Comments:                                                 |          |
|                                                           |          |
| The panel review process is efficient. As stated above,   |          |
| the process is effective in handling a large number of    |          |
| proposals in a fair and objective way. The COV notes that |          |
| the panel process may be the only way to deal effectively |          |
| with the large numbers of proposals that must be          |          |
| processed yearly by CMS.                                  |          |
|                                                           |          |
| Panel reviews are biased to seek consensus, which may     |          |
| overlook or de-emphasize high-risk, high-potential payoff |          |
| projects. It is important, therefore, for PDs to remind   |          |
| reviewers of the need to consider high-risk projects in   |          |
| their assessments. At the same time, it is important for  |          |
| CMS to develop a balanced portfolio of projects with a    |          |
| certain proportion of high-risk investigations, as        |          |
| discussed in forthcoming sections of the report.          |          |
+-----------------------------------------------------------+----------+
| Are reviews consistent with priorities and criteria       | **YES**  |
| stated in the program's solicitations, announcements, and |          |
| guidelines?                                               |          |
|                                                           |          |
| Comments:                                                 |          |
|                                                           |          |
| Reviewers were sensitive and responsive to special        |          |
| solicitations.                                            |          |
+-----------------------------------------------------------+----------+
| Do the individual reviews (either mail or panel) provide  | ## YES   |
| sufficient information for the principal investigator(s)  |          |
| to understand the basis for the reviewer's                |          |
| recommendation?                                           |          |
|                                                           |          |
| Comments:                                                 |          |
|                                                           |          |
| Most of the individual reviews provide sufficient         |          |
| information for the principal investigators to understand |          |
| the basis for the reviewers' recommendations. Many of     |          |
| them are solid and informative reviews. In some cases,    |          |
| the reviews consisted of just a few lines, but these      |          |
| types of review appeared to be a relatively small         |          |
| percentage of the total that were examined by the COV.    |          |
+-----------------------------------------------------------+----------+
| Do the panel summaries provide sufficient information for | ## YES   |
| the principal investigator(s) to understand the basis for |          |
| the panel recommendation?                                 |          |
|                                                           |          |
| Comments:                                                 |          |
|                                                           |          |
| In general, the panel summaries provided a reasonable     |          |
| overview of the discussions during the review, addressing |          |
| both the technical merits and the broader impacts of the  |          |
| research.                                                 |          |
+-----------------------------------------------------------+----------+
| Is the documentation for recommendations complete, and    | **YES**  |
| does the program officer provide sufficient information   |          |
| and justification for her/his recommendation?             |          |
|                                                           |          |
| Comments:                                                 |          |
|                                                           |          |
| In general the documentation is very good. For most       |          |
| cases, the information appears to be complete and         |          |
| consistent with the panel information.                    |          |
+-----------------------------------------------------------+----------+
| Is the time to decision appropriate?                      | **YES**  |
|                                                           |          |
| Comments                                                  |          |
|                                                           |          |
| The COV finds that CMS has performed in an exemplary      |          |
| manner with respect to rapid and effective processing of  |          |
| proposals under demanding conditions that involve         |          |
| consistent, yearly increases in the numbers of proposals. |          |
| CMS should be commended. Average dwell time has decreased |          |
| to less than six months. The overall average dwell time   |          |
| is 5.4 months, according to statistics provided by CMS.   |          |
| This is a very good record, and especially noteworthy     |          |
| given the increasing number of proposals that have been   |          |
| processed in recent years.                                |          |
+-----------------------------------------------------------+----------+
| Discuss issues identified by the COV concerning the       |          |
| quality and effectiveness of the program's use of merit   |          |
| review procedures:                                        |          |
|                                                           |          |
| In most cases, the merit review procedures are very       |          |
| successful at identifying high quality proposals.         |          |
+-----------------------------------------------------------+----------+

A.2 Questions concerning the implementation of the NSF Merit Review
Criteria (intellectual merit and broader impacts) by reviewers and
program officers. Provide comments in the space below the question.
Discuss issues or concerns in the space provided.

+---------------------------------------------------------+------------+
| **IMPLEMENTATION OF NSF MERIT REVIEW CRITERIA**         | **YES,     |
|                                                         | NO,**      |
|                                                         |            |
|                                                         | **DATA NOT |
|                                                         | AVAILABLE, |
|                                                         | or NOT     |
|                                                         | AP         |
|                                                         | PLICABLE** |
+---------------------------------------------------------+------------+
| Have the individual reviews (either mail or panel)      | ## YES     |
| addressed whether the proposal contributes to both      |            |
| merit review criteria?                                  |            |
|                                                         |            |
| Comments:                                               |            |
|                                                         |            |
| The use of the broader impacts criterion improved over  |            |
| the three-year period of this evaluation. The reviewers |            |
| now appear to be cognizant of the importance of broader |            |
| impacts, but they do not necessarily weigh that         |            |
| criterion as heavily as intellectual merit. Since a     |            |
| project needs to be grounded in an intellectually       |            |
| meritorious activity for it to have appropriate broader |            |
| impacts, a bias towards the intellectual merit          |            |
| criterion is both understandable and a sensible basis   |            |
| for decisions.                                          |            |
|                                                         |            |
| The COV notes that the interpretation of the meaning of |            |
| broad impact seems to vary among panels. This again is  |            |
| understandable because each panel has a unique          |            |
| composition, and its decision-making will vary          |            |
| according to composition. Nevertheless, it is desirable |            |
| to seek a more consistent understanding and application |            |
| of the broader impacts criterion. This consistency      |            |
| should be achievable with continued vigilance on the    |            |
| part of the PDs in providing examples and guiding       |            |
| panels in applying the broader impacts criterion.       |            |
+---------------------------------------------------------+------------+
| Have the panel summary reviews addressed whether the    | ## YES     |
| proposal contributes to both merit review criteria?     |            |
|                                                         |            |
| Comments:                                               |            |
|                                                         |            |
| The panel summaries now typically address both merit    |            |
| review criteria.                                        |            |
+---------------------------------------------------------+------------+
| Have the *review analyses* (Form 7s) addressed whether  | ## YES     |
| the proposal contributes to both merit review criteria? |            |
|                                                         |            |
| Comments:                                               |            |
|                                                         |            |
| Review analyses now typically address both merit review |            |
| criteria.                                               |            |
+---------------------------------------------------------+------------+
| Discuss any issues or concerns the COV has identified   |            |
| with respect to NSF's merit review system.              |            |
+---------------------------------------------------------+------------+

A.3 Questions concerning the selection of reviewers. Provide comments in
the space below the question. Discuss areas of concern in the space
provided.

+---------------------------------------------------------+------------+
| Selection of Reviewers                                  | YES , NO,  |
|                                                         |            |
|                                                         | DATA NOT   |
|                                                         | AVAILABLE, |
|                                                         |            |
|                                                         | or NOT     |
|                                                         | APPLICABLE |
+---------------------------------------------------------+------------+
| Did the program make use of an adequate number of       | YES        |
| reviewers for a balanced review?                        |            |
|                                                         |            |
| Comments:                                               |            |
|                                                         |            |
| In most cases, four panelists reviewed each proposal,   |            |
| and the entire panel participated in the discussions.   |            |
| Given the very high proposal load, this appears to be a |            |
| reasonable number.                                      |            |
+---------------------------------------------------------+------------+
| Did the program make use of reviewers having            | Not Enough |
| appropriate expertise and/or qualifications?            | Data       |
|                                                         | Available  |
| Comments:                                               |            |
|                                                         |            |
| It appears that the program directors have considered   |            |
| technical expertise when assigning panelists. However,  |            |
| there is not sufficient information in the jackets for  |            |
| the COV to confirm this observation. A good remedial    |            |
| step would be to ask the reviewers to provide a short   |            |
| biographical sketch with their reviews. The             |            |
| biographical information would be included in the       |            |
| jackets.                                                |            |
+---------------------------------------------------------+------------+
| Did the program make appropriate use of reviewers to    | YES        |
| reflect balance among characteristics such as           |            |
| geography, type of institution, and underrepresented    |            |
| groups?                                                 |            |
|                                                         |            |
| Comments:                                               |            |
|                                                         |            |
| The geographic distribution was good. CMS did a         |            |
| laudable job of implementing the 2001 COV               |            |
| recommendations for involving non-academic reviewers.   |            |
| Most panels included women and underrepresented         |            |
| minorities.                                             |            |
+---------------------------------------------------------+------------+
| Did the program recognize and resolve conflicts of      | YES        |
| interest when appropriate?                              |            |
|                                                         |            |
| Comments:                                               |            |
|                                                         |            |
| Conflicts of interest were noted in the program         |            |
| folders, and panelists with conflicts did not           |            |
| participate in the discussions.                         |            |
+---------------------------------------------------------+------------+
| Discuss any concerns identified that are relevant to    |            |
| selection of reviewers.                                 |            |
+---------------------------------------------------------+------------+

A.4 Questions concerning the resulting portfolio of awards under review.
Provide comments in the space below the question. Discuss areas of
concern in the space provided.

+------------------------------------------------------+---------------+
| **RESULTING PORTFOLIO OF AWARDS**                    | **A           |
|                                                      | PPROPRIATE,** |
|                                                      |               |
|                                                      | **NOT         |
|                                                      | A             |
|                                                      | PPROPRIATE,** |
|                                                      |               |
|                                                      | **OR DATA NOT |
|                                                      | AVAILABLE**   |
+------------------------------------------------------+---------------+
| Overall quality of the research and/or education     | **            |
| projects supported by the program.                   | Appropriate** |
|                                                      |               |
| Comments:                                            |               |
|                                                      |               |
| The overall quality of the research and education    |               |
| projects is very high.                               |               |
+------------------------------------------------------+---------------+
| Are awards appropriate in size and duration for the  | **            |
| scope of the projects?                               | Appropriate** |
|                                                      |               |
| Comments:                                            |               |
|                                                      |               |
| The size of the typical award is relatively modest,  |               |
| and in most cases insufficient to support            |               |
| experimental research at a significant scale. Given  |               |
| budget constraints and the number of high quality    |               |
| proposals, the size of awards appears to be a        |               |
| reasonable compromise.                               |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | **            |
| balance of:                                          | Appropriate** |
|                                                      |               |
| -   High Risk Proposals?                             |               |
|                                                      |               |
| Comments:                                            |               |
|                                                      |               |
| Typically in CMS, high risk proposals are funded     |               |
| through the SGER process. The funding limit on SEGRs |               |
| has been increased recently to \$200,000, which      |               |
| appears adequate to engage in some substantive       |               |
| high-risk projects. However, the increasing numbers  |               |
| of proposals in the face of flat, and even           |               |
| declining, budgets means that the opportunities to   |               |
| utilize the SEGRs is, in turn, constrained.          |               |
|                                                      |               |
| It should be noted that CMS manages NEES, which is   |               |
| as a high-risk, high-opportunity program. The risk   |               |
| is balanced by the pioneering aspects of the         |               |
| program, which involve experimental sites            |               |
| interactively engaged and accessible by worldwide    |               |
| users though high performance Internet. The system   |               |
| will allow co-synchronous physical and computational |               |
| simulation at geographically dispersed sites. Its    |               |
| success depends on intense collaboration among civil |               |
| and mechanical engineers and IT specialists.         |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | **            |
| balance of:                                          | Appropriate** |
|                                                      |               |
| -   Multidisciplinary Proposals?                     |               |
|                                                      |               |
| Comments:                                            |               |
|                                                      |               |
| CMS is involved in interdisciplinary research with   |               |
| HUD, USDOT, SANDIA, and DOE. It also works with BES, |               |
| SBE (DRMS) and engages in initiatives in             |               |
| biocomplexity, sensors, information technology, and  |               |
| nanoscience.                                         |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | **Not         |
| balance of:                                          | Enough**      |
|                                                      |               |
| -   Innovative Proposals?                            | **Data        |
|                                                      | Available**   |
| Comments:                                            |               |
|                                                      |               |
| The 112 jackets sampled by the COV did not provide   |               |
| sufficient information to draw a conclusion.         |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | **            |
| balance of:                                          | Appropriate** |
|                                                      |               |
| -   Funding for centers, groups and awards to        |               |
|     individuals?                                     |               |
|                                                      |               |
| Comments:                                            |               |
|                                                      |               |
| The CMS portfolio is a reasonable mix of awards to   |               |
| groups and individuals.                              |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | **            |
| balance of:                                          | Appropriate** |
|                                                      |               |
| -   Awards to new investigators?                     |               |
|                                                      |               |
| Comments:                                            |               |
|                                                      |               |
| The CMS portfolio has an appropriate balance of      |               |
| awards to new investigators, who receive 30% of the  |               |
| awards. This includes substantial support for CAREER |               |
| proposals (approximately 50% of the support for new  |               |
| investigators, or 15% of the CMS portfolio).         |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | **            |
| balance of:                                          | Appropriate** |
|                                                      |               |
| -   Geographical distribution of Principal           |               |
|     Investigators?                                   |               |
|                                                      |               |
| Comments:                                            |               |
|                                                      |               |
| The CMS portfolio has an appropriate geographical    |               |
| distribution of PIs.                                 |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | **            |
| balance of:                                          | Appropriate** |
|                                                      |               |
| -   Institutional types?                             |               |
|                                                      |               |
| Comments:                                            |               |
|                                                      |               |
| The program portfolio appears to have an appropriate |               |
| balance of institutions. Consistent with NSF         |               |
| guidelines, 40% of MRIs were awarded to              |               |
| non-PhD-granting institutions. There is evidence of  |               |
| awards to EPSCoR, PUI and minority institutions.     |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | **            |
| balance of:                                          | Appropriate** |
|                                                      |               |
| -   Projects that integrate research and education?  |               |
|                                                      |               |
| Comments:                                            |               |
|                                                      |               |
| Some proposals have a very strong educational        |               |
| component integrated with research. As a result of   |               |
| the broader impact criterion, recently successful    |               |
| proposals tend to have a fairly significant          |               |
| educational component. Supplements are often used to |               |
| fund educational activities.                         |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | **Not Enough  |
| balance:                                             | Data          |
|                                                      | Available**   |
| -   Across disciplines and sub-disciplines of the    |               |
|     activity and of emerging opportunities?          |               |
|                                                      |               |
| Comments:                                            |               |
|                                                      |               |
| The 112 jackets sampled by the COV did not provide   |               |
| sufficient information to draw a conclusion.         |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have appropriate          | **            |
| participation of underrepresented groups?            | Appropriate** |
|                                                      |               |
| Comments:                                            |               |
|                                                      |               |
| The program has appropriately funded                 |               |
| underrepresented groups.                             |               |
+------------------------------------------------------+---------------+
| Is the program relevant to national priorities,      | **            |
| agency mission, relevant fields and other customer   | Appropriate** |
| needs? Include citations of relevant external        |               |
| reports.                                             |               |
|                                                      |               |
| Comments:                                            |               |
|                                                      |               |
| The CMS program is highly relevant to national       |               |
| priorities, agency mission, and relevant fields,     |               |
| During the period of 2001-2003, CMS has supported    |               |
| the development and publication of more that 50      |               |
| national reports and national/international          |               |
| proceedings to identify and focus on new research    |               |
| directions. Some reports were developed by the       |               |
| National Research Council. Others were developed by  |               |
| external organizations that assembled diverse,       |               |
| representative groups to summarize priorities for    |               |
| various research and practicing communities.         |               |
|                                                      |               |
| Ample evidence of relevant external studies and      |               |
| guidance documents were provided by CMS when         |               |
| requested by the COV. The list of reports cited      |               |
| below are examples of the types of studies engaged   |               |
| by CMS to guide and focus its selection of projects  |               |
| relevant to national priorities and relevant fields: |               |
|                                                      |               |
| -   National Research Council. *Preventing           |               |
|     Earthquake Disasters - A Research Agenda for the |               |
|     Network for Earthquake Engineering Simulation    |               |
|     (NEES)*. Committee to Develop a Long-Term        |               |
|     Research Agenda for the Network for Earthquake   |               |
|     Engineering Simulation. Board on Infrastructure  |               |
|     and the Constructed Environment Division on      |               |
|     Engineering and Physical Sciences. The National  |               |
|     Academies Press. Washington, DC, 2003            |               |
|                                                      |               |
| -   NSF National Sensors Workshop Report (Aug 2001   |               |
|     Lake Tahoe)                                      |               |
|                                                      |               |
| -   Proceedings US-Korea Workshop on Smart           |               |
|     Infrastructural Systems, 23-24 August 2002,      |               |
|     Busan, Korea.                                    |               |
|                                                      |               |
| -   Control for Fuel Cells Workshop, University of   |               |
|     California, Irvine, CA, April 3-4, 2003.         |               |
|                                                      |               |
| -   NSF Workshop on Future Directions in Nano-Scale  |               |
|     Systems, Dynamics and Control, Metin Sitti       |               |
|     (ed.), NanoRobotics Laboratory, Department of    |               |
|     Mechanical Engineering and the Robotics          |               |
|     Institute, Carnegie Mellon University,           |               |
|     Pittsburgh, PA.                                  |               |
|                                                      |               |
| -   R. Shoureshi (ed.), *Proceedings of the ESF-NSF  |               |
|     Workshop on Advancing Technological Frontiers    |               |
|     for Feasibility of Ageless Structures,* in       |               |
|     press.                                           |               |
|                                                      |               |
| -   Mita and B.F. Spencer, Jr. (eds.), *Proceedings  |               |
|     of the International Workshop on Advanced        |               |
|     Sensors, Structural Health Monitoring, and Smart |               |
|     Structures*.                                     |               |
+------------------------------------------------------+---------------+
| -   Digital Infrastructures: Enabling Civil and      |               |
|     Environmental Systems through Information        |               |
|     Technology, edited by Rae Zimmerman, ICIS, New   |               |
|     York University and Thomas A. Horan, Claremont   |               |
|     Graduate University, Routledge, September 2004.  |               |
|                                                      |               |
| -   *Beyond September 11th: An Account of            |               |
|     Post-Disaster Research*, (2003). Book,           |               |
|     Published, Editor(s): Natural Hazards Research & |               |
|     Applications Information Center, Public Entity   |               |
|     Risk Institute, and Institute for Civil          |               |
|     Infrastructure Systems, Collection: University   |               |
|     of Colorado-Boulder Special Publications Series, |               |
|     Bibliography: Special Publication #39. Boulder,  |               |
|     CO: University of Colorado                       |               |
|                                                      |               |
| -   Responding to the Unexpected, NSF Workshop, New  |               |
|     York, N.Y. February 27 -- March 1, 2002.         |               |
|                                                      |               |
| -   NSF Summer Institute on Nanomechanics and        |               |
|     Materials, Northwestern University, Evanston,    |               |
|     IL, August 11-15, 2003.                          |               |
|                                                      |               |
| -   NSF Workshop on Nano- and Micro-Mechanics of     |               |
|     Solids for Emerging Science and Technology,      |               |
|     Kyung-Suk Kim, Brown University                  |               |
|                                                      |               |
| -   *Future Research Directions in Solid Mechanics*, |               |
|     Am. Academy of Mechanics, PI: Francis Moon, July |               |
|     2003.                                            |               |
|                                                      |               |
| -   EERI. *Securing Society Against Catastrophic     |               |
|     Earthquake Losses - A Research and Outreach Plan |               |
|     in Earthquake Engineering*. Earthquake           |               |
|     Engineering Research Institute. Richmond, CA,    |               |
|     2003.                                            |               |
+------------------------------------------------------+---------------+
| Discuss any concerns identified that are relevant to |               |
| the quality of the projects or the balance of the    |               |
| portfolio.                                           |               |
|                                                      |               |
| The quality of the projects is excellent, but the    |               |
| available resources do not allow funding of many     |               |
| worthy projects. As CMS has engaged in an increasing |               |
| number of initiatives in recent years, there has     |               |
| been an increasing larger proportion of its          |               |
| portfolio invested in fenced projects, i.e. projects |               |
| that are pre-committed in PD budgets to initiatives  |               |
| outside CMS programs. The percentage of fenced       |               |
| projects for 2005 is projected to exceed 50% of the  |               |
| CMS portfolio.                                       |               |
|                                                      |               |
| The COV is very concerned that the increasing number |               |
| of fenced projects threatens the balance in the CMS  |               |
| research portfolio. The COV believes that attention  |               |
| should be directed to emerging and multidisciplinary |               |
| areas, but not at the expense of being overly        |               |
| restrictive of core programs. It is important to     |               |
| recognize that innovation is not just synonymous     |               |
| with emerging technology, but often is a key         |               |
| attribute of research in core disciplines. It is     |               |
| critical, therefore, to support research that        |               |
| stimulates innovation and breakthroughs in core      |               |
| programs. The COV recommends that a proper balance   |               |
| be maintained between fenced initiatives and the     |               |
| funding of core competencies.                        |               |
+------------------------------------------------------+---------------+

A.5 Management of the program under review. Please comment on:

+-----------------------------------------------------------------------+
| Management of the program.                                            |
|                                                                       |
| Comments:                                                             |
|                                                                       |
| The current PDs are doing an excellent job of managing the program.   |
| Proposals are reviewed in a timely manner. The review panels include  |
| a variety of junior and senior researchers. Given the funding         |
| constraints, the portfolio of projects is well balanced.              |
+-----------------------------------------------------------------------+
| Responsiveness of the program to emerging research and education      |
| trends.                                                               |
|                                                                       |
| Comments:                                                             |
|                                                                       |
| The PDs have been responsive to NSF initiatives. There has been       |
| participation in NSF priority areas, such as ADVANCE (Advance         |
| Participation of Women in Science and Engineering), ITR (Information  |
| Technology Research), BE (Bio-complexity in the Environment), NSE     |
| (Nanoscale Science and Technology), and HSD (Human and Social         |
| Dynamics). CMS is engaged in Engineering Directorate initiatives,     |
| such as Sensors and Sensor Networks. CMS initiatives and priorities   |
| include NEES Research.                                                |
+-----------------------------------------------------------------------+
| Program planning and prioritization process (internal and external)   |
| that guided the development of the portfolio u under review.          |
|                                                                       |
| Comments:                                                             |
|                                                                       |
| The PDs have responded well to internal and external pressures in     |
| establishing priorities and developing a portfolio of projects.       |
+-----------------------------------------------------------------------+
| Discuss any concerns identified that are relevant to the management   |
| of the program.                                                       |
|                                                                       |
| The COV notes that the number of proposals that CMS handles continues |
| to rise in an environment where the number the Division currently     |
| handles threatens to overload the system. Furthermore, there has been |
| a significant decrease in the unfenced funds available for            |
| unsolicited proposals. Success rates for unsolicited proposals may    |
| fall to less than 10% in CMS for FY 2004. This decrease in funding    |
| rate threatens the sustainability of the unsolicited proposal         |
| program. Moreover, this is threatening on two additional counts. On   |
| the one hand, as the success rates go down and the chances of funding |
| diminish, a lowering of morale and consequent loss of influence in    |
| the technical community may result. On the other hand, PDs are losing |
| time to handle the very important non-administrative part of their    |
| duties (e.g., developing new research directions, mentoring young     |
| faculty, participating in workshops and conferences, and              |
| collaboration with other research agencies) again potentially leading |
| to lowering of morale and loss of influence. The obvious remedy,      |
| increased resources for both personnel and for research funding, may  |
| not be available in the near term. This situation calls for           |
| consideration of alternate measures -- taking steps to limit the      |
| number of proposals to be considered and giving very high priority to |
| adding program officers and support staff.                            |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| **B.1 [OUTCOME GOAL for PEOPLE]{.underline}: Developing "a diverse,   |
| competitive and globally engaged workforce of scientists, engineers,  |
| technologists and well-prepared citizens."**                          |
|                                                                       |
| Comments:                                                             |
|                                                                       |
| > CMS is attentive to the NSF strategic goal of developing "a         |
| > diverse, competitive, and globally engaged workforce of scientists, |
| > engineers, technologists and well-prepared citizens." The emphasis  |
| > on this goal is articulated by: (1) the use of CAREER, REU, and     |
| > other NSF-funding mechanisms to promote education and diversity,    |
| > (2) funding of innovative activities that specifically support this |
| > objective, (3) program solicitation and panel review with           |
| > appropriate emphasis on the people, and (4) encouragement of a      |
| > diverse and globally involved CMS PD workforce. As a whole, our     |
| > impression is that the "people" objective is taken very seriously   |
| > within CMS and the record with respect to it is very strong. As     |
| > important is the observation that the CMS culture of PDs and review |
| > processes has institutionalized attention to these considerations   |
| > providing some assurance that the strong record will continue.      |
| >                                                                     |
| > The traditional mechanisms for promoting development of new         |
| > researchers at NSF are the CAREER, REU, and IGERT mechanisms. CMS   |
| > maintains strong involvement with the first two, with a total of 66 |
| > CAREER awards during FY 2001-2003, comprising some 15% of the CMS   |
| > total budget. Twenty percent of these went to women PIs and 13.6 %  |
| > went to PIs from under-represented minority groups. CMS PDs have    |
| > promoted the REU mechanism among the relevant research communities  |
| > with substantial PI involvement (284 projects in FY 2001 -- FY 2003 |
| > had REUs ). There has been much less involvement with the IGERT     |
| > program for which PDs from CMS have virtually no control.           |
| >                                                                     |
| > CMS has funded projects aimed at developing a diverse research      |
| > community, involving new researchers, and extending international   |
| > connections. Two examples stand out:                                |
|                                                                       |
| -   A series of "Workshops for Faculty Diversity in Civil and         |
|     Mechanical Engineering" have been funded over the past eight      |
|     years (most recently, CMS-0305673, PI-Mattei, University of New   |
|     Orleans) to bring together faculty and doctoral students from     |
|     under-represented groups to encourage a more diverse research     |
|     community.                                                        |
|                                                                       |
| -   A more focused activity aimed at encouraging new researchers to   |
|     enter the field of hazards research is the "Enabling Project"     |
|     (CMS-0218413, PI-Burby, University of North Carolina). This       |
|     project involves mentoring of 16 assistant professors, many of    |
|     whom are outside the hazards field, about research opportunities  |
|     and development of research proposals.                            |
|                                                                       |
| > More generally, CMS has been effective in promoting international   |
| > collaboration through funding international workshops over the past |
| > three years and in funding projects that specifically involve such  |
| > collaboration. One example is the funding of a series of US-Japan   |
| > workshops (most recently CMS 0244365, PI-Tubbesing, Earthquake      |
| > Engineering Research Institute) that involve sharing research       |
| > findings among American and Japanese researchers.                   |
| >                                                                     |
| > An example of support for training in emerging technologies is the  |
| > program of Professor Liu at Northwestern University (CMS 0318907).  |
| > This program supports two classes each summer that train            |
| > approximately 100 professors, post-docs, graduate students, and     |
| > industry participants in the fundamentals of nanotechnology.        |
| >                                                                     |
| > The PDs have done an excellent job integrating educational          |
| > objectives into their research programs through funding for PUIs    |
| > and K-12 initiatives. For example, support of the MRI proposal of   |
| > Ruggles (CMS 0215809) will greatly impact a PUI, and improve        |
| > undergraduate research.                                             |
| >                                                                     |
| > NEES is intended to provide an environment for education and        |
| > training not only for graduate students, but at the undergraduate   |
| > and K-12 levels. The groundwork has been laid for a strong training |
| > and education component that will be integrated with future         |
| > research projects.                                                  |
|                                                                       |
| Emphasis on "people" issues was evident in our examination of         |
| external reviews, panel deliberations, and PD recommendations. The    |
| positive outcomes of this emphasis are evident from statistics about  |
| CMS proposals and funding for underrepresented groups relative to the |
| Engineering Directorate as a whole. For FY 2001-2003, an average of   |
| 10.7 % of proposals were from females (11.7 % ENG wide), 4.9 % from   |
| Hispanic PI's (3.5 % ENG wide), and 2.2 % from Afro-American PI's     |
| (2.9 % ENG wide). These percentages can be put in context with        |
| recognition that in 2002, 9.2 % of engineering faculty were women,    |
| 3.1 % were Hispanic, and 2 % were Afro-American (American Society for |
| Engineering Education, 2002).                                         |
|                                                                       |
| > Compared to the CMS-wide success ratio of 14.3% for FY 2001-2003,   |
| > the success ratio for female PIs was 19.7%, Hispanic PIs 18%, and   |
| > Afro-American PIs 20.7 %. These success ratios demonstrate a        |
| > commitment within the review process to support a diverse research  |
| > community.                                                          |
| >                                                                     |
| > The issues of diversity also carry over to the composition of the   |
| > CMS PD workforce. Diversity here is important, given the visibility |
| > of PDs in their respective research communities. For FY 2001-2003,  |
| > more than one-quarter of the PD staff were women.                   |
| >                                                                     |
| > Three areas for potential improvement related to the people         |
| > objective were discussed within the COV and with PDs. One involves  |
| > more attention to the IGERT mechanism for encouraging the           |
| > development of innovative, multi-disciplinary graduate training     |
| > programs of relevance to CMS. The obvious constraint here is the    |
| > limited ability of CMS personnel to affect the flow and review of   |
| > IGERT proposals. A second area is the extent to which CMS is        |
| > stressing international collaboration and connections, particularly |
| > given the realization of the strength of research in CMS-related    |
| > programs in a number of other countries. While CMS does support     |
| > international collaboration, it would be advantageous for such      |
| > funding to increase in the future. The third is the tendency of PIs |
| > to present boilerplate responses to the integration of research and |
| > education. While it is common for individual proposals to fund and  |
| > involve graduate and undergraduate (through REU) students,          |
| > proposals could also explore less conventional activities that      |
| > involve high school teachers, access to K-12 students, and          |
| > collaboration with science museums. PIs should be strongly          |
| > encouraged to review representative activities at the NSF website   |
| > (<http://www.nsf.gov/pubs/2002/nsf022/bicexamples.pdf>) that        |
| > satisfy the requirements of broader impacts.                        |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| **B.2 [OUTCOME GOAL for IDEAS]{.underline}: Enabling "discovery       |
| across the frontier of science and engineering, connected to          |
| learning, innovation, and service to society."**                      |
|                                                                       |
| Comments:                                                             |
|                                                                       |
| # All CMS programs                                                    |
| are making a strong contribution in the way of new ideas and the impl |
| ementation of those ideas. Examples are given below for each program. |
|                                                                       |
| #                                                                     |
|                                                                       |
| # Solid Mechanics and Materials Engineering (SMME)                    |
|                                                                       |
| #                                                                     |
|                                                                       |
| -   CMS0239130 (Shaofan Li, UC-Berkeley) CAREER proposal exploring    |
|     the linkage between nanoscale and microscale behavior (using the  |
|     quasi-continuum model and the handshake approach) with meshfree   |
|     computational techniques, which is quite creative and             |
|     interesting.                                                      |
|                                                                       |
| -   CMS0115954 (Glaucio Paulino, UIUC) "for a major study of the      |
|     mechanics of functionally graded materials" (extracted from the   |
|     Annual Report for FY2001, Mechanics and Materials Engineering     |
|     Program, CMS Division).                                           |
|                                                                       |
| -   CMS0338438 (Pipes, Akron) SGER to investigate carbon nanotube     |
|     polyimide microsensors by micro raman spectroscopy. High risk     |
|     research funding                                                  |
|                                                                       |
| -   CMS 0324601 (Weaver, Alabama Tuscaloosa) to study dendrimer-based |
|     nanocomposites for tribological applications. Novel contribution  |
|     to nano-research.                                                 |
|                                                                       |
| -   CMS0335390 (Wong, North Dakota) SGER to investigate               |
|     cost-effective substitutes for carbon nanotubes and other         |
|     nanocomposites. High-risk seed grant on nano-science.             |
|                                                                       |
| -   CMS0324461 (Subhash, Michigan Tech) GOALI to investigate          |
|     ultrafine grained and nanostructured ceramics (influence of       |
|     processing, grain size and strain rate on fracture). Example of   |
|     integration of academia and industry contributions.               |
|                                                                       |
| -   CMS0310596 (Shephard, RPI) for Multiscale Systems Engineering.    |
|     This is a research center on novel developments of multiscale and |
|     nanosystems involving advanced modeling concepts and data         |
|     structures.                                                       |
|                                                                       |
| #                                                                     |
|                                                                       |
| # Infrastructure and Information Systems (ISS)                        |
|                                                                       |
| The ISS programs have produced both short and long term impacts       |
| through discovery and innovation. The SGER mechanism for funding      |
| projects related to the World Trade Center collapse was critical to   |
| developing concepts related to blast effects and fire, understanding  |
| the economic impacts of catastrophic events, and collecting data on   |
| building performance.                                                 |
|                                                                       |
| Other innovative ideas relate to the use of sensors and visualization |
| in construction. Laser scanning and embedded sensors are used on      |
| construction sites to identify construction defects (Akinci --CMS     |
| 0121549). This includes inspection planning, object recognition       |
| algorithms, design representation and formalisms for defect           |
| detection. Similarly, visualization using animated 3D worlds, is used |
| for project control (Martinez -- CMS 0113890). The 3D animation       |
| language allows people to navigate and immerse themselves in a        |
| virtual world.                                                        |
|                                                                       |
| # Dynamic System Modeling, Sensing and Control (DSMSC)                |
|                                                                       |
| CMS-0201386 (T. Iwasaki, University of Virginia) "Dynamic Interaction |
| between Mechanical Rectifier Biological Oscillator." This proposal    |
| tackled a very tough problem in control theory from a biology         |
| standpoint. Integration of adaptive control and robust control has    |
| been an open question, which is difficult to address based on the     |
| control theory currently available. The PI proposed innovative        |
| control architecture inspired by biological systems, i.e. animal      |
| locomotion. In animal locomotion, robust control and adaptive control |
| co-exist, and are highly integrated. This research has already made a |
| significant breakthrough, and the PI has formed an interdisciplinary  |
| research group based on the NSF funding.                              |
|                                                                       |
| **[Geotechnical and Geohazards Engineering (GGE)]{.underline}**       |
|                                                                       |
| Innovative projects with high potential include:                      |
|                                                                       |
| CMS-0092447- (Schaefer) This proposal investigated a novel technology |
| for landslide assessment using laser swath mapping.                   |
|                                                                       |
| CMS-0234009 This action supported fundamental research in the         |
| modeling of soil liquefaction within a general thermodynamic          |
| framework. The project has the potential for a true breakthrough by   |
| providing a comprehensive basis for modeling the transition of        |
| liquefied soil from a solid-like to fluid-like state during an        |
| earthquake.                                                           |
|                                                                       |
| ### **[Structural Systems and Engineering (SSE)]{.underline}**        |
|                                                                       |
| CMS-0220006 -- Study of Tornado-Induced Wind Loads on Built           |
| Structures                                                            |
|                                                                       |
| The project combines engineers with expertise in fluid-structure      |
| interaction and scientists with expertise in meteorology, and         |
| provides a mechanism for integrating computational and experimental   |
| simulations of structural systems with data measured during severe    |
| storms. Each year, more than 800 tornados occur in the US, which      |
| cause on average 80 deaths, 1500 injuries, and \$850 million in       |
| damage. The research is focused on quantifying tornadic wind loads on |
| structures, which is the first step to improving the safety of the    |
| infrastructure.                                                       |
|                                                                       |
| CMS -0215808 -- Acquisition of Instrumentation for Development of     |
| Reliable Technologies for Harsh Environments                          |
|                                                                       |
| As part of this Major Research Instrumentation award, facilities were |
| developed to evaluate the reliability of new technologies for use in  |
| harsh environments. The award complements a NASA Grand Challenge      |
| Award at the University of Idaho, and will investigate the use of     |
| MEMS in space. MEMS are expected to revolutionize instruments used    |
| for aerospace applications due to their small size and weight.        |
+-----------------------------------------------------------------------+
| **B.3 [OUTCOME GOAL for TOOLS:]{.underline} Providing "broadly        |
| accessible, state-of-the-art S&E facilities, tools and other          |
| infrastructure that enable discovery, learning and innovation."**     |
|                                                                       |
| > **TOOLS Strategic Outcome Goal: Providing broadly accessible,       |
| > state-of-the-art information-bases and shared research and          |
| > education tools**                                                   |
|                                                                       |
| Within the limits of their funding, CMS programs are quite successful |
| in providing tools for the research community. Significant examples   |
| are discussed under the following headings:                           |
|                                                                       |
| > **a. George E. Brown, Jr Network for Earthquake Engineering         |
| > Simulation (NEES)**                                                 |
|                                                                       |
| NEES is a major research equipment and facilities construction        |
| project, to be completed in October 2004. It creates a national,      |
| geographically-distributed infrastructure for earthquake engineering  |
| research and education, in which equipment sites and users and linked |
| through high performance Internet and specially designed network      |
| software packages. This system promises to revolutionize the field of |
| earthquake engineering. It also provides an example of how to develop |
| collaboratories in other divisions of the Engineering Directorate as  |
| well as other directorates of NSF. NEES is discussed in greater       |
| detail in a separate section of this document.                        |
|                                                                       |
| > **b. Major Research Initiative (MRI)**                              |
| >                                                                     |
| > Approximately 10 of 30 proposals for the acquisition of research    |
| > equipment were funded by CMS in 2003. This is a significantly       |
| > higher success ratio than CMS overall, which was 15% in 2003. The   |
| > procedure for allocating funding is quite involved. Proposals in    |
| > the million dollar range are in a separate category and the final   |
| > selection from a pool is made by the overall NSF administration. Of |
| > the smaller proposals, 40% of the funded proposals are aimed at     |
| > colleges that focus on undergraduate education. In 2003 CMS         |
| > contributed \$350K to help fund a directional random wave generator |
| > for Texas A&M (total cost = \$800 K).                               |
|                                                                       |
| Many of the successful proposals address both the need of the         |
| equipment for research and its potential for undergraduate or K-12    |
| education.                                                            |
|                                                                       |
| Examples of outstanding contributions to tools for information        |
| services and data exchange and for technology in widespread use to    |
| improve the quality of life include:                                  |
|                                                                       |
| -   Natural Hazards Research and Applications Information Center at   |
|     the University of Boulder Colorado (CMS 0354424, CMS 0301156).    |
|     The center serves as a national and international clearinghouse   |
|     for research data on natural disasters, related technological     |
|     events, and programs to reduce damage. It provides not just data, |
|     but also serves as the place "to go to" (a nexus and facilitator) |
|     for data and resources related to natural hazards. The center     |
|     awards travel grants to social scientists for "quick response"    |
|     studies, disseminates both web-based and hard copy information,   |
|     and provides dissertation grants. The center also conducts an     |
|     annual workshop. Its bimonthly newsletter is mailed to over       |
|     15,000 subscribers. Some of the hazards are non-traditional, such |
|     as debris collection from the space shuttle Columbia. The Center  |
|     also supports REU students.                                       |
|                                                                       |
| -   Magnetic Resonance Imagery (CMS-8008629) PI: P.C. Lauterbur, SUNY |
|     at Stoneybrook. Research on this technology was funded in the     |
|     1980s to refine nuclear magnetic resonance into a routine         |
|     diagnostic technique. In 2002, over 20,000 such machines were     |
|     used by health care facilities worldwide, with more than 60       |
|     million examinations using the technology. For his contributions, |
|     Prof. Lauterbur won the 2003 Nobel Prize in Medicine.             |
+-----------------------------------------------------------------------+
| **B.4 [OUTCOME GOAL for ORGANIZATIONAL EXCELLENCE]{.underline}:       |
| Providing "an agile, innovative organization that fulfills its        |
| mission through leadership in state-of-the-art business practices."** |
|                                                                       |
| E-jackets are used extensively in CMS, with electronic handling of    |
| almost the entire proposal-servicing process. From the point of view  |
| of the supply side, Fastlane is now a well-accepted practice.         |
|                                                                       |
| As described elsewhere, CMS has decreased the average proposal dwell  |
| time to 5.4 months. From 2000 to 2003 the number of proposals         |
| processed by CMS has increased by nearly 50%.                         |
+-----------------------------------------------------------------------+

**George E. Brown, Jr. Network for Earthquake Engineering Simulation**

(**NEES)**

**Overview**

NEES is a unique program within CMS. Great care has been taken to meet
the special needs of this large integrated program. The bottom line is
that the program is working very effectively and is achieving its
objectives.

**Review Process**

A two-stage review process has been developed for the equipment site
proposals that could be used as a model for other large-scale projects.
Initial reviews are conducted by Technical Panels. Based on the results
of this review, a set of proposals is selected for stage-two review. In
the second stage, selected PIs are invited to respond to the results of
the initial review and to present their proposals to a Portfolio Panel.
This Portfolio Panel makes final recommendations on the rating and
ranking of proposals. After the second stage review, the Program
Directors make the final funding decisions.

This two-stage review process has worked well. The examined jackets
showed that the review decisions were fair and well documented. Major
decisions were supported by the panel recommendations. This approach is
recommended for other large-scale programs within CMS. In particular,
the model should be considered for large NEESR projects.

#### Management of Projects

NSF management has had to deal with difficult personnel issues. The
handling of these problems has been performed with integrity and
professionalism. To deal with such issues, it has been necessary that
the PDs take an active involvement in the program.

**Challenges**

There will always be challenges for the management of a major research
program such as NEES, including:

-   Judging and documenting "close calls" between very similar high
    priority proposals.

-   Identification of review panelists that provide both breadth and
    depth of understanding.

-   Providing a clear charge to panels while still allowing flexibility
    in proposal evaluation.

-   Providing clear instructions to second stage PIs

-   Integrating the results of different panels that might have
    different standards for review of proposals.

-   Developing appropriate metrics for the initial evaluation of
    proposals and for monitoring objectives, especially when these
    objectives evolve over time.

Program Directors have dealt with these challenges. Future projects of
similar scale and scope (e.g. cyberinfrastructure efforts) will require
concerted PD involvement in collaboration with the research community.

# Broader Impacts

Regarding the broader impacts, NEES has the potential to revolutionize
the field of earthquake engineering. Thus, training, education and
outreach need to be integrated into NEES activities. The equipment sites
have proposed varying levels of education and training programs. It is
too early to determine the effectiveness of the use of sites as an
instrument for training, so it is not yet clear whether each institution
is currently meeting its obligations regarding the implementation of
training plans.  Quarterly reports refer to activities, such as
attending workshops to reach out to potential users. However, activities
discussed in proposals may not be discussed further in annual reports.
This raises questions about the commitment to training and education.
The NEES Consortium operational team will have personnel responsible for
this area, but the source of the funding for these activities will have
to be outside of CMS. To achieve the full promise of NEES, additional
funding could be directed toward these activities, possibly through
establishing partnerships with other programs at NSF and elsewhere that
focus on education and outreach.

**Other Issues**

External Advisory Boards can play an important positive role in
large-scale research projects. They need to be viewed by PIs as an
asset, not a liability.

Large-scale research programs such as NEES place a heavy burden on NSF
professional and support staff. It is vital that PDs have adequate
resources to perform their work effectively. In particular, they should
receive the necessary travel assistance to visit equipment and research
sites on a regular basis, and to maintain close contact with key
individuals within the research and user communities.

NEES involves not only significant technical challenges, but entails
significant social and cultural challenges as members of the civil
engineering and computer science communities work together at an
unprecedented level of collaboration. Research involving the social
science community about how NEES addresses these challenges and the
broader management of engineering technology is appropriate for NSF
funding as part of the NEES program.

**Outcomes**

**People.** The education of the next generation of earthquake
engineers to utilize effectively the tools provided by NEES is an
important outcome. The Consortium Development team is developing a plan
for these activities and the equipment sites are currently developing
training materials.

**Ideas.** The NEES program is one of the first to generate a
cyberinfrastructure for a particular discipline. This asset and the
lessons learned throughout this program will be very useful to other
communities of researchers who are or will be involved in similar
efforts.

 

**Tools.** A major NEES outcome will be the establishment of a
collection of equipment sites, connected by high-speed internet with the
possibility of remote use and collaboration through telepresence and
centralized data repository. This effort will result in a highly
innovative virtual tool with capabilities beyond anything that has been
established to date. The Consortium Development is on schedule, and the
equipment sites are expected to meet the deadlines set forth. The System
Integration effort faces many challenges. It is making progress with
strong involvement and guidance from the PDs and Associate Directors for
the program.

##### 

##### PART C. OTHER TOPICS

C.1 Please comment on any program areas in need of improvement or gaps
(if any) within program areas.

It would be advantageous to have a mechanism for division-level
strategic advice. The COV is not well suited to this mission. Its charge
is to assess program-level technical and managerial matters pertaining
to program decisions. Moreover, the review and advice provided by the
Engineering Advisory Committee to the Engineering Directorate are
generally at a strategic level that addresses cross-cutting divisional
issues and areas of broader NSF policy. Hence, there appears to be a gap
and corresponding opportunity with respect to division-level guidance.
The COV therefore recommends that consideration be given to establishing
a division-level advisory committee composed of external experts from
universities, industry, and government. It is likely that this
recommendation applies to other divisions as well.

C.2 Please provide comments as appropriate on the program's performance
in meeting program-specific goals and objectives that are not covered by
the above questions.

The COV notes that use of "nuggets" is important in conveying the
successes of NSF-sponsored research to Congress and other stakeholders.
Considering the importance of nuggets, it is advisable for PDs and PIs
to improve the ways of collecting and editing them so that their clarity
and quality are enhanced. We recommend that guidance and examples be
provided to PIs so they can explain the significance of their research
and its relevance with respect to the public.

**C.3 Please identify agency-wide issues that should be addressed by NSF
to help improve the program\'s performance.**

The George E. Brown, Jr. Network for Earthquake Engineering Research
(NEES) should be a top priority at the division, directorate, and
central management levels of NSF. This project provides the opportunity
to explore the use of the cyberinfrastructure in its application to
geographically distributed experimental facilities for cost-effective
investments in large scale experimentation through shared-use facilities
and experiments and more efficient utilization of major research
equipment. NEES also provides unique opportunities with respect to
database management and retrieval, advanced computational modeling, and
linkage with the research, academic, industrial, and K-12 communities.
It involves not only significant technical challenges, but entails
significant social and cultural challenges as members of the civil
engineering and computer science communities work together at an
unprecedented level of collaboration. The potential payoff is very high.
Much can be learned and applied from NEES that is relevant to future
projects at NSF. It is in the interest of all to ensure the success of
NEES.

Currently, support of NEES requires that financial resources be
reallocated from other programs within CMS and other divisions within
the Engineering Directorate. Operations, maintenance, and management
(OM&M) of NEES require approximately \$20 million/yr. Approximately \$9
million/yr is being invested by NSF in research projects that use NEES,
which is less than half the OM&M cost. An NRC study, which was
commissioned to provide guidance on NEES, recommended that \$40
million/yr would be an appropriate level of investment in NEES research
projects (National Research Council, 2003). The first solicitation for
research projects using NEES (NEESR) generated 115 proposals. It is
estimated that the research budget only provides for about a 5% success
rate for these proposals.

It appears that resources are not sufficient within CMS and the
Engineering Directorate to realize the full potential of NEES.
Furthermore, funds will be reallocated from other programs at the
division and directorate levels just to support NEES with a resource
base significantly below its capabilities. The COV does not believe that
NEES should drain resources from other programs in CMS and the
Engineering Directorate. Because of the importance of this project for
NSF, we strongly recommend exploring with upper level NSF management
ways to obtain additional funds for NEES as a supplement to the
Engineering Directorate budget.

We note that about half of CMS funding is pre-committed to research
initiatives and other mandated projects, or fenced. We appreciate that
this is a complicated topic subject to pressures from many
constituencies. We advise carefully monitoring the ratio of fenced funds
to total funds with a view toward being sure enough funds remain
available for flexible use. We recommend that a proper balance be
maintained between fenced initiatives and the funding of core
competencies.

C.4 Please provide comments on any other issues the COV feels are
relevant.

There is an excellent opportunity for CMS to take a continuing lead role
in developing and directing NSF research in the area of homeland
security. The Division has distinguished itself to date by undertaking a
major research effort on the effects of September 11, 2001, which
culminated in a special publication and press conference dealing with
the research results. It is the COV understanding that NSF is currently
in discussion with the Department of Homeland Security (DHS) regarding
joint research activities. We recommend that CMS pursue research on
homeland security issues and renew its leadership position in this area.

CMS is facing a growing challenge with the increasing number of
proposals that are submitted for review by division PDs. This situation
creates significant pressures. With the funding for the programs
remaining static (or even declining), success rates are diminishing. The
success rate for CMS is currently 15%, the lowest in the Engineering
Directorate, and approximately one third lower that the average NSF
success rate. If the number of proposals continues to increase, the
success rate will continue to go down. Current funding levels for
unsolicited proposals is such that the success rate within CMS for them
in FY2004 is likely to be less than 10%. At the same time, the proposal
load per PD (not including NEES PDs) has increased dramatically from
approximately 100 per year to about 140 per year since 2000. In
addition, the PD load also includes handling each about 140 supplements
per year.

To meet the challenge of increasing numbers of proposals, we recommend
that additional staff be assigned to CMS at both the PD and support
staff levels. Additional funds are also sorely needed to support the
many worthy projects that are proposed, but unable to be funded. The COV
recognizes that the need for additional money is expressed by virtually
every division within NSF, and that substantial increases in funding are
not likely in the near term. Therefore, it is necessary to deal with the
increasing proposal loads under the assumption of relatively flat
funding. Options include, but are not be limited to, restricting the
number of proposals from a single PI, readjusting the levels of support
provided for various activities, etc. As discussed earlier, a
division-level advisory committee could be very helpful in choosing the
most appropriate options and in planning future CMS activities.

In recommending that options be explored for reallocating resources to
support more worthy proposals, the COV affirms its support of
unsolicited proposals. We believe that the support of unsolicited
proposals by NSF is a significant, and perhaps unique, strength that
should be preserved in developing the CMS portfolio of sponsored
research.

**C.5 NSF would appreciate your comments on how to improve the COV
review process, format and report template.**

The time allotted for the COV involved two days, beginning Monday, 22
March at 1:00pm and ending Wednesday, 24 March at approximately noon.
There is too much information to digest and required interaction among
the COV and the NSF staff to comfortably produce a quality report in
this period of time. Three days would be better.

In addition, it appears to us that there are too many questions to
address in the template under section A. If the volume of material in
this section, which is in large part related to time consuming
accounting, could be reduced, the COV could spend more time on
substantive issues. As a minimum, the template should be designed to ask
the most important questions first within each subdivision of Section A.
External committees are inclined to address questions in order of their
appearance. Very lengthy and detailed sections, such as A.4, receive
disproportionate attention to the initial questions, leaving
insufficient time and concentration for what the COV regards as the more
important questions at the end of the section.

If detailed questions related to quantities and proportions of funds
required for appropriate portfolio balance remain in future templates,
then statistics specific to the questions should be developed in advance
and shared with the COV. The COV can then discuss whether the statistics
represent an appropriate balance, and perform an approximate check on
that balance with the limited number of jackets and time available for
evaluation.

The template provided to future COVs should be absolutely clean and free
of previous changes and variations in formatting rules. It should be as
flexible and adaptive as possible.

National Research Council. *Preventing Earthquake Disasters - A Research
Agenda for the Network for Earthquake Engineering Simulation (NEES)*.
Committee to Develop a Long-Term Research Agenda for the Network for
Earthquake Engineering Simulation. Board on Infrastructure and the
Constructed Environment Division on Engineering and Physical Sciences.
The National Academies Press. Washington, DC, 2003